Juha,
I'm worried about truncating too far because I could be close to the emitters which would cause a large SNR which (as far as I understand) also means more dynamic range. Losing bits will chop dynamic range :( Now, I assume that as long as I am on top of the RX gain I can ensure that I don't get massive SNR signals thus not needing the extra bits. Please correct me if that's wrong as I'm not 100% sure that's right.
As for the various parallel compression algorithms, I found a list at
http://askubuntu.com/questions/258202/multi-core-compression-tools (about half way down) of tools. I tried out pbzip2 earlier with `cat largeFile.sc16 | pzip2 -1 | pv > /dev/null` and found that it was only running at ~ 25 MBytes/s which is too slow. That number is from my i7 4870HD Macbook Pro. If I run pgzip with `cat largeFile.sc16 | pigz -1 | pv > /dev/null` I get around 120 MBytes/s. Also, the file was stored in /dev/shm for best results. I know that my desktop machine which will be doing the recording can run an sc16 file through at around 300-350 MBytes/s but that maxes out the entire machine. Do you know of any tweaks that can be done to increase the speed with a slight trade off in compression?
Marcus,
I should probably have stated the problem a little more clearly. I would like to be able to record 100+ MHz swaths of busy spectrum. Think 900 MHz in a busy city or LTE, WiFi, 433 MHz, etc. Just trying to build up a large collection of signals to learn how to demodulate with. I also hope one day to be able to pull down entire transponders of satellites :D So, the basic point here is that there will be a lot going on in the spectrum that I am recording. Oh, and the captures could be several minutes long. Sometimes I find really interesting signals that look crazy in the waterfall (fosphor) plot so I want to record large files of those to poke at later.
I'm really intrigued by your idea of storing the log magnitudes. It will take me a bit to wrap my head around it, but it sounds really interesting :D
Thank you both very much!!
-Dave