May 29th, 2017 by

Sample compression experiment

No posts in a while, time for a technical writeup!

I (Tomas) did a small experiment around compressing 24-bit sample data today, in an effort to try to get more throughput out of the 115200 baud RS-485 connection we’ll have in our lunar electrostatic instrument.

Our hardware currently delivers three channels of 24-bit data sampled at around 1800 samples/s. With an overhead equivalent to about one channel’s worth of data this comes to 1800 * 4 * 24 = 172800 bit/s, more than our poor RS-485 bus can handle. The final version of our instrument will have as many as nine channels active simultaneously, which means spending 4-5x more time downloading than actually capturing data!

Since we want to get as much science out of our instrument as possible, I spent some time trying to investigate ways to increase throughput. Simplicity is key since we’re running on a 7.37 MHz 8-bit microcontroller.

The idea for a codec for this is based on two observations:

The combination of these two means that each sample can be “trimmed”, lopping off both most-significant and least-significant bits in a lossless process. An experiment with a 1027 B block of actual instrument data (7 B header + 1020 B data) yielded these results:

Case Size Notes
original 1027
bzip2 -9 1101 worse!
lz4 1011 LZ4_compress_default()
lz4 v2 1018 data transposed prior to compression (least significant bytes first)
gzip -9 945
this codec 762 fuzzed with american fuzzy lop (afl)

With the codec the samples compress to 16 bits each (from 24), which with overhead should result in a throughput increase of 34%. Not super great, but potentially useful. afl was quite useful for discovering bugs in the implementation. The other options performed worse (especially bzip2), and will most definitely not be worth the cycles.

TL;DR: a simple lossless codec can often work much better than a general-purpose compressor :]

RE: the project overall, we’re very busy getting our current prototype ready to send to IRF Kiruna. Hence the lack of updates lately.

See you next time!


AddressScube, Fabriksgatan 8b kv., 903 31 Umeå, Sweden
E-Mail (Press)