Skip to main content

Compressing your data

You should consider compressing your data to make maximum use of every byte in your payload.

Consider an application that needs to send 3 measurement readings, each reading is a decimal from 0 to 100, with 2 decimal places.

A typical CSV representation of this might look like this:

10.25,77.77,23.45

If transmitted as CSV, in text mode, that data would consume 17 bytes.

Consider the accuracy required

Do we really need 2 decimal places? This sounds obvious, but a really simple way to reduce the size of your data is to only transmit what you need. Let's consider that we can live with only 1 d.p:

10.3,77.8,23.5

That's reduced the payload to 14 bytes.

Binary mode

Now let's convert our measurements to integers by multiplying by 10:

103,778,235

(that's saved another 3 bytes, and we're still in text mode - and we could go further by removing the delimiters).

Each measurement now requires an integer range of 0-1000. 1000 in hex is 0x3e8, so we need a 10-bit field to store each measurement.

The table below shows how you could bit-pack these 10-bit fields into 8-bit bytes.

AAAAAAAA
AABBBBBB
BBBBCCCC
CCCCCC--

Using bitwise operators:

Measurement A = (byte[0] << 2) + (byte[1] >> 6)

Measurement B = ( (byte[1] & 0x3F) << 4 ) + ( byte[2] >> 4 )

Measurement C = ( (byte[2] & 0x0F) << 6 ) + ( byte[3] >> 2 )

The same data is now consuming less than 4 bytes!