Uploading sensor data using a cellular network

Keeping in mind the global view is very important when designing an IoT system. Why?

A first distinctive feature of IoT projects is that they require integration of technical blocks originating from three different domains: electronics, communications and software. And in every of these three domains, various different subdomains are usually involved: analog and digital electronics, wireless communication modules, protocol stacks, embedded software, user interfaces, database management, analytics, geospatial data, etc.

A second distinctive feature is the large number of totally different use cases. IoT encompasses smart cities (and what exactly is a smart city?), energy management, environmental monitoring, building automation, home automation, fleet management, tracking systems, etc.

Many different use cases and several different technologies: no wonder that designing a reliable system correctly fulfilling user needs requires a global view!

Let’s go through an example, that will allow us to better understand this.

In this example, a sensing equipment has to periodically upload a set of sensed data to a central system, using a GSM network in GPRS mode. The equipment is powered by a photovoltaic module and associated battery. Consequently, one of the main requirements is a low power consumption.

The design team is really experienced in sensors, and in digital electronics. But it is not so experienced in software development and communication protocols. Consequently, it decides to subcontract this part of the project.

For the hardware part, the design relies on a so-called “low power” cellular module, and a low-power microcontroller.

For the software part, the subcontractor decides to store sensed data in CSV files, and to upload every file using FTP. Size of every file is around 10 KB.

Let’s look at this part in a detailed way.

CSV file

Firstly, a CSV file contains human-readable characters. Associated data uses more room than if it was stored in binary form. That’s not such a big deal, for values that are usually coded on 2 bytes. For instance, 65000 uses 5 + 1 bytes (0x36 0x35 0x30 0x30 0x30 and the CSV separator, e.g. a semicolon), while 17 uses 2 + 1 bytes. But that’s far more inefficient for fields like timestamps. Usually, a binary-coded timestamp requires only 4 bytes. In our example, it has been decided to use following ASCII format for timestamps:

dd/mm/yyyy hh:mm:ss+hh:mm

which is 25 + 1 byte long (more than 6 times the size of the binary version).

The planned overall CSV file line format is:

"00:1B:16:2D:4C:67";"11/02/2014 08:01:16+02:00";"112";"26023";"702";"2800";
"3045";"21905";"1223";"64"

I don’t know the domain of definition of every transmitted integer value. But let’s consider that they are 2-byte (unsigned) integers. Consequently, storing the above line in binary format requires 6 + 4 + 8 x 2 = 26 bytes. Storing the line in ASCII format requires 101 bytes, plus an additional 1 or 2 character(s) (carriage return and / or linefeed). So, more than three times the number of binary format bytes!

Now, let’s look at data semantics. Every line contains the address of the equipment, a timestamp, and sensed data. Compliance with usual CSV format requires that equipment address is present on every line. Using another (binary) format would allow to write it only once into the file. This would additionally decrease transfer size of (number of lines – 1) x 6.

Additionally, a simple compression scheme could be used for sensed data. For instance, differential values could be transmitted, after a first absolute value.

Why reducing data volume? Because the less you transmit data, the less you consume power. With GPRS, data throughput can be as low as 1000 bytes / s. For a 10 KB file, this means a minimum duration of 10 s (actually, additional time is required: see below).

FTP

Now, let’s look at FTP. In usual conditions, FTP uses two TCP connections: one to control the FTP session, the other one to transmit data. A whole session looks like:

  • open control connection
  • wait for server acknowledge
  • send username, wait for server acknowledge
  • send password, wait for server acknowledge
  • set transfer type, wait for server acknowledge
  • open data connection
  • send file
  • terminate connections

As it can be seen, FTP protocol requires to wait for at least four acknowledges from remote side. For TCP, additional acknowledges are necessary, their exact number depending on parameter negociation and on file size. For GPRS, latency time can be around 1 s (almost the worst case). In our example, this means around 5 s just waiting for acknowledges.

Consequently, total communication time (data transmission and acknowledges) can be around 15 s.

If a binary protocol (even a very simple one) had been chosen, in order to reduce data volume and to minimize the number of downlink messages, communication time could have been reduced to around 3 or 4 s. And power consumption would have been reduced accordingly.

Conclusion

A low power consumption was very important, in this project. The hardware design complied with this requirement. But as the design team was not experienced enough in programmming, they decided to subcontract the software part. They didn’t realized that this part could impact power consumption as well, depending on the implementation chosen by the subcontractor. And the subcontractor chose the simplest solution for him: store data in CSV files, and use the FTP stack provided with the communication module. Energy savings coming from hardware design are partly neutralized (or perhaps totally neutralized, or even worse) by the software layer.