Processing Data from Serial Line in Arduino

Introduction

When writing Arduino programs, you sometimes need to receive some commands or data from serial line. It may seem easy but often you run into problems. Let me explain what I mean by an example. You have a program which is doing lot of things – reads the sensors, controls outputs, shows current status on a display, etc. You make your loop code run fast, for example, it is executed 10 times per second. Now you want your program to respond to commands sent from the serial line or to process data sent from another Arduino. How can you do this?

Well, you put if (Serial.available() > 0) into your loop and if there are some data (or command), you process it. But here, the problems start to show up. The condition is true if there are one or more characters available. If your data or command is more than 1 char long, there is a good chance that you cannot process it yet because it is not completely received. There may be just the first char, or the first two chars, etc. One solution would be to wait for all data to come; something like calling Serial.readStringUntil which will read until a terminator character is received or there is a timeout. But you cannot stop your loop and wait! You want to keep processing the inputs and controlling outputs 10 times per second – imagine your program controls a quadrocopter; it cannot just stop controlling the aircraft and wait for the data to arrive from serial line.

Another solution is to change the condition to something like this:

if ( Serial.available() == 3 )
    Process the data….

In this code, I assume that the data/command are 3 chars long and I only do the processing if there are exactly 3 characters received. Seems fine? Not always. Let me show you an example:
Suppose one Arduino is sending numbers (fixed 3 digits length) once per second. Our program in another Arduino is receiving the numbers and it needs to run its loop 10 times per seconds to control the nuclear reactor in your basement. Here is what the sequence of chars coming from the serial line could look like: 

001     002    003    004.

Now if the receiving program is “in sync” – if it starts reading from the beginning, it will receive the numbers all right. But imagine, for example, that the receiving Arduino is powered off and then back on. It may start receiving at any random position in the sequence. For example, the first char it receives may be “2”, and then the condition  if (Serial.available() == 3) will be true when we receive "200" instead of "002" or "003" – invalid number.

The situation may be even worse if the sender is a person rather than another Arduino. There may be invalid commands, random spaces between each char, etc. Plus you probably don’t want to limit the commands to fixed length. So this solution is also far from ideal.

Note: The Serial Monitor in Arduino IDE may somehow hide these problems because it lets the user type the whole string (command) and then send it by the Send button. So the data on the serial line come in short bursts like “command1    command2”  instead of  “c  o m m a  n d   1       c o  m “ etc., which will happen if the chars are sent immediately after the user types them  – which is what normal terminal programs do. Anyway, I am sure you want your program to be robust and handle all the situations nicely.

So here, I present a solution which should be able to handle the characters coming in random times and process the commands or data when it is really ready without blocking the loop for long time.

Using the Code

The first example processes commands from the user. The commands can be up to MAX_DATA_LEN chars long and they are received in a way which does not block the loop() function. The code doesn't make any assumptions such as that the program will be always receiving characters when the user starts to type.

Important Note: When you test the program in Arduino Serial Monitor, set it to send the Carriage Return characted in the bottom of the window instead of the default No line ending. The program expects the commands to be terminated by Return key (carriage return char).

About String Class

In the code, I use C language strings - that is an array of chars. There is no data type for strings in the C language. It is possible to use string class in C++ and there is a String class in Arduino which allows you to work with strings the way you may know from languages like C# or Java, but I don’t recommend using this class. For one, I’ve read some bad news about its efficiency, memory leaks, etc., and for second even if the class was well written, it will be still quite inefficient compared to the C language string handling because it will need to dynamically allocate and free memory when you do things like myString = myString + “ you too!”; Using dynamic memory and sometimes even C++ is not such a good idea on embedded systems like Arduino, as I tried to explain in my article here.

I know that working with C style strings looks scary, but it is in fact rather easy and once you learn it, you will be able to write efficient code and do all the cool tricks with the characters you receive.

What the Program Does

The example code below receives commands from serial line and executes them. These commands are supported:

• ledon to turn on the onboard LED on pin 13
• ledoff to turn off the onboard LED
• ver to print the verison of the program.

In the loop(), there is delay(100); which simulates some useful work the program should do, such as reading sensors, processing the inputs, etc. Then there is function processSerialData which takes care of processing the commands received from serial line. It does not block the program for long, just stores the character(s) received since it was last called and if a complete command is received, it will execute it.  The code is explained in more detail below:

// Processing serial data for  arduino
// Support commands:
// ledon - turn on LED on pin 13
// ledoff - turn off LED on pin 13
// ver - print program version
// Commnands must be terminated by Enter (return) - enable this in Arduino
// serial monitor - set "Carriage return" instead on "No line ending".
// Or change the TERMINATOR_CHAR in the code below.
 
#include <string.h>     // for strcmp()

// max length of a the data or command
#define   MAX_DATA_LEN    (8)

#define   TERMINATOR_CHAR ('\r')

// the buffer for the received chars
// 1 extra char for the terminating character "\0"
char g_buffer[MAX_DATA_LEN + 1];

// function prototype
void processSerialData(void);

void setup() {
  // put your setup code here, to run once:
  pinMode(13, OUTPUT);
  Serial.begin(9600);
}

void loop() {
 
  // Here we read the sensors, calculate the output etc.
  delay(100);

  // Here we process data from serial line
  processSerialData();

}

void processSerialData(void) {
  int data;
  bool dataReady;
  while ( Serial.available() > 0 ) {
      data = Serial.read();
      dataReady = addData((char)data);  
      if ( dataReady )
        processData();
  }
}

// Put received character into the buffere.
// When a complete command is received return true, otherwise return false.
// The command is terminated by Enter character ("\r")
bool addData(char nextChar)
{  
  // This is position in the buffer where we put next char.
  // Static var will remember its value across function calls.
  static uint8_t currentIndex = 0;

    // Ignore some characters - new line, space and tabs
    if ((nextChar == '\n') || (nextChar == ' ') || (nextChar == '\t'))
        return false;

    // If we receive Enter character...
    if (nextChar == TERMINATOR_CHAR) {
        // ...terminate the string by NULL character "\0" and return true
        g_buffer[currentIndex] = '\0';
        currentIndex = 0;
        return true;
    }

    // For normal character just store it in the buffer and move
    // position to next
    g_buffer[currentIndex] = nextChar;
    currentIndex++;

    // Check for too many chars
    if (currentIndex >= MAX_DATA_LEN) {
      // The data too long so reset our position and return true
      // so that the data received so far can be processed - the caller should
      // see if it is valid command or not...
      g_buffer[MAX_DATA_LEN] = '\0';
      currentIndex = 0;
      return true;
    }

    return false;
}

// process the data - command
// strcmp compares two strings and returns 0 if they are the same.
void processData(void)
{
    if ( strcmp(g_buffer, "ledon") == 0 ) {
       digitalWrite(13, HIGH);
       Serial.println("LED1 is on");
    }
    else if (strcmp(g_buffer, "ledoff") == 0 ) {
       digitalWrite(13, LOW);
       Serial.println("LED1 is off");
    }
    else if (strcmp(g_buffer, "ver") == 0 ) {    
       Serial.println("Version 1.0");}
    else {
      Serial.print("Unknown command ");
      Serial.println(g_buffer);
    }
}

Explanation of the Code

The processSerialData function reads all available characters from the serial line and calls addData() for each of them. The addData function puts the new char into buffer and decides if a complete command was received. For this decision, it simply checks whether the new char is special terminator character TERMINATOR_CHAR - in the code, I use the Enter key ('\r') but you can change it to something else.

The characters are stored into a C-style string, that is an array of char variables. One important thing to know about C strings is that the end of the string is marked by a special character written as '\0' and called NULL character  - which may be a confusing name but don't worry about it and just make sure you always put '\0' at the end of your string if you create it in a char-by-char way - just as the addData function does. You don't need to add the '\0' if you write the string as chars in double quotes, such as the "ledon" string in the processData function. In such case, the compiler adds the terminating char automatically. But always be sure to have space for this terminating char in your array - see how the g_buffer is declared with +1 to allow storing MAX_DATA_LEN of usable chars.

So the addData function stores each character into the global string (array of chars) g_buffer at a position which is then increased, so that the next char is stored as the next element in the array.

If complete command is received, the function processData is called. This function is responsible for executing the commands. In the example, it just turns on/off the LED or prints the version of this program. To decide which command is received, the function must compare the string stored in g_buffer with supported commands. Note that you cannot compare strings in C just like str1 == str2. You would be comparing just the addresses of the strings, but not the contents. You need to use standard library function strcmp - string compare. It returns 0 if the strings are the same.

Trying the Code

To try it, just copy-paste the above code to your Arduino sketch, upload it to Arduino and open Serial monitor. First, select Carriage Return in the combo box in the bottom of the Serial monitor so that it sends the Return key when you click the Send button or press Enter on your keyboard. Type ledon and press Enter to turn on LED on your Arduino board. If you enter invalid command, the program should tell you so.

Receiving Numbers from Serial Line

For those interested in passing some numbers to their programs, here is the modified version of the code which shows how to receive two numbers from the serial line. When sending numbers, you will need to define some protocol to be sure that the receiving program recognizes the numbers and does not mix them together - as explained above in the example with sending 001 002, etc. In this program, I use human friendly notation but you can easily change it to suit your needs. So in my program, the input is expected in this format: "x=10y=20#". There is x= and y= to mark the start of the numbers.

Here is the code. To try it, run it in your Arduino and from the Serial Monitor send, for example, x=10y=20#. The program should respond with "New params received. x = 10, y = 20". 

// processing serial data for  arduino
// This shows receiving two numbers, the format of the command
// is x=10y=20#
#include <string.h>     // for strcmp()

// max length of a the data or command
#define   MAX_DATA_LEN    (16)

#define   TERMINATOR_CHAR ('#')

// the buffer for the received chars
// 1 extra char for the terminating character "\0"
char g_buffer[MAX_DATA_LEN + 1];

// function prototype
void processSerialData(void);

void setup() {
  // put your setup code here, to run once:  
  Serial.begin(9600);
}

void loop() {
 
  // Here, we read the sensors, calculate the output, etc.
  delay(100);

  // Here, we process data from serial line
  processSerialData();
}

void processSerialData(void) {
  int data;
  bool dataReady;
  while ( Serial.available() > 0 ) {
      data = Serial.read();
      dataReady = addData((char)data);  
      if ( dataReady )
        processData();
  }
}

// Put received character into the buffere.
// When a complete command is received return true, otherwise return false.
// The command is terminated by Enter character ("\r")
bool addData(char nextChar)
{  
  // This is position in the buffer where we put next char.
  // Static var will remember its value across function calls.
  static uint8_t currentIndex = 0;

    // Ignore some characters - new line, space and tabs
    if ((nextChar == '\n') || (nextChar == ' ') || (nextChar == '\t'))
        return false;

    // If we receive Enter character...
    if (nextChar == TERMINATOR_CHAR) {
        // ...terminate the string by NULL character "\0" and return true
        g_buffer[currentIndex] = '\0';
        currentIndex = 0;
        return true;
    }

    // For normal character just store it in the buffer and move
    // position to next
    g_buffer[currentIndex] = nextChar;
    currentIndex++;

    // Check for too many chars
    if (currentIndex >= MAX_DATA_LEN) {
      // The data too long so reset our position and return true
      // so that the data received so far can be processed - the caller should
      // see if it is valid command or not...
      g_buffer[MAX_DATA_LEN] = '\0';
      currentIndex = 0;
      return true;
    }

    return false;
}

// process the data - we expect this format:
// x=10y=20
void processData(void)
{
  char* n1pos = strstr(g_buffer, "x=");
  int number1 = -1, number2 = -1;   // preset invalid values
  //int dataLen = strlen(g_buffer); // for checking if there is number after the x or y

  // n1pos points to the substring starting with "x=".
  // strlen(n1pos) > 2 makes sure there is something after the x=
  // but it does not check if there is number... just that the string doesn't end
  // right after the = char.
  if ( n1pos != NULL && strlen(n1pos) > 2) {
    // if first number is found, convert it to integer
    // n1pos points to "x=" so we need to add 2 to point
    // atoi() to the beginning of the number.
    number1 = atoi(n1pos+2);
    char* n2pos = strstr(g_buffer, "y=");
    if ( n2pos != NULL && strlen(n2pos) > 2  )
      number2 = atoi(n2pos+2);
  }

  if ( number1 >= 0 && number2 >= 0 ) {
    Serial.print("New params received. x = ");
    Serial.print(number1);
    Serial.print(", y = ");
    Serial.println(number2);
  } else {
    Serial.println("Wrong data format, please use x=123y=123");
  }   
}

Explanation of the Code

The core of the program is the same as in the first example. The difference is that the TERMINATOR_CHAR is set to # and the processData function is now different. It tries to parse the input string and obtain the two numbers from it. For this, it uses standard C function atoi which converts string to a number. If the numbers are obtained, they are printed to serial line. If not, an error message is printed. There is also strstr function used to find the x= and y= substrings in the received string.

The strstr is a standard C function which searches for a substring in another string. It returns NULL if the string is not found or the pointer to the beginning of the substring if it is found. I know that "pointers" are scary, but you don't need to understand all about them to use the strstr. It may be helpful if I tell you just this: the name of an array in C is a pointer to its first element. A string in C is an array of chars. So the name of a string is actually a pointer to the beginning of the string and you can think about it as the string. For example, g_buffer is in fact a pointer to the received string. If you define:

char* p = g_buffer;

the p points to the received string too.

In the code, I save the result of strstr into char* n1pos:

char* n1pos = strstr(g_buffer, "x=");

If the g_buffer contains "aaax=12y=10#" then n1pos will point to the "x" character. And since pointer is actually like a string, you can thing about n1pos as a string "x=12y=10#" - the substring of g_buffer starting at "x".

And you can add numbers to pointers and thus move the beginning of the "string" represented by n1pos. I use n1pos+2 as the input for the atoi function so that the function receives just the number starting after "x=". It is as if I call it like atoi("12y=10#"); The extra chars after the number 12 are not a problem; atoi will stop the conversion when it encounters character which is not a number.

I hope I did not confuse things too much with this attempt to explain pointers in two paragraphs. Please take a look at some good tutorials on the internet if you want to learn more about them.

History

• 6^th August, 2018: First version