CAN bus sniffer with an interactive console for $3

CAN sniffer

ESP32-C3 + SN65HVD230 in action

Introduction

The CAN protocol has become widespread, not only in the automotive industry, but also in enterprise, in various homemade products, and even in electric vehicles (such as VESC controllers).

Last November, I developed a handy tool to analyze and send CAN frames for my own use. Now, I would like to share my code publicly (under the MIT license) and discuss the project as a whole. If you’re interested in the code, here’s the link. Let’s get started!

First, I want to tell you why I decided to create this project. In my previous job, and later on in my hobby projects, I often had to deal with CAN bus analysis, where I needed to send frames for testing and debugging. At first, this problem was solved using the Arduino Uno and a standard CAN board from AliExpress, which were connected via SPI and contained a MCP2515 CAN controller and an MCP2551 transceiver.

The firmware was created in 5 to 10 minutes and was kept as simple as possible. It simply outputted received CAN frames to the UART, and it had the ability to send a limited number of frames. However, parsing data from the UART and converting it to CAN frames seemed like too much work for me. So, I came up with a better solution.

At work, I created several shields for the Raspberry Pi containing MCP2551 and SN65HVD230.At home, I made the same shield, and started working with CAN by opening two tmux panes - one for candump and one for cansend. After a few months, I realized that I was keeping a Raspberry Pi on the corner of my desk simply for the purpose of testing the CAN bus. I already had a powerful home server with a Core i7 processor and I didn’t see any tasks that would also require me to use a Raspberry Pi.

It was decided to create a compact, portable, affordable, and reliable device that would implement the following functions: receiving and transmitting CAN frames, selecting the frequency at which the CAN operates, setting customized filters for incoming messages, and monitoring errors on the CAN bus. Since I have been utilizing ESP32-C3 microcontrollers in all of my recent projects, it seemed reasonable to use that microcontroller.

Why esp32-c3?

At the moment, my favorite microcontroller series is the Espressif ESP32. One feature that is especially important for us is that almost all ESP32 models come with a CAN controller. This means we only need to purchase a transceiver, which will make assembly easier and reduce overall costs. Of course, there are some STM32 chips that also have CAN controllers, but at that moment I was doing all the current projects on esp32 and there were no STM32 devboards at hand. Specifically, the ESP32-C3 was selected because it has the lowest price and its peripherals meet our needs. It has a built-in USB port that supports Serial and JTAG output. Also, the errata section in the chip’s datasheet is much smaller than that of older models. If you search for “ESP32-C3 board” on AliExpress, you can find boards priced at only $1.8 each. I instantly liked them so much that I purchased about 20 of them. We will use the good old SN65HVD230 transceiver as a physical layer. AliExpress has modules with this chip priced between $0.5 and $1. The final price of the ESP32 development board + transceiver board + shipping works out to approximately $3.

So, what language will we use to write the software?

This time, it’s all about the good old C! We’ll be using the official ESP-IDF framework from Espressif. It’s a very powerful piece of software with good documentation and plenty of code examples. It has a forked FreeRTOS under the hood, but since our chip is single-core, the differences between the original FreeRTOS and the forked version are insignificant and can be safely ignored. This time we won’t be messing around with C++ support in ESP-IDF and linking together C and C++ code, we’re also leaving aside the toyish Arduino framework. The project mostly uses libraries that are already incuded in ESP-IDF and a simple implementation of a Linked List as a component. Once at uni, I wrote similar things in C as homework, but this time I got lazy, and just took the simplest possible implementation of Linked List from GitLab, fixed a thing or two and that’s it. The component for working with CAN is a part of ESP-IDF, and is perfectly documented. By the way, Espressif doesn’t call it CAN, but TWAI, because it does not support CAN FD, only classic CAN. I don’t see much point in such a name, but I guess it’s easier for some developers this way😀

gif with workflow

Interactive serial console

And now let’s talk about a very important thing, in fact, the main feature of this project: an interactive console that creates REPL environment and provides us with an ability to register commands, edit lines during input, multi-line input, auto-completion, hints, navigation through the command history, GNU-style arguments for commands. These features are provided to us by the ESP-IDF component called Console.

In more detail, linenoise is responsible for editing lines, hinting, completing, and input history, while argtable is responsible for parsing arguments. The console component itself is responsible for the REPL (read-eval-print loop) environment. Naturally, the libraries within the component are older versions that have been heavily modified for compatibility with ESP-IDF. However, this component didn’t allow me to implement exactly what I wanted, so I created a fork.

In the fork, I synced the changes in original linenoise with ESP-IDF’s version, fixed some unpleasant bugs, and added support for an asynchronous API. What is it and why? I really want the interface to be updated not just during user input but also when a new CAN frame is received. Also, they shouldn’t interfere with each other. To do this, we need to temporarily erase the prompt line, output a message, then redraw the prompt.At the same time, the user’s input text should not be lost. Additionally, I wanted to include useful information about the current status of the CAN bus in the prompt line. After significant refactoring and rewriting, asynchronous API support was added to linenoise. Therefore, I had to spend significant time synchronizing the new library’s functionality and patches from the ESP-IDF in my fork. Unfortunately, at the time, I did not find a way to do something similar to the Serial.available() function or the select(2) function in ESP-IDF. Specifically, I could not check for new characters without reading from the UART buffer. Subsequently, I discovered the uart_get_buffered_data_len() function. However, at that time, it was decided to introduce a semaphore called SemaphoreHandle_t stdout_taken_sem, which can block the process while another process is outputting text to the serial console. This semaphore prevents linenoise from printing data to the console until we complete our output.

More about the code structure

The entry point for an ESP-IDF application is the void app_main(void) function. In the functinon we first initialize uart_tx_ringbuf (an additional buffer used to output our CAN frames and logs to the console). Its purpose will be described in more details later. Next, we create the can_task process - it is responsible for monitoring the CAN status by periodically checking twai_read_alerts(), restoring the CAN bus after an error, as well as receiving frames, filtering them according to software filters and sending them to the Ring Buffer uart_tx_ringbuf for further output to the console. Also in can.h, SemaphoreHandle_t can_mutex is declared. We need it so that the user cannot stop the CAN interface with the candown command while the can_task process is blocked waiting for the twai_receive function - this would lead to panic and esp32 would go into reboot. Instead, to stop the interface, we wait for twai_receive to receive a frame, or exit after the timeout specified in the can_task_timeout variable. I set this value to 200 ms, taking it as the optimal one. If you set the value too high, there will be too much delay when trying to stop the interface, and if it is too small, the average delay between receiving the frame and printing it to the console will increase. Next, we initialize the file system. The command history is stored on a small fat32 partition in ESP32-C3’s flash memory (built-in or external, connected with SPI). Next comes the initialization of the console component, where we configure the parameters of the built-in USB-UART interface of our ESP32-C3, load the command history from the file system, register commands and their handlers. After that, the console_task_interactive process starts. This process creates a prompt, launches the linenoise handler, which provides all of the interactive input. It is also in this process that the commands entered by the user are processed. From this process, another one is created: console_task_tx, which is responsible for printing information to the console. It receives data from the previously mentioned uart_tx_ringbuf and outputs it to the console in this way: hides the prompt using linenoiseHide(), outputs data from the Ring Buffer and updates prompt (it contains the current CAN status and the number of errors), or simply updates prompt if the 200ms timeout has expired. Next, the promp is printed again using linenoiseShow(). The previously mentioned stdout_taken_sem is used here so that linenoise does not interfere with our output. The second semaphore console_taken_sem is also used for synchronization - it is needed so that during the processing of the entered command there are no attempts to output to the console - attempts to hide and show the prompt will otherwise work incorrectly, since the processing of the entered command occurs after linenoiseEditStop() and before the next call linenoiseEditStart().

Playing around with printf

A reasonable question that may arise is how does the output of information and logs to the console work? ESP-IDF uses the macros ESP_LOGI(), ESP_LOGE(), and ESP_LOGW() to output logs. These macros are not particularly concerned that linenoise might really dislike the output of something extra in the UART. (Remember how carefully we tried to synchronize the output of our information using semaphores?). Fortunately, ESP-IDF is flexible enough and provides us with the esp_log_set_vprintf function. With its help, we can set our printf_like_t function in this way: esp_log_set_vprintf(vsprintf); The implementation of the function:

/* This function will be called by the ESP log library every time ESP_LOG needs to be performed.
      @important Do NOT use the ESP_LOG* macro's in this function ELSE recursive loop
        and stack overflow! So use printf() instead for debug messages.
*/
int vxprintf(const char *fmt, va_list args) {
    char msg_to_send[300];
    const size_t str_len = vsnprintf(msg_to_send, 299, fmt, args);
    xRingbufferSend(uart_tx_ringbuf, msg_to_send, str_len + 1, pdMS_TO_TICKS(200));
    return str_len;
}

Great! Now, the ESP_LOG macros no longer print data to the console, but instead send it to our Ring Buffer, from which console_task_tx prints the data. However, what should we do about printf in our code? After all, it can also break things.

int xprintf(const char *fmt, ...) {
    va_list(args);
    va_start(args, fmt);
    return vxprintf(fmt, args);
}

No worries, instead of using printf, we will use the xprintf function that we have just written. We can use it the same way we use printf to print data.

Additionally, for greater convenience, I have implemented a function that can print text using the printf/xprintf functions (regular printf for output from the command handler when line noise is not active). We can set custom color of the output, and optionally, a timestamp can be printed before the message:

int print_w_clr_time(char *msg, char *color, bool use_printf) {
    print_func pr_func;
    if (use_printf) pr_func = printf;
    else pr_func = xprintf;
    char timestamp[20];
    timestamp[0] = '\0';
    if (timestamp_enabled) {
        snprintf(timestamp, 19, "[%s] ", esp_log_system_timestamp());
    }
    if (color != NULL) {
        return(pr_func("\033[0;%sm%s%s\033[0m\n", color, timestamp, msg));
    } else {
        return(pr_func("%s%s\n", timestamp, msg));
    }
}

The interactive console is fun! And what commands are implemented?

One of the most important commands ishelp - the command provides detailed information for all commands with examples of usage. Now let’s list all the other commands, grouping them by the files in which they are declared:

Writing a C code to parse arguments for these thing was really an unforgettable experience.

Sounds great, what’s so special about cansmartfilter?

Sheldon Cooper

The issue is that the CAN controller on the ESP 32 has relatively limited capabilities for filtering messages by ID. It can only handle 1-2 patterns at a time, and if you need two patterns with an extended ID, only part of the ID will be used for filtering. We can choose common bits and use them to filter CAN frames, but eventually, this may not be enough, and we will need to use software filtering instead. For example, the MCP2515 is a CAN controller that has a large number of pre-programmed filters and for using it we wouldn’t be so much constrained by the hardware.

However, we don’t have to worry much about that. This gives us a really interesting problem to solve! Our cansmartfilter command can accept from 1 to CONFIG_CAN_MAX_SMARTFILTERS_NUM filters. By default, I set this value to 10, but if desired, you can raise it, it’s all about available heap size, you can put 20 filters or more. Pay attention to max command length, you may need increase it as well. The filters are entered in the code#mask format. I have not implemented filtering of frames with standard ID in cansmartfilter yet, because this is not used in my projects, there is only filtering of frames with extended ID. To filter frames with standart ID, use canfilter. In general, the command looks like this: cansmartfilter 11223344#FFEECCBB 33123A#23BBE0 90#AB - here we have defined 3 smart filters. Mask and code are uint32_t numbers in hex format. The ones in the mask mean the bits that are taken into account by the filter, the zeros are the bits that are ignored. For example, such a filter 0000FF00#0000FFFF will accept only frames that start at FF00, there is no filtering for the remaining bits. I.e. 0029FF00 and 00ABFF00 will pass, but 00ABFF05 will be declined by our filter. As you can see, everything is quite simple, and you can set plenty of filters.

Now about how it works under the hood. Yes, that’s where Linked List came in handy for me - to store a list of filters. I store filters as elements of a Linked List, each filter is stored in a struct like this:

typedef struct {
  uint32_t filt;
  uint32_t mask;
} smart_filt_element_t;

In the process of parsing command arguments by using tricky bitwise logic we figure out whether it is possible to cover all filters with a hardwired filter. It is possible only in 2 cases: either we have only 1 filter, or the set of frames accepted by one filter is a subset of frames accepted by another filter. For example, if all of the filters are the same. In the above cases, software filtering is not enabled and the cansmartfilter command can be used as an alternative to canfilter, but with a more pleasant syntax. Next, the CAN interface is installed and started with the canup -f command and filtering begins to work. The bits common to all filters are filtered using the hardware filter, and those frames that pass the hardware filter are filtered in can_task when a new frame is received. It’s just as simple as that:

// somewhere in can_task
  const BaseType_t sem_res = xSemaphoreTake(can_mutex, 0);
  if (sem_res == pdTRUE) {
      while ((ret = twai_receive(&rx_msg, can_task_timeout)) == ESP_OK) {
          char data_bytes_str[70];
          if (adv_filters.sw_filtering) {
              if (!matches_filters(&rx_msg)) continue;
          }
          can_msg_to_str(&rx_msg, "recv ", data_bytes_str); 
          print_w_clr_time(data_bytes_str, LOG_COLOR_BLUE, false);
      }
      xSemaphoreGive(can_mutex);
      vTaskDelay(1);
  }
  if (sem_res != pdTRUE || ret == ESP_ERR_INVALID_STATE || ret == ESP_ERR_NOT_SUPPORTED) {
      vTaskDelay(can_task_timeout);
  }
// end of can_task fragment

bool matches_filters(const twai_message_t *msg) {
      const List *tmp_cursor = adv_filters.filters;
      while (tmp_cursor != NULL) {
          const smart_filt_element_t* curr_filter = tmp_cursor->data;
          if ((msg->identifier & curr_filter->mask) == curr_filter->filt) {
              return true;
          }
          tmp_cursor = tmp_cursor->next;
      }
      return false;
}

Time to build it!

It can be soldered and flashed in just 30 minutes. It may take another 30 minutes to sort out the commands. After that, you will have a very convenient CAN debugging tool, cheap and portable.

Build guide:

Lazy build guide:

If you don’t want to install ESP-IDF and compile the project I have a ready to use binary file for you. You can install it manually with esptool or use web installer below and install it directly from your web browser. If you use a binary build, then connect CAN_RX to GPIO9 and CAN_TX to GPIO8. You can get the binary build and instructions for flashing it here. If you prefer fully automated install - just go to Web installer section below - it doesn’t require any extra software for flashing - only your web-browser.

Web installer

Connect your esp32-c3 to your PC and press the button below.

Your browser is not supported, try using desktop versions of Chromium, Google Chrome or Edge. You are not allowed to use this on HTTP!


Demo