How to change the default I²C pins in Zephyr

When working with I²C devices in the Zephyr real-time operating system, a lot is configured behind the scenes using a devicetree, a hierarchical data structure that describes available hardware. It took me some time to figure out how to change the default I²C pins (SDA and SCL). Here are my notes.

I tried this with Zephyr's BME280 Humidity and Pressure Sensor sample. My goal was to flash the sample firmware to Nordic Semiconductor's nRF52840 Dongle with a BME280 breakout board.

What are the default I²C pins?

First I needed to find out the default I²C pins. To check this, I looked at the devicetree source include file for the nRF52840 Dongle's Pin Control:

koan@tux:~$ less ~/zephyrproject/zephyr/boards/arm/nrf52840dongle_nrf52840/nrf52840dongle_nrf52840-pinctrl.dtsi

The relevant configuration for the i2c0 peripheral is as follows:

i2c0_default: i2c0_default {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
        };
};

i2c0_sleep: i2c0_sleep {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
                low-power-enable;
        };
};

This configures pin 0.26 for SDA and pin 0.27 for SCL. My first thought was to simply connect the BME280 to these pins on the nRF52840 Dongle. However, upon examining the board's pinout, I discovered that pin 0.26 is only exposed as a pad on the bottom, and pin 0.27 isn't accessible anywhere:

/images/nrf52840-dongle-pinout.png

How to change the default I²C pins?

Since the default I²C pins weren't suitable on this board, I needed to change them. I decided to connect the BME280 sensor board as follows:

BME280

nRF52840 Dongle

SDA

0.31

SCL

0.29

GND

GND

VCC

VDD

This looks like this on a breadboard:

/images/nrf52840-dongle-bme280.jpg

Next, I copied the sample code so I could change it:

koan@tux:~$ cp -r ~/zephyrproject/zephyr/samples/sensor/bme280/ .
koan@tux:~$ cd bme280/

Then, I added a devicetree overlay file called nrf52840dongle_nrf52840.overlay to the project's boards directory. The content of the overlay file is as follows:

/*
 * Configuration of a BME280 device on an I2C bus.
 *
 * Device address 0x76 is assumed. Your device may have a different
 * address; check your device documentation if unsure.
 */
&pinctrl {
        i2c0_default: i2c0_default {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                };
        };

        i2c0_sleep: i2c0_sleep {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                        low-power-enable;
                };
        };
};

&i2c0 {
    status = "okay";
    bme280@76 {
        compatible = "bosch,bme280";
        reg = <0x76>;
    };
};

The &pinctrl section was copied from the devicetree source include file for the nRF52840 Dongle, with pins 26 and 27 changed to 31 and 29, respectively.

The &i2c0 part defines the bme280 sensor with the I²C address 0x76. We need to add this sensor definition here because the nRF52840 Dongle doesn't have it built in.

So, with this overlay, you can use the BME280 sensor connected to pins 0.31 (SDA) and 0.29 (SCL).

Because the file name of the devicetree overlay is the board name nrf52840dongle_nrf52840 with the extension .overlay, Zephyr's build system picks it up automatically if you're building the project for the nRF52840 Dongle and merges it with the default devicetree of the board. If you're building the project for another board, the overlay will be ignored.

Building the code

Let's build the sample code with this devicetree overlay, adding configuration options to initialize USB at boot and enable Zephyr's shell:

koan@tux:~/bme280$ source ~/zephyrproject/zephyr/zephyr-env.sh
koan@tux:~/bme280$ west build -b nrf52840dongle_nrf52840 . -- -DCONFIG_USB_DEVICE_INITIALIZE_AT_BOOT=y -DCONFIG_SHELL=y

Now let's examine the generated devicetree:

koan@tux:~/bme280$ less build/zephyr/zephyr.dts

This includes the bme280 sensor on the i2c0 bus:

i2c0: i2c@40003000 {
        compatible = "nordic,nrf-twi";
        #address-cells = < 0x1 >;
        #size-cells = < 0x0 >;
        reg = < 0x40003000 0x1000 >;
        clock-frequency = < 0x186a0 >;
        interrupts = < 0x3 0x1 >;
        status = "okay";
        pinctrl-0 = < &i2c0_default >;
        pinctrl-1 = < &i2c0_sleep >;
        pinctrl-names = "default", "sleep";
        bme280@76 {
                compatible = "bosch,bme280";
                reg = < 0x76 >;
        };
};

The Pin Control section shows the I²C pin defaults:

i2c0_default: i2c0_default {
        phandle = < 0x4 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
        };
};
i2c0_sleep: i2c0_sleep {
        phandle = < 0x5 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
                low-power-enable;
        };
};

Here, 1f is of course the hexadecimal representation of 31 and 1d corresponds to 29.

Running the sample

Now that we're confident that the devicetree is correct, let's create a firmware package and flash it to the nRF52840 Dongle:

koan@tux:~/bme280$ nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application build/zephyr/zephyr.hex --application-version 1 bme280.zip

|===============================================================|
|##      ##    ###    ########  ##    ## #### ##    ##  ######  |
|##  ##  ##   ## ##   ##     ## ###   ##  ##  ###   ## ##    ## |
|##  ##  ##  ##   ##  ##     ## ####  ##  ##  ####  ## ##       |
|##  ##  ## ##     ## ########  ## ## ##  ##  ## ## ## ##   ####|
|##  ##  ## ######### ##   ##   ##  ####  ##  ##  #### ##    ## |
|##  ##  ## ##     ## ##    ##  ##   ###  ##  ##   ### ##    ## |
| ###  ###  ##     ## ##     ## ##    ## #### ##    ##  ######  |
|===============================================================|
|You are not providing a signature key, which means the DFU     |
|files will not be signed, and are vulnerable to tampering.     |
|This is only compatible with a signature-less bootloader and is|
|not suitable for production environments.                      |
|===============================================================|

Zip created at bme280.zip
koan@tux:~/bme280$ nrfutil dfu usb-serial -pkg bme280.zip -p /dev/ttyACM0
  [####################################]  100%
Device programmed.

After this, connect to the UART interface over USB:

koan@tux:~$ screen /dev/ttyACM0

You should see that the I²C device has been detected, and the sensor values are being displayed:

[00:00:00.317,932] <dbg> BME280: bme280_chip_init: ID OK
[00:00:00.328,582] <dbg> BME280: bme280_chip_init: "bme280@76" OK
*** Booting Zephyr OS build zephyr-v3.4.0-837-gb4ed6c4300a2 ***
Found device "bme280@76", getting sensor data
temp: 25.850000; press: 101.987800; humidity: 59.248046
temp: 25.840000; press: 101.986144; humidity: 59.187500
temp: 25.850000; press: 101.985195; humidity: 59.176757

Using ESPHome on the Raspberry Pi Pico W and other RP2040 microcontroller boards

ESPHome is an open-source program that allows you to create your own home-automation devices using an ESP32, ESP8266, or RP2040 microcontroller board that you connect to LEDs, sensors, or switches. What sets ESPHome apart from other solutions like Arduino or MicroPython is that you don't need to program. Instead, you define your components and their respective pin connections in a YAML configuration file. ESPHome then generates the necessary C++ code and compiles it into firmware that you can install on the device. [1]

ESPHome is often used with ESP32 development boards. Support for the RP2040 platform, the chip in the popular Raspberry Pi Pico W, is still relatively new (introduced in ESPHome 2022.11). As a result, you may encounter some issues, some things are not clearly documented, and there aren't that many ESPHome example configurations using the RP2040. In this article, I'll share my findings after exploring ESPHome's RP2040 support.

Use the dashboard, not the wizard

A first issue you may encounter is that ESPHome's command-line wizard doesn't support the RP2040 platform. The wizard is typically used to create a new ESPHome project by guiding you through a series of steps and generating a default YAML file. However, as of ESPHome 2023.7.0, the wizard only accepts ESP32 or ESP8266 as a platform:

$ esphome wizard test.yaml
Hi there!
I'm the wizard of ESPHome :)
And I'm here to help you get started with ESPHome.
In 4 steps I'm going to guide you through creating a basic configuration file for your custom ESP8266/ESP32 firmware. Yay!



============= STEP 1 =============
    _____ ____  _____  ______
   / ____/ __ \|  __ \|  ____|
  | |   | |  | | |__) | |__
  | |   | |  | |  _  /|  __|
  | |___| |__| | | \ \| |____
   \_____\____/|_|  \_\______|

===================================
First up, please choose a name for your node.
It should be a unique name that can be used to identify the device later.
For example, I like calling the node in my living room livingroom.

(name): test
Great! Your node is now called "test".


============= STEP 2 =============
      ______  _____ _____
     |  ____|/ ____|  __ \\
     | |__  | (___ | |__) |
     |  __|  \___ \|  ___/
     | |____ ____) | |
     |______|_____/|_|

===================================
Now I'd like to know what microcontroller you're using so that I can compile firmwares for it.
Are you using an ESP32 or ESP8266 platform? (Choose ESP8266 for Sonoff devices)

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266): RP2040
Unfortunately, I can't find an espressif microcontroller called "RP2040". Please try again.

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266):

Fortunately, this limitation doesn't mean that you have to configure your Raspberry Pi Pico W from scratch. Instead, you can use the ESPHome dashboard.

To start the ESPHome dashboard and specify the directory where your ESPHome configuration files are located, run the following command: [2]

$ esphome dashboard config/

This will start a web server on http://0.0.0.0:6052. You can access this URL from your web browser. If you already have ESPHome devices on your network, the dashboard will automatically discover them.

Next, click on New device at the bottom right corner, and then on Continue. Give your device a name and enter the SSID and password for the Wi-Fi network that you want your device to connect to. Click on Next and choose your device type. For an ESP32 or ESP8266, you first need to choose the platform and then the specific board. For the RP2040 platform, the dashboard (as of ESPHome 2023.7.0) only allows you to choose Raspberry Pi Pico W, and also Raspberry Pi Pico if you uncheck Use recommended settings. You can always manually change the board later. For now, let's assume you want to install ESPHome on a Raspberry Pi Pico W. After choosing the platform, the dashboard creates a minimal configuration and shows an encryption key that you can use to allow the ESPHome device to communicate with Home Assistant, the popular open-source home automation gateway developed by the same team behind ESPHome. Finally, click on Install.

ESPHome offers several installation methods, but not all of them are supported by every device. Since there's no ESPHome firmware running on your device yet, the first method (over Wi-Fi) is not yet possible. Plug into the computer running ESPHome Dashboard isn't available either, for the same reason. However, you can always choose Manual download. This gives you instructions on how to accomplish the installation. For the Raspberry Pi Pico W, you need to disconnect the board from USB, hold down the BOOTSEL button while reconnecting the board, and then release the button. This will cause a USB drive named RPI-RP2 to appear in your file manager. In the ESPHome dashboard, click on Download project and then drag the .uf2 file to the USB drive. Once the drive disappears, the board runs your ESPHome firmware, and you can click on Close.

Blinking the built-in LED on the Raspberry Pi Pico W

On the Raspberry Pi Pico, the built-in LED is connected to GPIO25. However, on the Raspberry Pi Pico W, the built-in LED is connected to the Wi-Fi chip, the Infineon CYW43439. To blink the LED on the Raspberry Pi Pico W, you need to use GPIO32. [3] For example, edit your configuration file by clicking on Edit in the box representing your device, and add this YAML configuration:

output:
  - platform: gpio
    pin: 32
    id: led

interval:
  - interval: 1000ms
    then:
      - output.turn_on: led
      - delay: 500ms
      - output.turn_off: led

This configuration adds an output component with the gpio platform, assigning it to GPIO pin 32, which corresponds to the built-in LED of the Raspberry Pi Pico W. Additionally, an interval component is defined to trigger the LED to turn on, wait for 500 ms, and then turn off every 1000ms.

After saving the file (in the web editor at the top right), click on Install. Choose your installation method. Since your Raspberry Pi Pico W is already running ESPHome and connected to your Wi-Fi network, you can choose Wirelessly as the installation method. The board doesn't even need to be connected to your computer's USB port anymore. Your YAML configuration is now transformed into C++ code and compiled. If you see the message INFO Successfully compiled program., the dashboard will upload the new firmware. Once the board reboots, the LED starts blinking.

Using PWM output

If you want to send a PWM (pulse-width modulation) signal to a GPIO pin on the RP2040, you can use the output component with the rp2040_pwm platform (not yet documented on ESPHome's web site). Here's an example how you can make a dimmable LED connected to GPIO15: [5]

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    name: "Dimmable LED"
    output: led

On a breadboard, connect the anode (the longer leg) of an LED to the Pico W's GPIO15 pin. Then connect the cathode (the shorter leg), via a 220 Ω resistor, to GND. The circuit should look like this:

/images/picow-led-gpio15_bb.png

After uploading this firmware to your board, you can control the LED's brightness using Home Assistant's dashboard:

/images/ha-dimmable-led.png

You can also use a PWM output to create various light effects. This is how you define a slow pulse effect that continuously pulses the LED:

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    output: led
    id: pulsating_led
    effects:
      - pulse:
          name: "Slow pulse"
          transition_length: 2s
          update_interval: 2s

To start this effect, add the following automation to the on_boot section of the esphome core configuration:

esphome:
  name: raspberry-pi-pico-w
  friendly_name: Raspberry Pi Pico W
  on_boot:
    then:
      - light.turn_on:
          id: pulsating_led
          effect: "Slow pulse"

After uploading the firmware to the board, the LED will start slowly pulsing.

Only use I2C0

When using I²C devices with the RP2040, it's important to note that ESPHome's I²C component for the RP2040 only works for the i2c0 bus. To determine which pins to use for I²C, always refer to the Raspberry Pi Pico W's pinout:

/images/picow-pinout.svg

Only the pins defined as I2C0 SDA and I2C0 SCL can be used for I²C communication. This means you can use the following pin combinations for SDA/SCL: GP0/GP1, GP4/GP5, GP8/GP9, GP12/GP13, GP16/GP17, or GP20/GP21.

Since the GP20/GP21 pins on the Raspberry Pi Pico W are dedicated to I2C0 SDA/I2C0 SCL and do not have any alternative functions, I like to use these pins for the I²C bus. Here's an example configuration to read the temperature, pressure, and humidity from a BME280 sensor board:

i2c:
  sda: 20
  scl: 21

sensor:
  - platform: bme280
    temperature:
      name: "BME280 Temperature"
    pressure:
      name: "BME280 Pressure"
    humidity:
      name: "BME280 Humidity"
    address: 0x77

For some BME280 boards, you need to specify an alternative address, 0x76.

Connect the pins as shown in the following circuit diagram for the BME280:

/images/picow-bme280_bb.png

Driving addressable LED strips

Traditionally, ESPHome supported adressable LED strips using the NeoPixelBus and FastLED platforms for the Light component. However, these platforms don't work with the RP2040. Instead, you can use the RP2040 PIO LED Strip platform, which uses the RP2040 PIO (Programmable Input Output) peripheral to control various addressable LED strips.

For example, an ESPHome configuration to drive a LED strip of eight WS2812B LEDs looks like this:

light:
  - platform: rp2040_pio_led_strip
    name: led_strip
    id: led_strip
    pin: 13
    num_leds: 8
    pio: 0
    rgb_order: GRB
    chipset: WS2812B

Connect the LED strip's DIN to the Pico W's GPIO13, GND to GND, and the LED strip's power pin to VBUS (3V3 isn't enough voltage, while on VBUS there's 5V from the USB connector).

You can now add various light effects to your configuration.

Other RP2040 microcontroller boards

If you look at the configuration file generated by the ESPHome dashboard, by clicking on Edit in the box representing your device, you'll notice the following section for the platform configuration:

rp2040:
  board: rpipicow
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

Since ESPHome uses PlatformIO under the hood, it depends on PlatformIO's support for microcontroller platforms. However, this doesn't support the Raspberry Pi Pico W yet. Max Gerhardt forked PlatformIO's repository and added support for the Raspberry Pi Pico W, as well as other RP2040-based boards.

Initially, I thought that I could just run ESPHome on RP2040 microcontroller boards other than the Raspberry Pi Pico W by specifying a different board name. To test this, I created a new device in the ESPHome dashboard, chose Raspberry Pi Pico W as the device type, and then pressed Skip to not install the firmware immediately to the board. I clicked on Edit in the box representing my device, and changed the board definition rpipicow to another supported board in Max Gerhardt's fork of the RP2040 development platform for PlatformIO, the Arduino Nano RP2040 Connect (arduino_nano_connect). After saving the configuration, I clicked on Install at the top right corner.

However, when I chose Manual download to download the .uf2 file, the image preparation failed, and I encountered several "undefined reference" errors related to the Wi-Fi chip. I should've expected this actually, as the Arduino Nano RP2040 Connect uses the u-blox NINA-W102 instead of the Raspberry Pi Pico W's Infineon CYW43439 for Wi-Fi. The same holds for the Challenger RP2040 WiFi (challenger_2040_wifi), which uses the ESP8285 for Wi-Fi. It seems that additional work on the ESPHome side is required to support these other boards. I don't know any RP2040 boards from other manufacturers than Raspberry Pi using the CYW43439. If they exist, I suspect they should work with ESPHome.

You can still use non-Wi-Fi RP2040 boards this way. For example, here's a configuration for the Seeed Studio XIAO RP2040 (seeed_xiao_rp2040):

esphome:
  name: xiao-rp2040
  friendly_name: Xiao RP2040
  on_boot:
    then:
      - output.turn_on:
          id: led_rgb_enable
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: seeed_xiao_rp2040
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: gpio
    pin: 11
    id: led_rgb_enable

light:
  - platform: rp2040_pio_led_strip
    id: led_rgb
    pin: 12
    num_leds: 1
    pio: 0
    rgb_order: GRB
    chipset: WS2812
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has a WS2812 RGB LED connected to GPIO12, so you can address it using the rp2040_pio_led_strip platform with num_leds set to 1. The light component also defines a light effect with random colors. But before you can use the RGB LED on this board, you need to set GPIO11 high, so that's what the output component is for. On boot, we first enable the LED, and then start the light effect.

Even if a board isn't supported yet by Max Gerhardt's repository, you can still use it with ESPHome by specifying the board as rpipico (or rpipicow if you find a board with the CYW43439 Wi-Fi chip). This will probably work if you're not doing anything exotic. As an example, this is a configuration for Pimoroni's Tiny 2040 board:

esphome:
  name: tiny-2040
  friendly_name: Tiny 2040
  on_boot:
    then:
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: rpipico
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: rp2040_pwm
    id: led_red
    pin: 18
    inverted: True
  - platform: rp2040_pwm
    id: led_green
    pin: 19
    inverted: True
  - platform: rp2040_pwm
    id: led_blue
    pin: 20
    inverted: True

light:
  - platform: rgb
    id: led_rgb
    red: led_red
    green: led_green
    blue: led_blue
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has an RGB LED, which the R, G, and B components connected active low (that's why each output definition has inverted: True) to GPIO18, GPIO19, and GPIO20, respectively. This configuration creates a PWM output for each color component, combines them into one RGB light, and starts a light effect with random colors on boot.

Conclusion

In conclusion, the support for the RP2040 platform in ESPHome is not as mature as the support for ESP32 or ESP8266. It lacks the same level of testing and comprehensive documentation. There are several issues reported, and certain functionalities such as MQTT and Bluetooth Low Energy are not yet supported on the RP2040.

However, for a lot of tasks, ESPHome on the Raspberry Pi Pico W is still quite usable. If you have a spare Raspberry Pi Pico W lying around, give it a try! I'm also hopeful that support for additional RP2040 boards with Wi-Fi will be added in the near future.

When BlueZ has connected once to a BLE device, it uses the Device Name characteristic for the device name

The title of this article may seem trivial, but it represents the conclusion I reached after being astonished for a long time while debugging some issues related to decoding Bluetooth Low Energy (BLE) advertisements that rely on the device name. Like many solutions, the actual solution seemed simple in hindsight.

For Theengs Decoder I wrote decoders for two of Shelly's devices: the ShellyBLU Button1 and the ShellyBLU Door/Window sensor. Both devices broadcast BLE advertisements using the BTHome v2 format. Before we can decode these advertisements, we first need to identify the device that is sending a specific advertisement.

In Theengs Decoder, device detection is achieved using a model condition. The Button1 model condition is as follows:

"condition":["servicedata", "=", 14, "index", 0, "40", "|", "servicedata", "=", 14, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBBT-002C"],

This means: the service data (an array of bytes) consists of 14 hexadecimal characters and begins with the byte 0x40 (the first firmware versions) or 0x44 (the latest firmware). Additionally, the service data UUID is fcd2 and the name starts with SBBT-002C.

The Door/Window model condition is similar:

"condition":["servicedata", "=", 28, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBDW-002C"],

Here, the service data consists of 28 hexadecimal characters and starts with the byte 0x44. The service data UUID is fcd2 and the name starts with SBDW-002C.

Therefore, when decoding BLE advertisements, Theengs Decoder tries all model conditions for the supported devices. If any of the conditions is true, the properties defined in the associated decoder are extracted from the advertised data.

After writing these decoders, I tested them with Theengs Gateway on my Linux laptop. Theengs Gateway uses the cross-platform Python library Bleak, which on Linux uses BlueZ, for receiving BLE advertisements. It then decodes them using Theengs Decoder and publishes the decoded properties (in this case a button push or the state of the contact sensor) to an MQTT broker. The decoders worked.

It's important to know that Theengs Decoder purely works on BLE advertisements, which are broadcasted. This is the most basic way to communicate between BLE devices. All information is extracted from these advertisements: manufacturer-specific data, service data, and the local name. [1]

However, when I later experimented with additional functionality of both devices, I connected to them. A couple of days later, I noticed that the decoders no longer worked. When I looked into it, I saw that Theengs Gateway had detected the devices with different names: Shelly Blue Button 1 and Shelly BLE DW. Naturally, the previous model conditions no longer applied, because they were checking for the names SBBT-002C and SBDW-002C, respectively. Interestingly, the same version of Theengs Gateway running on a Raspberry Pi still managed to detect and decode both devices correctly, because it recognized them by their original names. I didn't understand this difference.

It took me a while before I remembered that I had previously connected to both devices on my laptop. So then I started exploring their GATT characteristics. Each BLE device that allows connections has a Generic Access Profile (GAP) service with mandatory Device Name and Appearance characteristics. I read the Device Name characteristic from both devices, and to my surprise they were identified as Shelly Blue Button 1 and Shelly BLE DW.

So that's when I started to connect the dots. Once I had connected to both devices from my laptop, BlueZ seemed to remember their names based on the Device Name characteristic, even after disconnecting. Apparently, when BlueZ later encounters advertisements from the same Bluetooth address, it doesn't use the advertised name (Complete Local Name or Shortened Local Name data type) to identify this device. Instead, it relies on the name it had previously stored from the Device Name characteristic. As a result, when Bleak requests the local name from BlueZ, it receives the device's Device Name characteristic instead of the advertised local name. Consequently, Theengs Gateway sent the 'wrong' name to Theengs Decoder, which caused the devices to remain undetected.

You can check this with bluetoothctl:

$ bluetoothctl
[bluetooth]# info 5C:C7:C1:XX:XX:XX
Device 5C:C7:C1:XX:XX:XX (public)
        Name: Shelly Blue Button 1
        Alias: Shelly Blue Button 1
        Appearance: 0x8001
        Paired: no
        Trusted: yes
        Blocked: no
        Connected: no
        LegacyPairing: no
        UUID: Generic Access Profile    (00001800-0000-1000-8000-00805f9b34fb)
        UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
        UUID: Device Information        (0000180a-0000-1000-8000-00805f9b34fb)
        UUID: Vendor specific           (1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0)
        UUID: Vendor specific           (de8a5aac-a99b-c315-0c80-60d4cbb51225)
        ManufacturerData Key: 0x0ba9
        ManufacturerData Value:
  01 09 00 0b 01 00 0a aa bb cc c1 c7 5c           ............\
        ServiceData Key: 0000fcd2-0000-1000-8000-00805f9b34fb
        ServiceData Value:
  40 00 19 01 5a 3a 01                             @...Z:.
        RSSI: -68
        AdvertisingFlags:
  06                                               .

Fortunately, once I had figured this out, the solution was quite simple: BlueZ needed to forget the devices. You can remove a device from BlueZ's cache with the following command:

$ bluetoothctl remove 3C:2E:F5:XX:XX:XX

Alternatively, you can manually delete the directory where BlueZ stores device information. First, find your Bluetooth adapter's address:

$ hciconfig
hci0: Type: Primary  Bus: USB
      BD Address: 9C:FC:E8:XX:XX:XX  ACL MTU: 1021:4  SCO MTU: 96:6
      UP RUNNING
      RX bytes:1203288 acl:31 sco:0 events:34298 errors:0
      TX bytes:898242 acl:29 sco:0 commands:11700 errors:0

Then delete the following directory, based on the adapter address and the device address:

$ sudo rm -rf /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX

If you're curious about the cached information, first have a look at the culprit before removing the directory:

$ sudo cat /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX/info
[General]
Name=Shelly BLE DW
AddressType=public
SupportedTechnologies=LE;
Trusted=true
Blocked=false
Services=00001800-0000-1000-8000-00805f9b34fb;00001801-0000-1000-8000-00805f9b34fb;0000180a-0000-1000-8000-00805f9b34fb;1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0;de8a5aac-a99b-c315-0c80-60d4cbb51225;
Appearance=0x8001

It's this value of the Name field that BlueZ returns to Bleak, and subsequently to Theengs Gateway and Theengs Decoder, leading to the wrong detection. After removing the device from BlueZ's cache, the advertised local name was detected correctly, and the decoders worked again.

So now I'm wondering: is it possible to get the real local name advertised by a device? Bleak uses the Name property of a device in its BlueZ backend for scanning. Does BlueZ expose the local name, and could Bleak be adapted to use this instead? I couldn't find the answer to this question.

Zephyr's growing hardware support includes popular microcontroller boards for makers

Zephyr, an open-source real-time operating system (RTOS) launched by the Linux Foundation in 2016, has made lots of progress seven years after its announcement, and it now has an active ecosystem surrounding it. It's used in Internet of Things (IoT) sensors, Bluetooth trackers, heart rate monitors, smartwatches, and embedded controllers. A few months ago I wrote an overview article for LWN.net about the project, Zephyr: a modular OS for resource-constrained devices.

The Zephyr RTOS is used in a lot of commercial products as well as some open-source projects. Developers are also continuously adding support for new development boards. Currently, Zephyr supports more than 450 boards from various architectures.

While most of these boards are the typical developer boards used by professional developers, recent years have seen an uptick in support for more boards popular with hobbyists. Here's a list of some supported boards that will surely ring a bell if you're active in the maker community:

So what can you do with Zephyr? An interesting use case for the operating system is developing Bluetooth Low Energy applications running on one of the above boards. A previous blog article explaining Zephyr's iBeacon example code gives you an idea about how such code looks like. If this piques your curiosity, consider delving deeper into the subject by reading my book Develop your own Bluetooth Low Energy Applications for Raspberry Pi, ESP32 and nRF52 with Python, Arduino and Zephyr, which uses Nordic Semiconductor's nRF52840 Dongle and nRF52840 Development Kit. But Zephyr supports a plethora of communication protocols, including Thread, a low-power IPv6-based wireless mesh-networking technology for home-automation applications, and other protocols listed in the aforementioned LWN.net article.

Even if you're not up to the task of developing applications in the C programming language, Zephyr can be interesting for you. If you prefer Python, you've surely heard about MicroPython, a programming language that implements a sizable subset of Python that can run on microcontrollers. You can read more about it in my recent LWN.net article MicroPython 1.20: Python for microcontrollers. MicroPython offers firmware downloads for more than 150 microcontroller boards, but it also has been ported to Zephyr.

In MicroPython 1.20, this port is based on Zephyr 3.1.0, which was released in June 2022; the current Zephyr release is 3.4.0. A Zephyr development environment can be used to build MicroPython for every target board supported by Zephyr, although not all have been tested. It also gives MicroPython code access to Zephyr's uniform sensor API using the zsensor module. More information can be found in MicroPython's documentation about the Zephyr port.

Thanks to Zephyr's broad hardware support, the Zephyr port allows MicroPython to run on the BBC micro:bit v2, whereas only a direct MicroPython port for the BBC micro:bit v1 exists. Other interesting boards from the above list that don't have a MicroPython firmware download but that are supported thanks to the Zephyr port are the Arduino Nano 33 IoT, the PineTime, and the RuuviTag. The process of setting up the development environment, building the Zephyr port for the target board, and then flashing the firmware to the board is explained in the port's README file.

How to use the await keyword in the Python REPL without asyncio.run()

The Python REPL (read-eval-print loop), which is Python's interactive interpreter, is a great way for quickly testing simple Python commands. I use it quite often as a powerful command-line calculator, but also for exploring new Python libraries.

Many interesting Python libraries use asynchronous I/O with asyncio. This means that, instead of just calling a function directly, you have to await a coroutine. However, if you try this in the REPL, you encounter the following error message:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>>

This is expected, because this line of code would never run in a Python script either. You would need to define a top-level coroutine with async def that calls one or more coroutines with await, and run the top-level coroutine with asyncio.run(). The canonical "Hello world" example from the Python documentation of coroutines looks something like this:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> async def main():
...     print("Hello")
...     await asyncio.sleep(5)
...     print("world")
...
>>> asyncio.run(main())
Hello
world
>>>

There's a five-second delay between the output of "Hello" and "world".

This seems a bit cumbersome. If you just want to call the asyncio.sleep() coroutine, you can simplify this to:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.run(asyncio.sleep(5))
>>>

This is already better! However, it still means that every time you want to call a coroutine, you need to remember to wrap it inside an asyncio.run() call.

Fortunately, Python 3.8 introduced a top-level await if you run the asyncio module as a script:

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
>>>

The REPL helpfully mentions Use "await" directly instead of "asyncio.run()". before importing the asyncio module. [1] Then you can simply type await asyncio.sleep(5) without having to call asyncio.run().

Although this might not seem like much of an improvement in this particular case, when using asynchronous libraries in a Python REPL, having to add asyncio.run() for every coroutine call quickly becomes tedious.

For example, I can now easily request the Bluetooth adapters on my system (using Home Assistant's bluetooth-adapters package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bluetooth_adapters import get_adapters
>>> adapters = get_adapters()
>>> await adapters.refresh()
>>> adapters.adapters
{'hci0': {'address': '9C:FC:E8:XX:XX:XX', 'sw_version': 'tux', 'hw_version': 'usb:v1D6Bp0246d0540', 'passive_scan': True, 'manufacturer': 'Intel Corporate', 'product': '0029', 'vendor_id': '8087', 'product_id': '0029'}}

Or, I can conduct a quick scan for Bluetooth Low Energy (BLE) devices in the vicinity (using the Bleak package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bleak import BleakScanner
>>> devices = await BleakScanner.discover()
>>> [device.name for device in devices]
['Qingping Alarm Clock', 'abeacon_AC7D', 'TP358 (52C6)', '1B-0F-09-4F-89-F3', 'ThermoBeacon', 'F9-DA-D2-0D-62-24', '6A-C8-79-F4-E1-E5', 'Qingping BT Clock Lite', 'LYWSD02', 'Ruuvi 7E0E', 'TY', 'TP393 (2A3D)', 'Flower care', '52-9E-F1-64-DB-DF']

Give this top-level await approach a try with some of your favorite asynchronous Python libraries. You'll definitely become more productive in the REPL.

Creating terminal user interfaces in Python with Textual

Last year, I began using Textual to develop HumBLE Explorer, a cross-platform, command-line and human-friendly Bluetooth Low Energy scanner. Textual caught my attention because it promised to be a rapid application development framework for Python terminal user interface (TUI) applications. Thanks to Textual, I was able to create an application like this, including scroll bars, switches and tables:

/images/humble-explorer-light.png

One of Textual's key features is that it's inspired by web development practices. This allows for a clear separation of design and code using CSS files (in the Textual CSS dialect), making the framework both developer-friendly and highly customizable. It also has reactive attributes, as well as a growing library of widgets. Additionally, the framework provides useful tools for debugging and live editing of CSS files during development.

If you want to have an idea about Textual's capabilities, install it from PyPI and run the demo:

pip install textual
python -m textual

In addition, one of Textual's developers maintains a list of Textual-based applications.

As a relatively new project (started in 2021), Textual still experiences occasional breaking changes in new releases. However, the developers are easily accessible for support on their Discord server and provide regular blog updates. Moreover, Textual has excellent documentation. I plan to use Textual again for new terminal user interfaces in Python.

For some more background, read my article Textual: a framework for terminal user interfaces on LWN.

Creating a diagnostics module for Python software

For Theengs Gateway we regularly got bug reports that were difficult to debug, as they depend on the operating system, Python version, application configuration, and Bluetooth adapter. After some back and forth we got an idea of the user's environment and started to identify the issue (or not). Then I discovered that Textual has a convenient solution for this: a diagnose command that prints information about the Textual library and its environment to help diagnose problems.

I borrowed this code from Textual and adapted it for Theengs Gateway's usage. So now we simply ask the user to run this command and include its output in the issue description on GitHub. Theengs Gateway's diagnose module looks like this:

theengs_gateway/diagnose.py (Source)

import asyncio
import json
import os
import platform
import re
import sys

from importlib_metadata import PackageNotFoundError, version

_conf_path = os.path.expanduser("~") + "/theengsgw.conf"
_ADDR_RE = re.compile(r"^(([0-9A-F]{2}:){3})([0-9A-F]{2}:){2}[0-9A-F]{2}$")


def _anonymize_strings(fields, config) -> None:
    for field in fields:
        if field in config:
            config[field] = "***"


def _anonymize_address(address) -> str:
    addr_parts = _ADDR_RE.match(address)
    if addr_parts:
        return f"{addr_parts.group(1)}XX:XX:XX"
    else:
        return "INVALID ADDRESS"


def _anonymize_addresses(field, config) -> None:
    try:
        config[field] = [
            _anonymize_address(address) for address in config[field]
        ]
    except KeyError:
        pass


# This function is taken from Textual
def _section(title, values) -> None:
    """Print a collection of named values within a titled section.
    Args:
        title: The title for the section.
        values: The values to print out.
    """
    max_name = max(map(len, values.keys()))
    max_value = max(map(len, [str(value) for value in values.values()]))
    print(f"## {title}")
    print()
    print(f"| {'Name':{max_name}} | {'Value':{max_value}} |")
    print(f"|-{'-' * max_name}-|-{'-'*max_value}-|")
    for name, value in values.items():
        print(f"| {name:{max_name}} | {str(value):{max_value}} |")
    print()


def _versions() -> None:
    """Print useful version numbers."""
    try:
        packages = {
            "Theengs Gateway": version("TheengsGateway"),
            "Theengs Decoder": version("TheengsDecoder"),
            "Bleak": version("bleak"),
            "Bluetooth Clocks": version("bluetooth-clocks"),
            "Bluetooth Numbers": version("bluetooth-numbers"),
            "Paho MQTT": version("paho-mqtt"),
        }
    except PackageNotFoundError as e:
        print(f"Package {e.name} not found. Please install it with:")
        print()
        print(f"    pip install {e.name}")
        print()

    if sys.version_info[:2] >= (3, 9):
        try:
            packages["Bluetooth Adapters"] = version("bluetooth-adapters")
        except PackageNotFoundError as e:
            print(f"Package {e.name} not found. Please install it with:")
            print()
            print(f"    pip install {e.name}")
            print()

    _section("Package Versions", packages)


def _python() -> None:
    """Print information about Python."""
    _section(
        "Python",
        {
            "Version": platform.python_version(),
            "Implementation": platform.python_implementation(),
            "Compiler": platform.python_compiler(),
            "Executable": sys.executable,
        },
    )


def _os() -> None:
    os_parameters = {
        "System": platform.system(),
        "Release": platform.release(),
        "Version": platform.version(),
        "Machine type": platform.machine(),
    }
    if platform.system() == "Linux" and sys.version_info[:2] >= (3, 10):
        os_parameters["Distribution"] = platform.freedesktop_os_release()[
            "PRETTY_NAME"
        ]

    _section("Operating System", os_parameters)


def _config() -> None:
    print("## Configuration")
    print()
    try:
        with open(_conf_path, encoding="utf-8") as config_file:
            config = json.load(config_file)
            _anonymize_strings(["user", "pass"], config)
            _anonymize_addresses("time_sync", config)
        print("```")
        print(json.dumps(config, sort_keys=True, indent=4))
        print("```")
        print()
    except FileNotFoundError:
        print(f"Configuration file not found: {_conf_path}")
        print()


async def _adapters() -> None:
    if sys.version_info[:2] >= (3, 9):
        from bluetooth_adapters import get_adapters

        print("## Bluetooth adapters")
        print()
        bluetooth_adapters = get_adapters()
        await bluetooth_adapters.refresh()
        print(f"Default adapter: {bluetooth_adapters.default_adapter}")
        print()

        for adapter, properties in sorted(bluetooth_adapters.adapters.items()):
            properties["address"] = _anonymize_address(properties["address"])
            print("#", end="")
            _section(adapter, properties)


async def diagnostics():
    print("# Theengs Gateway Diagnostics")
    print()
    _versions()
    _python()
    _os()
    _config()
    await _adapters()


if __name__ == "__main__":
    asyncio.run(diagnostics())

When you run this module, it prints a level one Markdown title (# Theengs Gateway Diagnostics) and then calls several functions. Each of these functions prints a level two Markdown title and some diagnostic information.

First, it displays the version numbers of the Python package for Theengs Gateway and some of its dependencies. This helps us immediately identify outdated versions, and we can suggest an update. Next, it shows information about the Python platform and the operating system. These functions are all borrowed from Textual's diagnose module, including the _section helper function to print a collection of named values within a titled section.

Since many Theengs Gateway issues depend on the exact configuration used, I also added a section that displays the contents of the configuration file (a JSON file). However, this configuration file contains some information that shouldn't be shared publicly, such as a username and password for an MQTT broker, or Bluetooth addresses. I could remove these fields in the code, but then we wouldn't know if the bug might be a result of a configuration file lacking one of these fields. So I created a simple function to anonymize specific fields:

def _anonymize_strings(fields, config) -> None:
    for field in fields:
        if field in config:
            config[field] = "***"

Then I can call this function on the configuration to anonymize the user and pass fields:

_anonymize_strings(["user", "pass"], config)

For Bluetooth addresses, I created a similar function. I want to keep the first three bytes of an address, which can point to the device manufacturer and be helpful for debugging purposes. Using a regular expression, I extract these bytes and add XX:XX:XX. This function looks like this:

_ADDR_RE = re.compile(r"^(([0-9A-F]{2}:){3})([0-9A-F]{2}:){2}[0-9A-F]{2}$")


def _anonymize_address(address) -> str:
    addr_parts = _ADDR_RE.match(address)
    if addr_parts:
        return f"{addr_parts.group(1)}XX:XX:XX"
    else:
        return "INVALID ADDRESS"

In the last part of the diagnostic information, where I display the information of the computer's Bluetooth adapters, I can call this function to anonymize the adapter's Bluetooth address:

properties["address"] = _anonymize_address(properties["address"])

Running the python -m TheengsGateway.diagnose command shows output like this:

# Theengs Gateway Diagnostics

## Package Versions

| Name               | Value  |
|--------------------|--------|
| Theengs Gateway    | 3.0    |
| Theengs Decoder    | 1.4.0  |
| Bleak              | 0.20.0 |
| Bluetooth Clocks   | 0.1.0  |
| Bluetooth Numbers  | 1.1.0  |
| Paho MQTT          | 1.6.1  |
| Bluetooth Adapters | 0.15.3 |

## Python

| Name           | Value           |
|----------------|-----------------|
| Version        | 3.10.6          |
| Implementation | CPython         |
| Compiler       | GCC 11.3.0      |
| Executable     | /usr/bin/python |

## Operating System

| Name         | Value                                               |
|--------------|-----------------------------------------------------|
| System       | Linux                                               |
| Release      | 6.2.0-10005-tuxedo                                  |
| Version      | #5 SMP PREEMPT_DYNAMIC Wed Mar 22 12:42:40 UTC 2023 |
| Machine type | x86_64                                              |
| Distribution | Ubuntu 22.04.1 LTS                                  |

## Configuration

```
{
    "adapter": "hci0",
    "ble_scan_time": 1000,
    "ble_time_between_scans": 5,
    "discovery": 1,
    "discovery_device_name": "TheengsGateway",
    "discovery_filter": [
        "IBEACON",
        "GAEN",
        "MS-CDP"
    ],
    "discovery_topic": "homeassistant/sensor",
    "hass_discovery": 1,
    "host": "rhasspy",
    "log_level": "DEBUG",
    "lwt_topic": "home/TheengsGateway/LWT",
    "pass": "***",
    "port": 1883,
    "presence": 0,
    "presence_topic": "home/TheengsGateway/presence",
    "publish_advdata": 1,
    "publish_all": 1,
    "publish_topic": "home/TheengsGateway/BTtoMQTT",
    "scanning_mode": "active",
    "subscribe_topic": "home/+/BTtoMQTT/undecoded",
    "time_format": 1,
    "time_sync": [
        "58:2D:34:XX:XX:XX",
        "E7:2E:00:XX:XX:XX",
        "BC:C7:DA:XX:XX:XX",
        "10:76:36:XX:XX:XX"
    ],
    "user": "***"
}
```

## Bluetooth adapters

Default adapter: hci0

### hci0

| Name         | Value               |
|--------------|---------------------|
| address      | 9C:FC:E8:XX:XX:XX   |
| sw_version   | tux                 |
| hw_version   | usb:v1D6Bp0246d0540 |
| passive_scan | True                |
| manufacturer | Intel Corporate     |
| product      | 0029                |
| vendor_id    | 8087                |
| product_id   | 0029                |

### hci1

| Name         | Value                   |
|--------------|-------------------------|
| address      | 00:01:95:XX:XX:XX       |
| sw_version   | tux #2                  |
| hw_version   | usb:v1D6Bp0246d0540     |
| passive_scan | True                    |
| manufacturer | Sena Technologies, Inc. |
| product      | 0001                    |
| vendor_id    | 0a12                    |
| product_id   | 0001                    |

In the repository's issue template for bug reports, we ask for the output of this command. The user simply has to copy the output, which is already formatted in Markdown syntax. This displays titles, subtitles, and even tables cleanly, providing us the necessary information:

/images/theengs-gateway-diagnose.png

Hopes and promises for open-source voice assistants

Paulus Schoutsen, founder of the open-source home-automation project Home Assistant, declared 2023 as "the year of voice" for the popular platform. The goal of the initiative is to enable users to control their homes through offline voice commands in their own language.

Voice control is a complex and computationally intensive task, which is usually delegated to the cloud. Companies like Google, Amazon and Apple make us believe that we need their cloud-based services to be able to use voice control. Of course, this comes with downsides: users don't have any control over what happens with their voice recordings, posing a significant privacy risk. But, fundamentally, the problem lies even deeper. It just makes no sense for users to have their voices make a long detour through the internet just to turn on a light in the same room.

In the past, projects like Snips and Mycroft attempted offline voice control but faced business challenges. Rhasspy, an independent open-source voice-assistant project that has been active for a few years now, was quite successful among the niche crowd of tinkerers and those who built their own voice assistants around the flexible services the project offered. [1] However, the core of Rhasspy was mainly developed by one person, and the project wasn't backed financially.

Last month, I wrote an article for LWN.net about these three projects: Hopes and promises for open-source voice assistants. I expressed the hope that Rhasspy would finally give us the ability to control our homes with a user-friendly voice assistant that is both privacy respecting and made from open-source software. Rhasspy's developer, Michael Hansen, has been hired by Nabu Casa, the company behind Home Assistant, and they're tightly integrating Rhasspy into their home-automation software.

In the mean time, OpenVoiceOS, a community that forked Mycroft, has published a FAQ about the future of Mycroft. I already alluded to Mycroft's revival in my article, but the plans were still vague at the time. By now, it looks like Mycroft has a real chance to live on in OpenVoiceOS.

Overall, these are exciting times for open-source voice control.

Using low-cost 433.92 MHz wireless sensors

When it comes to home automation, people often end up with devices supporting the Zigbee or Z-Wave protocols, but those devices are relatively expensive. When I was looking for a way to keep an eye on the temperature at home a few years ago, I bought a bunch of cheap temperature and humidity sensors emitting radio signals in the unlicensed ISM (Industrial, Scientific, and Medical) frequency bands instead. Thanks to Benjamin Larsson's rtl_433 and, more recently, NorthernMan54's rtl_433_ESP and Florian Robert's OpenMQTTGateway, I was able to integrate their measurements easily into my home-automation system.

I wrote an article for LWN.net describing these projects and what you need to integrate them into your home-automation system such as Home Assistant: Using low-cost wireless sensors in the unlicensed bands. The article also describes how I'm migrating from a Raspberry Pi-based setup with RTL-SDR dongle running rtl_433 (as described in my home automation book Control Your Home with Raspberry Pi) towards a more distributed setup with multiple LILYGO boards with 433 MHz receiver and running the OpenMQTTGateway firmware around the house. They have less range than the RTL-SDR dongle, but they are cheap and all send their decoded sensor values to the same MQTT broker, so the result is the same as having a single receiver with a longer range.

/images/lilygo-board-openmqttgateway.jpg

The article doesn't go into detail about rtl_433_ESP, but this is a fairly recent and interesting development. While rtl_433 implements signal demodulation in software, rtl_433_ESP uses the transceiver chipset (SX127X on the LILYGO LoRa32 V2.1_1.6.1 433MHz board) to do this. This makes the ESP32 implementation more limited in the signals it can receive, because the transceiver only supports a single modulation scheme at a time. As NorthernMan54 had a lot of devices with OOK (on-off keying) modulation at the time he started the port and not one with FSK (frequency-shift keying) modulation, OOK devices are currently the only ones supported. More specifically, rtl_433_ESP supports rtl_433's Pulse Position Modulation (OOK_PPM), Pulse Width Modulation (OOK_PWM) and Pulse Manchester Zero Bit (OOK_PULSE_MANCHESTER_ZEROBIT) demodulation modules. This limits the available device decoders to 81 of the 234 decoders of rtl_433.

There are other microcontroller implementations of decoders for 433 MHz sensors, but the work NorthernMan54 has done building on rtl_433's code base and making it easy to use it with OpenMQTTGateway is impressive. NorthernMan54 told me he created some scripts that should help automate the port and keep his code synchronized with the roughly annual release cycle of rtl_433.

How to stop brltty from claiming your USB UART interface on Linux

Today I wanted to program an ESP32 development board, the ESP-Pico-Kit v4, but when I connected it to my computer's USB port, the serial connection didn't appear in Linux. Suspecting a hardware issue, I tried another ESP32 board, the ESP32-DevKitC v4, but this didn't appear either, so then I tried another one, a NodeMCU ESP8266 board, which had the same problem. Time to investigate...

The dmesg output looked suspicious:

[14965.786079] usb 1-1: new full-speed USB device number 5 using xhci_hcd
[14965.939902] usb 1-1: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[14965.939915] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[14965.939920] usb 1-1: Product: CP2102 USB to UART Bridge Controller
[14965.939925] usb 1-1: Manufacturer: Silicon Labs
[14965.939929] usb 1-1: SerialNumber: 0001
[14966.023629] usbcore: registered new interface driver usbserial_generic
[14966.023646] usbserial: USB Serial support registered for generic
[14966.026835] usbcore: registered new interface driver cp210x
[14966.026849] usbserial: USB Serial support registered for cp210x
[14966.026881] cp210x 1-1:1.0: cp210x converter detected
[14966.031460] usb 1-1: cp210x converter now attached to ttyUSB0
[14966.090714] input: PC Speaker as /devices/platform/pcspkr/input/input18
[14966.613388] input: BRLTTY 6.4 Linux Screen Driver Keyboard as /devices/virtual/input/input19
[14966.752131] usb 1-1: usbfs: interface 0 claimed by cp210x while 'brltty' sets config #1
[14966.753382] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
[14966.754671] cp210x 1-1:1.0: device disconnected

So the ESP32 board, with a Silicon Labs, CP2102 USB to UART controller chip, was recognized, and it was attached to the /dev/ttyUSB0 device, as it should normally do. But then suddenly the brltty command intervened and disconnected the serial device.

I looked up what brltty is doing, and apparently this is a system daemon that provides access to the console for a blind person using a braille display. When looking into the contents of the package on my Ubuntu 22.04 system (with dpkg -L brltty), I saw a udev rules file, so I grepped for the product ID of my USB device in the file:

$ grep ea60 /lib/udev/rules.d/85-brltty.rules
ENV{PRODUCT}=="10c4/ea60/*", ATTRS{manufacturer}=="Silicon Labs", ENV{BRLTTY_BRAILLE_DRIVER}="sk", GOTO="brltty_usb_run"

Looking at the context, this file shows:

# Device: 10C4:EA60
# Generic Identifier
# Vendor: Cygnal Integrated Products, Inc.
# Product: CP210x UART Bridge / myAVR mySmartUSB light
# BrailleMemo [Pocket]
# Seika [Braille Display]
ENV{PRODUCT}=="10c4/ea60/*", ATTRS{manufacturer}=="Silicon Labs", ENV{BRLTTY_BRAILLE_DRIVER}="sk", GOTO="brltty_usb_run"

So apparently there's a Braille display with the same CP210x USB to UART controller as a lot of microcontroller development boards have. And because this udev rule claims the interface for the brltty daemon, UART communication with all these development boards isn't possible anymore.

As I'm not using these Braille displays, the fix for me was easy: just find the systemd unit that loads these rules, mask and stop it.

$ systemctl list-units | grep brltty
brltty-udev.service loaded active running Braille Device Support
$ sudo systemctl mask brltty-udev.service
Created symlink /etc/systemd/system/brltty-udev.service → /dev/null.
$ sudo systemctl stop brltty-udev.service

After this, I was able to use the serial interface again on all my development boards.