AlsaMixer's poetic error message from Lewis Carroll's "The Hunting of the Snark"

When I mindlessly unplugged a USB audio card from my laptop while viewing its mixer settings in AlsaMixer, I was surprised by the following poetic error message: [1]

In the midst of the word he was trying to say, In the midst of his laughter and glee, He had softly and suddenly vanished away— For the Snark was a Boojum, you see.

These lines are actually the final verse of Lewis Carroll's whimsical poem The Hunting of the Snark:

In the midst of the word he was trying to say,
  In the midst of his laughter and glee,
He had softly and suddenly vanished away—
  For the Snark was a Boojum, you see.

-- Lewis Carroll, "The Hunting of the Snark"

Initially, I didn't even notice the bright red error message The sound device was unplugged. Press F6 to select another sound card. The poetic lines were enough to convey what had happened and they put a smile on my face.

Please, developers, keep putting a smile on my face when something unexpected happens.

Building wireless sensor networks with OpenThread, CoAP, and Zephyr

Last week my new book has been published, Building Wireless Sensor Networks with OpenThread: Developing CoAP Applications for Thread Networks with Zephyr.

Thread is a protocol for building efficient, secure and scalable wireless mesh networks for the Internet of Things (IoT), based on IPv6. OpenThread is an open-source implementation of the Thread protocol, originally developed by Google. It offers a comprehensive Application Programming Interface (API) that is both operating system and platform agnostic. OpenThread is the industry’s reference implementation and the go-to platform for professional Thread application developers.

This book uses OpenThread in conjunction with Zephyr, an open-source real-time operating system designed for use with resource-constrained devices. This allows you to develop Thread applications that work on various hardware platforms, without the need to delve into low-level details or to learn another API when switching hardware platforms.

With its practical approach, this book not only explains theoretical concepts, but also demonstrates Thread’s network features with the use of practical examples. It explains the code used for a variety of basic Thread applications based on Constrained Application Protocol (CoAP). This includes advanced topics such as service discovery and security. As you work through this book, you’ll build on both your knowledge and skills. By the time you finish the final chapter, you’ll have the confidence to effectively implement Thread-based wireless networks for your own IoT projects.

What is Thread?

Thread is a low-power, wireless, and IPv6-based networking protocol designed specifically for IoT devices. Development of the protocol is managed by the Thread Group, an alliance founded in 2015. This is a consortium of major industry players, including Google, Apple, Amazon, Qualcomm, Silicon Labs, and Nordic Semiconductor.

You can download the Thread specification for free, although registration is required. The version history is as follows:

Versions of the Thread specification

Version

Release year

Remarks

Thread 1.0

2015

Never implemented in commercial products

Thread 1.1

2017

  • The ability to automatically move to a clear channel on detecting interference (channel agility)

  • The ability to reset a master key and drive new rotating keys in the network

Thread 1.2

2019

  • Enhancements to scalability and energy efficiency

  • Support for large-scale networking applications, including the ability to integrate multiple Thread networks into one singular Thread domain

Thread 1.3

2023

  • Enhancements to scalability, reliability, and robustness of Thread networks

  • Standardization of Thread Border Routers, Matter support, and firmware upgrades for Thread devices

All versions of the Thread specification maintain backward compatibility.

Thread’s architecture

Thread employs a mesh networking architecture that allows devices to communicate directly with each other, eliminating the need for a central hub or gateway. This architecture offers inherent redundancy, as devices can relay data using multiple paths, ensuring increased reliability in case of any single node failure.

The Thread protocol stack is built upon the widely used IPv6 protocol, which simplifies the integration of Thread networks into existing IP infrastructures. Thread is designed to be lightweight, streamlined and efficient, making it an excellent protocol for IoT devices running in resource-constrained environments.

Before discussing Thread, it’s essential to understand its place in the network protocol environment. Most modern networks are based on the Internet Protocol suite, which uses a four-layer architecture. The layers of the Internet Protocol suite are, from bottom to top:

Link layer

Defines the device’s connection with a local network. Protocols like Ethernet and Wi-Fi operate within this layer. In the more complex Open Systems Interconnection (OSI) model, this layer includes the physical and data link layers.

Network layer

Enables communication across network boundaries, known as routing. The Internet Protocol (IP) operates within this layer and defines IP addresses.

Transport layer

Enables communication between two devices, either on the same network or on different networks with routers in between. The transport layer also defines the concept of a port. User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) operate within this layer.

Application layer

Facilitates communication between applications on the same device or different devices. Protocols like HyperText Transfer Protocol (HTTP), Domain Name System (DNS), Simple Mail Transfer Protocol (SMTP), and Message Queuing Telemetry Transport (MQTT) operate within this layer.

When data is transmitted across the network, the data of each layer is embedded in the layer below it, as shown in this diagram:

/images/internet-protocol-encapsulation.png

Network data is encapsulated in the four layers of the Internet Protocol suite (based on: Colin Burnett, CC BY-SA 3.0)

Thread operates in the network and transport layers. However, it’s important to know what’s going on in all layers.

Network layer: 6LoWPAN and IPv6

Thread’s network layer consists of IPv6 over Low-Power Wireless Access Networks (6LoWPAN). This is an IETF specification described in RFC 4944, "Transmission of IPv6 Packets over IEEE 802.15.4 Networks", with an update for the compression mechanism in RFC 6282, "Compression Format for IPv6 Datagrams over IEEE 802.15.4-Based Networks". 6LoWPAN’s purpose is to allow even the smallest devices with limited processing power and low energy requirements to be part of the Internet of Things.

6LoWPAN essentially enables you to run an IPv6 network over the IEEE 802.15.4 link layer. It acts as an ‘adaptation layer’ between IPv6 and IEEE 802.15.4 and is therefore sometimes considered part of the link layer. This adaptation poses some challenges. For example, IPv6 mandates that all links can handle datagram sizes of at least 1280 bytes, but IEEE 802.15.4, as explained in the previous subsection, limits a frame to 127 bytes. 6LoWPAN solves this by excluding information in the 6LoWPAN header that can already be derived from IEEE 802.15.4 frames and by using local, shortened addresses. If the payload is still too large, it’s divided into fragments.

The link-local IPv6 addresses of 6LoWPAN devices are derived from the IEEE 802.15.4 EUI-64 addresses and their shortened 16-bit addresses. The devices in a 6LoWPAN network can communicate directly with ‘normal’ IPv6 devices in a network if connected via an edge router. This means there is no need for translation via a gateway, in contrast to non-IP networks such as Zigbee or Z-Wave. Communication with non-6LoWPAN networks purely involves forwarding data packets at the network layer.

In this layer, Thread builds upon IEEE 802.15.4 to create an IPv6-based mesh network. In IEEE 802.15.4 only communication between devices that are in immediate radio range is possible, whereas routing allows devices that aren’t in immediate range to communicate.

If you want to delve into more details of Thread’s network layer, consult the Thread Group’s white paper Thread Usage of 6LoWPAN.

Transport layer: UDP

On top of 6LoWPAN and IPv6, Thread’s transport layer employs the User Datagram Protocol (UDP), which is the lesser known alternative to Transmission Control Protocol (TCP).

UDP has the following properties:

Connectionless

Unlike TCP, UDP doesn’t require establishing a connection before transferring data. It sends datagrams (packets) directly without any prior setup.

No error checking and recovery

UDP doesn’t provide built-in error checking, nor does it ensure the delivery of packets. If data is lost or corrupted during transmission, UDP doesn’t attempt to recover or resend it.

Speed

UDP is faster than TCP, as it doesn’t involve the overhead of establishing and maintaining a connection, error checking, or guaranteeing packet delivery.

UDP is ideal for use by Thread for several reasons:

Resource constraints

Thread devices often have limited processing power, memory, and energy. The simplicity and low overhead of UDP make it a better fitting choice for such resource-constrained devices compared to TCP.

Lower latency

UDP’s connectionless and lightweight nature guarantees data transmission with low latency, making it appropriate for wireless sensors.

Thread lacks an application layer

Unlike home automation protocols such as Zigbee, Z-Wave, or Matter, the Thread standard doesn’t define an application layer. A Thread network merely offers network infrastructure that applications can use to communicate. Just as your web browser and email client use TCP/IP over Ethernet or Wi-Fi, home automation devices can use Thread over IEEE 802.15.4 as their network infrastructure.

Several application layers exist that can make use of Thread:

Constrained Application Protocol (CoAP)

A simpler version of HTTP, designed for resource-constrained devices and using UDP instead of TCP.

MQTT for Sensor Networks (MQTT-SN)

A variant of MQTT designed to be used over UDP.

Apple HomeKit

Apple’s application protocol for home automation devices.

Matter

A new home automation standard developed by the Connectivity Standards Alliance (CSA).

Like your home network which simultaneously hosts a lot of application protocols including HTTP, DNS, and SMTP, a Thread network can also run all these application protocols concurrently.

/images/thread-stack.png

The Thread network stack supports multiple application protocols simultaneously.

Throughout this book, I will be using CoAP as an application protocol in a Thread network. Once you have a Thread network set up, you can also use it for Apple HomeKit and Matter devices.

Advantages of Thread

Some of Thread’s key benefits include:

Scalability

Thread’s mesh network architecture allows for large-scale deployment of IoT devices, supporting hundreds of devices within a single network. Thread accommodates up to 32 Routers per network and up to 511 End Devices per Router. In addition, from Thread 1.2, multiple Thread networks can be integrated into one single Thread domain, allowing thousands of devices within a mesh network.

Security

Devices can’t access a Thread network without authorization. Moreover, all communication on a Thread network is encrypted using IEEE 802.15.4 security mechanisms. As a result, an outsider without access to the network credentials can’t read network traffic.

Reliability

Thread networks are robust and support self-healing and self-organizing network properties at various levels, ensuring the network’s resilience against failures. This all happens transparently to the user; messages are automatically routed around any bad node via alternative paths.

For example, an End Device requires a parent Router to communicate with the rest of the network. If communication with this parent Router fails for any reason and it becomes unavailable the End Device will choose another parent Router in its neighborhood, after which communication resumes.

Thread Routers also relay packets from their End Devices to other Routers using the most efficient route they can find. In case of connection problems, the Router will immediately seek an alternate route. Router-Eligible End Devices can also temporarily upgrade their status to Routers if necessary. Each Thread network has a Leader who supervises the Routers and whose role is dynamically selected by the Routers. If the Leader fails, another Router automatically takes over as Leader.

Border Routers have the same built-in resilience. In a Thread network with multiple Border Routers, communication between Thread devices and devices on another IP network (such as a home network) occurs along multiple routes. If one Border Router loses connectivity, communications will be rerouted via the other Border Routers. If your Thread network only has a single Border Router, this will become a single point of failure for communication with the outside network.

Low power consumption

Thread is based on the power-efficient IEEE 802.15.4 link layer. This enables devices to operate on batteries for prolonged periods. Sleepy End Devices will also switch off their radio during idle periods and only wake up periodically to communicate with their parent Router, thereby giving even longer battery life.

Interoperability

As the protocol is built upon IPv6, Thread devices are straightforward to incorporate into existing networks, both in industrial and home infrastructure, and for both local networks and cloud connections. You don’t need a proprietary gateway; every IP-based device is able to communicate with Thread devices, as long as there’s a route between both devices facilitated by a Border Router.

Disadvantages of Thread

In life, they say there’s no such thing as a free lunch and Thread is no exception to this rule. Some of its limitations are:

Limited range

Thread’s emphasis on low power consumption can lead to a restricted wireless range compared with other technologies such as Wi-Fi. Although this can be offset by its mesh architecture, where Routers relay messages, it still means that devices in general can’t be placed too far apart.

Low data rates

Thread supports lower data rates than some rival IoT technologies. This makes Thread unsuitable for applications requiring high throughput.

Complexity

The Thread protocol stack may be more challenging to implement compared to other IoT networking solutions, potentially ramping up development costs and time.

Platforms used in this book

In this book I focus on building wireless sensor networks using the Thread protocol together with the following hardware and software environments:

Nordic Semiconductor’s nRF52840 SoC

A powerful yet energy-efficient and low cost hardware solution for Thread-based IoT devices

OpenThread

An open-source implementation of the Thread protocol, originally developed by Google, making it easy to incorporate Thread into your IoT projects

Zephyr

An open-source real-time operating system designed for resource-constrained devices

Armed with these tools and by using practical examples I will go on to guide you step-by-step through the details of the hardware and software required to build Thread networks and OpenThread-based applications.

Nordic Semiconductor’s nRF52840 SoC

Nordic Semiconductor’s nRF52840 is a SoC built around the 32-bit ARM Cortex-M4 CPU running at 64 MHz. This chip supports Bluetooth Low Energy (BLE), Bluetooth Mesh, Thread, Zigbee, IEEE 802.15.4, ANT and 2.4 GHz proprietary stacks. With its 1 MB flash storage and 256 KB RAM, it offers ample resources for advanced Thread applications.

This SoC is available for developers in the form of Nordic Semiconductor’s user-friendly nRF52840 Dongle. You can power and program this small, low-cost USB dongle via a computer USB port. The documentation lists comprehensive information about the dongle.

/images/nrf52840-dongle.png

Nordic Semiconductor’s nRF52840 Dongle is a low-cost USB dongle ideal for experimenting with Thread. (image source: Nordic Semiconductor)

OpenThread

Thread is basically a networking protocol; in order to work with it you need an implementation of the Thread networking protocol. The industry’s reference implementation, used even by professional Thread device developers, is OpenThread. It’s a BSD-licensed implementation, with development ongoing via its GitHub repository. The license for this software allows for its use in both open-source and proprietary applications.

OpenThread provides an Application Programming Interface (API) that’s operating system and platform agnostic. It has a narrow platform abstraction layer to achieve this, and has a small memory footprint, making it highly portable. In this book, I will be using OpenThread in conjunction with Zephyr, but the same API can be used in other combinations, such as ESP-IDF for Espressif’s Thread SoCs.

/images/openthread-website.png

The BSD-licensed OpenThread project is the industry’s standard implementation of the Thread protocol.

Zephyr

Zephyr is an open-source real-time operating system (RTOS) designed for resource-constrained devices. It incorporates OpenThread as a module, simplifying the process of creating Thread applications based on Zephyr and OpenThread. Because of Zephyr’s hardware abstraction layer, these applications will run on all SoCs supported by Zephyr that have an IEEE 802.15.4 radio facility.

/images/zephyr-website.png

Zephyr is an open-source real-time operating system (RTOS) with excellent support for OpenThread.

How to change the default I²C pins in Zephyr

When working with I²C devices in the Zephyr real-time operating system, a lot is configured behind the scenes using a devicetree, a hierarchical data structure that describes available hardware. It took me some time to figure out how to change the default I²C pins (SDA and SCL). Here are my notes.

I tried this with Zephyr's BME280 Humidity and Pressure Sensor sample. My goal was to flash the sample firmware to Nordic Semiconductor's nRF52840 Dongle with a BME280 breakout board.

What are the default I²C pins?

First I needed to find out the default I²C pins. To check this, I looked at the devicetree source include file for the nRF52840 Dongle's Pin Control:

koan@tux:~$ less ~/zephyrproject/zephyr/boards/arm/nrf52840dongle_nrf52840/nrf52840dongle_nrf52840-pinctrl.dtsi

The relevant configuration for the i2c0 peripheral is as follows:

i2c0_default: i2c0_default {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
        };
};

i2c0_sleep: i2c0_sleep {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
                low-power-enable;
        };
};

This configures pin 0.26 for SDA and pin 0.27 for SCL. My first thought was to simply connect the BME280 to these pins on the nRF52840 Dongle. However, upon examining the board's pinout, I discovered that pin 0.26 is only exposed as a pad on the bottom, and pin 0.27 isn't accessible anywhere:

/images/nrf52840-dongle-pinout.png

How to change the default I²C pins?

Since the default I²C pins weren't suitable on this board, I needed to change them. I decided to connect the BME280 sensor board as follows:

BME280

nRF52840 Dongle

SDA

0.31

SCL

0.29

GND

GND

VCC

VDD

This looks like this on a breadboard:

/images/nrf52840-dongle-bme280.jpg

Next, I copied the sample code so I could change it:

koan@tux:~$ cp -r ~/zephyrproject/zephyr/samples/sensor/bme280/ .
koan@tux:~$ cd bme280/

Then, I added a devicetree overlay file called nrf52840dongle_nrf52840.overlay to the project's boards directory. The content of the overlay file is as follows:

/*
 * Configuration of a BME280 device on an I2C bus.
 *
 * Device address 0x76 is assumed. Your device may have a different
 * address; check your device documentation if unsure.
 */
&pinctrl {
        i2c0_default: i2c0_default {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                };
        };

        i2c0_sleep: i2c0_sleep {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                        low-power-enable;
                };
        };
};

&i2c0 {
    status = "okay";
    bme280@76 {
        compatible = "bosch,bme280";
        reg = <0x76>;
    };
};

The &pinctrl section was copied from the devicetree source include file for the nRF52840 Dongle, with pins 26 and 27 changed to 31 and 29, respectively.

The &i2c0 part defines the bme280 sensor with the I²C address 0x76. We need to add this sensor definition here because the nRF52840 Dongle doesn't have it built in.

So, with this overlay, you can use the BME280 sensor connected to pins 0.31 (SDA) and 0.29 (SCL).

Because the file name of the devicetree overlay is the board name nrf52840dongle_nrf52840 with the extension .overlay, Zephyr's build system picks it up automatically if you're building the project for the nRF52840 Dongle and merges it with the default devicetree of the board. If you're building the project for another board, the overlay will be ignored.

Building the code

Let's build the sample code with this devicetree overlay, adding configuration options to initialize USB at boot and enable Zephyr's shell:

koan@tux:~/bme280$ source ~/zephyrproject/zephyr/zephyr-env.sh
koan@tux:~/bme280$ west build -b nrf52840dongle_nrf52840 . -- -DCONFIG_USB_DEVICE_INITIALIZE_AT_BOOT=y -DCONFIG_SHELL=y

Now let's examine the generated devicetree:

koan@tux:~/bme280$ less build/zephyr/zephyr.dts

This includes the bme280 sensor on the i2c0 bus:

i2c0: i2c@40003000 {
        compatible = "nordic,nrf-twi";
        #address-cells = < 0x1 >;
        #size-cells = < 0x0 >;
        reg = < 0x40003000 0x1000 >;
        clock-frequency = < 0x186a0 >;
        interrupts = < 0x3 0x1 >;
        status = "okay";
        pinctrl-0 = < &i2c0_default >;
        pinctrl-1 = < &i2c0_sleep >;
        pinctrl-names = "default", "sleep";
        bme280@76 {
                compatible = "bosch,bme280";
                reg = < 0x76 >;
        };
};

The Pin Control section shows the I²C pin defaults:

i2c0_default: i2c0_default {
        phandle = < 0x4 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
        };
};
i2c0_sleep: i2c0_sleep {
        phandle = < 0x5 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
                low-power-enable;
        };
};

Here, 1f is of course the hexadecimal representation of 31 and 1d corresponds to 29.

Running the sample

Now that we're confident that the devicetree is correct, let's create a firmware package and flash it to the nRF52840 Dongle:

koan@tux:~/bme280$ nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application build/zephyr/zephyr.hex --application-version 1 bme280.zip

|===============================================================|
|##      ##    ###    ########  ##    ## #### ##    ##  ######  |
|##  ##  ##   ## ##   ##     ## ###   ##  ##  ###   ## ##    ## |
|##  ##  ##  ##   ##  ##     ## ####  ##  ##  ####  ## ##       |
|##  ##  ## ##     ## ########  ## ## ##  ##  ## ## ## ##   ####|
|##  ##  ## ######### ##   ##   ##  ####  ##  ##  #### ##    ## |
|##  ##  ## ##     ## ##    ##  ##   ###  ##  ##   ### ##    ## |
| ###  ###  ##     ## ##     ## ##    ## #### ##    ##  ######  |
|===============================================================|
|You are not providing a signature key, which means the DFU     |
|files will not be signed, and are vulnerable to tampering.     |
|This is only compatible with a signature-less bootloader and is|
|not suitable for production environments.                      |
|===============================================================|

Zip created at bme280.zip
koan@tux:~/bme280$ nrfutil dfu usb-serial -pkg bme280.zip -p /dev/ttyACM0
  [####################################]  100%
Device programmed.

After this, connect to the UART interface over USB:

koan@tux:~$ screen /dev/ttyACM0

You should see that the I²C device has been detected, and the sensor values are being displayed:

[00:00:00.317,932] <dbg> BME280: bme280_chip_init: ID OK
[00:00:00.328,582] <dbg> BME280: bme280_chip_init: "bme280@76" OK
*** Booting Zephyr OS build zephyr-v3.4.0-837-gb4ed6c4300a2 ***
Found device "bme280@76", getting sensor data
temp: 25.850000; press: 101.987800; humidity: 59.248046
temp: 25.840000; press: 101.986144; humidity: 59.187500
temp: 25.850000; press: 101.985195; humidity: 59.176757

Using ESPHome on the Raspberry Pi Pico W and other RP2040 microcontroller boards

ESPHome is an open-source program that allows you to create your own home-automation devices using an ESP32, ESP8266, or RP2040 microcontroller board that you connect to LEDs, sensors, or switches. What sets ESPHome apart from other solutions like Arduino or MicroPython is that you don't need to program. Instead, you define your components and their respective pin connections in a YAML configuration file. ESPHome then generates the necessary C++ code and compiles it into firmware that you can install on the device. [1]

ESPHome is often used with ESP32 development boards. Support for the RP2040 platform, the chip in the popular Raspberry Pi Pico W, is still relatively new (introduced in ESPHome 2022.11). As a result, you may encounter some issues, some things are not clearly documented, and there aren't that many ESPHome example configurations using the RP2040. In this article, I'll share my findings after exploring ESPHome's RP2040 support.

Use the dashboard, not the wizard

A first issue you may encounter is that ESPHome's command-line wizard doesn't support the RP2040 platform. The wizard is typically used to create a new ESPHome project by guiding you through a series of steps and generating a default YAML file. However, as of ESPHome 2023.7.0, the wizard only accepts ESP32 or ESP8266 as a platform:

$ esphome wizard test.yaml
Hi there!
I'm the wizard of ESPHome :)
And I'm here to help you get started with ESPHome.
In 4 steps I'm going to guide you through creating a basic configuration file for your custom ESP8266/ESP32 firmware. Yay!



============= STEP 1 =============
    _____ ____  _____  ______
   / ____/ __ \|  __ \|  ____|
  | |   | |  | | |__) | |__
  | |   | |  | |  _  /|  __|
  | |___| |__| | | \ \| |____
   \_____\____/|_|  \_\______|

===================================
First up, please choose a name for your node.
It should be a unique name that can be used to identify the device later.
For example, I like calling the node in my living room livingroom.

(name): test
Great! Your node is now called "test".


============= STEP 2 =============
      ______  _____ _____
     |  ____|/ ____|  __ \\
     | |__  | (___ | |__) |
     |  __|  \___ \|  ___/
     | |____ ____) | |
     |______|_____/|_|

===================================
Now I'd like to know what microcontroller you're using so that I can compile firmwares for it.
Are you using an ESP32 or ESP8266 platform? (Choose ESP8266 for Sonoff devices)

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266): RP2040
Unfortunately, I can't find an espressif microcontroller called "RP2040". Please try again.

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266):

Fortunately, this limitation doesn't mean that you have to configure your Raspberry Pi Pico W from scratch. Instead, you can use the ESPHome dashboard.

To start the ESPHome dashboard and specify the directory where your ESPHome configuration files are located, run the following command: [2]

$ esphome dashboard config/

This will start a web server on http://0.0.0.0:6052. You can access this URL from your web browser. If you already have ESPHome devices on your network, the dashboard will automatically discover them.

Next, click on New device at the bottom right corner, and then on Continue. Give your device a name and enter the SSID and password for the Wi-Fi network that you want your device to connect to. Click on Next and choose your device type. For an ESP32 or ESP8266, you first need to choose the platform and then the specific board. For the RP2040 platform, the dashboard (as of ESPHome 2023.7.0) only allows you to choose Raspberry Pi Pico W, and also Raspberry Pi Pico if you uncheck Use recommended settings. You can always manually change the board later. For now, let's assume you want to install ESPHome on a Raspberry Pi Pico W. After choosing the platform, the dashboard creates a minimal configuration and shows an encryption key that you can use to allow the ESPHome device to communicate with Home Assistant, the popular open-source home automation gateway developed by the same team behind ESPHome. Finally, click on Install.

ESPHome offers several installation methods, but not all of them are supported by every device. Since there's no ESPHome firmware running on your device yet, the first method (over Wi-Fi) is not yet possible. Plug into the computer running ESPHome Dashboard isn't available either, for the same reason. However, you can always choose Manual download. This gives you instructions on how to accomplish the installation. For the Raspberry Pi Pico W, you need to disconnect the board from USB, hold down the BOOTSEL button while reconnecting the board, and then release the button. This will cause a USB drive named RPI-RP2 to appear in your file manager. In the ESPHome dashboard, click on Download project and then drag the .uf2 file to the USB drive. Once the drive disappears, the board runs your ESPHome firmware, and you can click on Close.

Blinking the built-in LED on the Raspberry Pi Pico W

On the Raspberry Pi Pico, the built-in LED is connected to GPIO25. However, on the Raspberry Pi Pico W, the built-in LED is connected to the Wi-Fi chip, the Infineon CYW43439. To blink the LED on the Raspberry Pi Pico W, you need to use GPIO32. [3] For example, edit your configuration file by clicking on Edit in the box representing your device, and add this YAML configuration:

output:
  - platform: gpio
    pin: 32
    id: led

interval:
  - interval: 1000ms
    then:
      - output.turn_on: led
      - delay: 500ms
      - output.turn_off: led

This configuration adds an output component with the gpio platform, assigning it to GPIO pin 32, which corresponds to the built-in LED of the Raspberry Pi Pico W. Additionally, an interval component is defined to trigger the LED to turn on, wait for 500 ms, and then turn off every 1000ms.

After saving the file (in the web editor at the top right), click on Install. Choose your installation method. Since your Raspberry Pi Pico W is already running ESPHome and connected to your Wi-Fi network, you can choose Wirelessly as the installation method. The board doesn't even need to be connected to your computer's USB port anymore. Your YAML configuration is now transformed into C++ code and compiled. If you see the message INFO Successfully compiled program., the dashboard will upload the new firmware. Once the board reboots, the LED starts blinking.

Using PWM output

If you want to send a PWM (pulse-width modulation) signal to a GPIO pin on the RP2040, you can use the output component with the rp2040_pwm platform (not yet documented on ESPHome's web site). Here's an example how you can make a dimmable LED connected to GPIO15: [5]

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    name: "Dimmable LED"
    output: led

On a breadboard, connect the anode (the longer leg) of an LED to the Pico W's GPIO15 pin. Then connect the cathode (the shorter leg), via a 220 Ω resistor, to GND. The circuit should look like this:

/images/picow-led-gpio15_bb.png

After uploading this firmware to your board, you can control the LED's brightness using Home Assistant's dashboard:

/images/ha-dimmable-led.png

You can also use a PWM output to create various light effects. This is how you define a slow pulse effect that continuously pulses the LED:

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    output: led
    id: pulsating_led
    effects:
      - pulse:
          name: "Slow pulse"
          transition_length: 2s
          update_interval: 2s

To start this effect, add the following automation to the on_boot section of the esphome core configuration:

esphome:
  name: raspberry-pi-pico-w
  friendly_name: Raspberry Pi Pico W
  on_boot:
    then:
      - light.turn_on:
          id: pulsating_led
          effect: "Slow pulse"

After uploading the firmware to the board, the LED will start slowly pulsing.

Only use I2C0

When using I²C devices with the RP2040, it's important to note that ESPHome's I²C component for the RP2040 only works for the i2c0 bus. To determine which pins to use for I²C, always refer to the Raspberry Pi Pico W's pinout:

/images/picow-pinout.svg

Only the pins defined as I2C0 SDA and I2C0 SCL can be used for I²C communication. This means you can use the following pin combinations for SDA/SCL: GP0/GP1, GP4/GP5, GP8/GP9, GP12/GP13, GP16/GP17, or GP20/GP21.

Since the GP20/GP21 pins on the Raspberry Pi Pico W are dedicated to I2C0 SDA/I2C0 SCL and do not have any alternative functions, I like to use these pins for the I²C bus. Here's an example configuration to read the temperature, pressure, and humidity from a BME280 sensor board:

i2c:
  sda: 20
  scl: 21

sensor:
  - platform: bme280
    temperature:
      name: "BME280 Temperature"
    pressure:
      name: "BME280 Pressure"
    humidity:
      name: "BME280 Humidity"
    address: 0x77

For some BME280 boards, you need to specify an alternative address, 0x76.

Connect the pins as shown in the following circuit diagram for the BME280:

/images/picow-bme280_bb.png

Driving addressable LED strips

Traditionally, ESPHome supported adressable LED strips using the NeoPixelBus and FastLED platforms for the Light component. However, these platforms don't work with the RP2040. Instead, you can use the RP2040 PIO LED Strip platform, which uses the RP2040 PIO (Programmable Input Output) peripheral to control various addressable LED strips.

For example, an ESPHome configuration to drive a LED strip of eight WS2812B LEDs looks like this:

light:
  - platform: rp2040_pio_led_strip
    name: led_strip
    id: led_strip
    pin: 13
    num_leds: 8
    pio: 0
    rgb_order: GRB
    chipset: WS2812B

Connect the LED strip's DIN to the Pico W's GPIO13, GND to GND, and the LED strip's power pin to VBUS (3V3 isn't enough voltage, while on VBUS there's 5V from the USB connector).

You can now add various light effects to your configuration.

Other RP2040 microcontroller boards

If you look at the configuration file generated by the ESPHome dashboard, by clicking on Edit in the box representing your device, you'll notice the following section for the platform configuration:

rp2040:
  board: rpipicow
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

Since ESPHome uses PlatformIO under the hood, it depends on PlatformIO's support for microcontroller platforms. However, this doesn't support the Raspberry Pi Pico W yet. Max Gerhardt forked PlatformIO's repository and added support for the Raspberry Pi Pico W, as well as other RP2040-based boards.

Initially, I thought that I could just run ESPHome on RP2040 microcontroller boards other than the Raspberry Pi Pico W by specifying a different board name. To test this, I created a new device in the ESPHome dashboard, chose Raspberry Pi Pico W as the device type, and then pressed Skip to not install the firmware immediately to the board. I clicked on Edit in the box representing my device, and changed the board definition rpipicow to another supported board in Max Gerhardt's fork of the RP2040 development platform for PlatformIO, the Arduino Nano RP2040 Connect (arduino_nano_connect). After saving the configuration, I clicked on Install at the top right corner.

However, when I chose Manual download to download the .uf2 file, the image preparation failed, and I encountered several "undefined reference" errors related to the Wi-Fi chip. I should've expected this actually, as the Arduino Nano RP2040 Connect uses the u-blox NINA-W102 instead of the Raspberry Pi Pico W's Infineon CYW43439 for Wi-Fi. The same holds for the Challenger RP2040 WiFi (challenger_2040_wifi), which uses the ESP8285 for Wi-Fi. It seems that additional work on the ESPHome side is required to support these other boards. I don't know any RP2040 boards from other manufacturers than Raspberry Pi using the CYW43439. If they exist, I suspect they should work with ESPHome.

You can still use non-Wi-Fi RP2040 boards this way. For example, here's a configuration for the Seeed Studio XIAO RP2040 (seeed_xiao_rp2040):

esphome:
  name: xiao-rp2040
  friendly_name: Xiao RP2040
  on_boot:
    then:
      - output.turn_on:
          id: led_rgb_enable
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: seeed_xiao_rp2040
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: gpio
    pin: 11
    id: led_rgb_enable

light:
  - platform: rp2040_pio_led_strip
    id: led_rgb
    pin: 12
    num_leds: 1
    pio: 0
    rgb_order: GRB
    chipset: WS2812
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has a WS2812 RGB LED connected to GPIO12, so you can address it using the rp2040_pio_led_strip platform with num_leds set to 1. The light component also defines a light effect with random colors. But before you can use the RGB LED on this board, you need to set GPIO11 high, so that's what the output component is for. On boot, we first enable the LED, and then start the light effect.

Even if a board isn't supported yet by Max Gerhardt's repository, you can still use it with ESPHome by specifying the board as rpipico (or rpipicow if you find a board with the CYW43439 Wi-Fi chip). This will probably work if you're not doing anything exotic. As an example, this is a configuration for Pimoroni's Tiny 2040 board:

esphome:
  name: tiny-2040
  friendly_name: Tiny 2040
  on_boot:
    then:
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: rpipico
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: rp2040_pwm
    id: led_red
    pin: 18
    inverted: True
  - platform: rp2040_pwm
    id: led_green
    pin: 19
    inverted: True
  - platform: rp2040_pwm
    id: led_blue
    pin: 20
    inverted: True

light:
  - platform: rgb
    id: led_rgb
    red: led_red
    green: led_green
    blue: led_blue
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has an RGB LED, which the R, G, and B components connected active low (that's why each output definition has inverted: True) to GPIO18, GPIO19, and GPIO20, respectively. This configuration creates a PWM output for each color component, combines them into one RGB light, and starts a light effect with random colors on boot.

Conclusion

In conclusion, the support for the RP2040 platform in ESPHome is not as mature as the support for ESP32 or ESP8266. It lacks the same level of testing and comprehensive documentation. There are several issues reported, and certain functionalities such as MQTT and Bluetooth Low Energy are not yet supported on the RP2040.

However, for a lot of tasks, ESPHome on the Raspberry Pi Pico W is still quite usable. If you have a spare Raspberry Pi Pico W lying around, give it a try! I'm also hopeful that support for additional RP2040 boards with Wi-Fi will be added in the near future.

When BlueZ has connected once to a BLE device, it uses the Device Name characteristic for the device name

The title of this article may seem trivial, but it represents the conclusion I reached after being astonished for a long time while debugging some issues related to decoding Bluetooth Low Energy (BLE) advertisements that rely on the device name. Like many solutions, the actual solution seemed simple in hindsight.

For Theengs Decoder I wrote decoders for two of Shelly's devices: the ShellyBLU Button1 and the ShellyBLU Door/Window sensor. Both devices broadcast BLE advertisements using the BTHome v2 format. Before we can decode these advertisements, we first need to identify the device that is sending a specific advertisement.

In Theengs Decoder, device detection is achieved using a model condition. The Button1 model condition is as follows:

"condition":["servicedata", "=", 14, "index", 0, "40", "|", "servicedata", "=", 14, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBBT-002C"],

This means: the service data (an array of bytes) consists of 14 hexadecimal characters and begins with the byte 0x40 (the first firmware versions) or 0x44 (the latest firmware). Additionally, the service data UUID is fcd2 and the name starts with SBBT-002C.

The Door/Window model condition is similar:

"condition":["servicedata", "=", 28, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBDW-002C"],

Here, the service data consists of 28 hexadecimal characters and starts with the byte 0x44. The service data UUID is fcd2 and the name starts with SBDW-002C.

Therefore, when decoding BLE advertisements, Theengs Decoder tries all model conditions for the supported devices. If any of the conditions is true, the properties defined in the associated decoder are extracted from the advertised data.

After writing these decoders, I tested them with Theengs Gateway on my Linux laptop. Theengs Gateway uses the cross-platform Python library Bleak, which on Linux uses BlueZ, for receiving BLE advertisements. It then decodes them using Theengs Decoder and publishes the decoded properties (in this case a button push or the state of the contact sensor) to an MQTT broker. The decoders worked.

It's important to know that Theengs Decoder purely works on BLE advertisements, which are broadcasted. This is the most basic way to communicate between BLE devices. All information is extracted from these advertisements: manufacturer-specific data, service data, and the local name. [1]

However, when I later experimented with additional functionality of both devices, I connected to them. A couple of days later, I noticed that the decoders no longer worked. When I looked into it, I saw that Theengs Gateway had detected the devices with different names: Shelly Blue Button 1 and Shelly BLE DW. Naturally, the previous model conditions no longer applied, because they were checking for the names SBBT-002C and SBDW-002C, respectively. Interestingly, the same version of Theengs Gateway running on a Raspberry Pi still managed to detect and decode both devices correctly, because it recognized them by their original names. I didn't understand this difference.

It took me a while before I remembered that I had previously connected to both devices on my laptop. So then I started exploring their GATT characteristics. Each BLE device that allows connections has a Generic Access Profile (GAP) service with mandatory Device Name and Appearance characteristics. I read the Device Name characteristic from both devices, and to my surprise they were identified as Shelly Blue Button 1 and Shelly BLE DW.

So that's when I started to connect the dots. Once I had connected to both devices from my laptop, BlueZ seemed to remember their names based on the Device Name characteristic, even after disconnecting. Apparently, when BlueZ later encounters advertisements from the same Bluetooth address, it doesn't use the advertised name (Complete Local Name or Shortened Local Name data type) to identify this device. Instead, it relies on the name it had previously stored from the Device Name characteristic. As a result, when Bleak requests the local name from BlueZ, it receives the device's Device Name characteristic instead of the advertised local name. Consequently, Theengs Gateway sent the 'wrong' name to Theengs Decoder, which caused the devices to remain undetected.

You can check this with bluetoothctl:

$ bluetoothctl
[bluetooth]# info 5C:C7:C1:XX:XX:XX
Device 5C:C7:C1:XX:XX:XX (public)
        Name: Shelly Blue Button 1
        Alias: Shelly Blue Button 1
        Appearance: 0x8001
        Paired: no
        Trusted: yes
        Blocked: no
        Connected: no
        LegacyPairing: no
        UUID: Generic Access Profile    (00001800-0000-1000-8000-00805f9b34fb)
        UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
        UUID: Device Information        (0000180a-0000-1000-8000-00805f9b34fb)
        UUID: Vendor specific           (1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0)
        UUID: Vendor specific           (de8a5aac-a99b-c315-0c80-60d4cbb51225)
        ManufacturerData Key: 0x0ba9
        ManufacturerData Value:
  01 09 00 0b 01 00 0a aa bb cc c1 c7 5c           ............\
        ServiceData Key: 0000fcd2-0000-1000-8000-00805f9b34fb
        ServiceData Value:
  40 00 19 01 5a 3a 01                             @...Z:.
        RSSI: -68
        AdvertisingFlags:
  06                                               .

Fortunately, once I had figured this out, the solution was quite simple: BlueZ needed to forget the devices. You can remove a device from BlueZ's cache with the following command:

$ bluetoothctl remove 3C:2E:F5:XX:XX:XX

Alternatively, you can manually delete the directory where BlueZ stores device information. First, find your Bluetooth adapter's address:

$ hciconfig
hci0: Type: Primary  Bus: USB
      BD Address: 9C:FC:E8:XX:XX:XX  ACL MTU: 1021:4  SCO MTU: 96:6
      UP RUNNING
      RX bytes:1203288 acl:31 sco:0 events:34298 errors:0
      TX bytes:898242 acl:29 sco:0 commands:11700 errors:0

Then delete the following directory, based on the adapter address and the device address:

$ sudo rm -rf /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX

If you're curious about the cached information, first have a look at the culprit before removing the directory:

$ sudo cat /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX/info
[General]
Name=Shelly BLE DW
AddressType=public
SupportedTechnologies=LE;
Trusted=true
Blocked=false
Services=00001800-0000-1000-8000-00805f9b34fb;00001801-0000-1000-8000-00805f9b34fb;0000180a-0000-1000-8000-00805f9b34fb;1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0;de8a5aac-a99b-c315-0c80-60d4cbb51225;
Appearance=0x8001

It's this value of the Name field that BlueZ returns to Bleak, and subsequently to Theengs Gateway and Theengs Decoder, leading to the wrong detection. After removing the device from BlueZ's cache, the advertised local name was detected correctly, and the decoders worked again.

So now I'm wondering: is it possible to get the real local name advertised by a device? Bleak uses the Name property of a device in its BlueZ backend for scanning. Does BlueZ expose the local name, and could Bleak be adapted to use this instead? I couldn't find the answer to this question.

Zephyr's growing hardware support includes popular microcontroller boards for makers

Zephyr, an open-source real-time operating system (RTOS) launched by the Linux Foundation in 2016, has made lots of progress seven years after its announcement, and it now has an active ecosystem surrounding it. It's used in Internet of Things (IoT) sensors, Bluetooth trackers, heart rate monitors, smartwatches, and embedded controllers. A few months ago I wrote an overview article for LWN.net about the project, Zephyr: a modular OS for resource-constrained devices.

The Zephyr RTOS is used in a lot of commercial products as well as some open-source projects. Developers are also continuously adding support for new development boards. Currently, Zephyr supports more than 450 boards from various architectures.

While most of these boards are the typical developer boards used by professional developers, recent years have seen an uptick in support for more boards popular with hobbyists. Here's a list of some supported boards that will surely ring a bell if you're active in the maker community:

So what can you do with Zephyr? An interesting use case for the operating system is developing Bluetooth Low Energy applications running on one of the above boards. A previous blog article explaining Zephyr's iBeacon example code gives you an idea about how such code looks like. If this piques your curiosity, consider delving deeper into the subject by reading my book Develop your own Bluetooth Low Energy Applications for Raspberry Pi, ESP32 and nRF52 with Python, Arduino and Zephyr, which uses Nordic Semiconductor's nRF52840 Dongle and nRF52840 Development Kit. But Zephyr supports a plethora of communication protocols, including Thread, a low-power IPv6-based wireless mesh-networking technology for home-automation applications, and other protocols listed in the aforementioned LWN.net article.

Even if you're not up to the task of developing applications in the C programming language, Zephyr can be interesting for you. If you prefer Python, you've surely heard about MicroPython, a programming language that implements a sizable subset of Python that can run on microcontrollers. You can read more about it in my recent LWN.net article MicroPython 1.20: Python for microcontrollers. MicroPython offers firmware downloads for more than 150 microcontroller boards, but it also has been ported to Zephyr.

In MicroPython 1.20, this port is based on Zephyr 3.1.0, which was released in June 2022; the current Zephyr release is 3.4.0. A Zephyr development environment can be used to build MicroPython for every target board supported by Zephyr, although not all have been tested. It also gives MicroPython code access to Zephyr's uniform sensor API using the zsensor module. More information can be found in MicroPython's documentation about the Zephyr port.

Thanks to Zephyr's broad hardware support, the Zephyr port allows MicroPython to run on the BBC micro:bit v2, whereas only a direct MicroPython port for the BBC micro:bit v1 exists. Other interesting boards from the above list that don't have a MicroPython firmware download but that are supported thanks to the Zephyr port are the Arduino Nano 33 IoT, the PineTime, and the RuuviTag. The process of setting up the development environment, building the Zephyr port for the target board, and then flashing the firmware to the board is explained in the port's README file.

How to use the await keyword in the Python REPL without asyncio.run()

The Python REPL (read-eval-print loop), which is Python's interactive interpreter, is a great way for quickly testing simple Python commands. I use it quite often as a powerful command-line calculator, but also for exploring new Python libraries.

Many interesting Python libraries use asynchronous I/O with asyncio. This means that, instead of just calling a function directly, you have to await a coroutine. However, if you try this in the REPL, you encounter the following error message:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>>

This is expected, because this line of code would never run in a Python script either. You would need to define a top-level coroutine with async def that calls one or more coroutines with await, and run the top-level coroutine with asyncio.run(). The canonical "Hello world" example from the Python documentation of coroutines looks something like this:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> async def main():
...     print("Hello")
...     await asyncio.sleep(5)
...     print("world")
...
>>> asyncio.run(main())
Hello
world
>>>

There's a five-second delay between the output of "Hello" and "world".

This seems a bit cumbersome. If you just want to call the asyncio.sleep() coroutine, you can simplify this to:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.run(asyncio.sleep(5))
>>>

This is already better! However, it still means that every time you want to call a coroutine, you need to remember to wrap it inside an asyncio.run() call.

Fortunately, Python 3.8 introduced a top-level await if you run the asyncio module as a script:

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
>>>

The REPL helpfully mentions Use "await" directly instead of "asyncio.run()". before importing the asyncio module. [1] Then you can simply type await asyncio.sleep(5) without having to call asyncio.run().

Although this might not seem like much of an improvement in this particular case, when using asynchronous libraries in a Python REPL, having to add asyncio.run() for every coroutine call quickly becomes tedious.

For example, I can now easily request the Bluetooth adapters on my system (using Home Assistant's bluetooth-adapters package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bluetooth_adapters import get_adapters
>>> adapters = get_adapters()
>>> await adapters.refresh()
>>> adapters.adapters
{'hci0': {'address': '9C:FC:E8:XX:XX:XX', 'sw_version': 'tux', 'hw_version': 'usb:v1D6Bp0246d0540', 'passive_scan': True, 'manufacturer': 'Intel Corporate', 'product': '0029', 'vendor_id': '8087', 'product_id': '0029'}}

Or, I can conduct a quick scan for Bluetooth Low Energy (BLE) devices in the vicinity (using the Bleak package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bleak import BleakScanner
>>> devices = await BleakScanner.discover()
>>> [device.name for device in devices]
['Qingping Alarm Clock', 'abeacon_AC7D', 'TP358 (52C6)', '1B-0F-09-4F-89-F3', 'ThermoBeacon', 'F9-DA-D2-0D-62-24', '6A-C8-79-F4-E1-E5', 'Qingping BT Clock Lite', 'LYWSD02', 'Ruuvi 7E0E', 'TY', 'TP393 (2A3D)', 'Flower care', '52-9E-F1-64-DB-DF']

Give this top-level await approach a try with some of your favorite asynchronous Python libraries. You'll definitely become more productive in the REPL.

Creating terminal user interfaces in Python with Textual

Last year, I began using Textual to develop HumBLE Explorer, a cross-platform, command-line and human-friendly Bluetooth Low Energy scanner. Textual caught my attention because it promised to be a rapid application development framework for Python terminal user interface (TUI) applications. Thanks to Textual, I was able to create an application like this, including scroll bars, switches and tables:

/images/humble-explorer-light.png

One of Textual's key features is that it's inspired by web development practices. This allows for a clear separation of design and code using CSS files (in the Textual CSS dialect), making the framework both developer-friendly and highly customizable. It also has reactive attributes, as well as a growing library of widgets. Additionally, the framework provides useful tools for debugging and live editing of CSS files during development.

If you want to have an idea about Textual's capabilities, install it from PyPI and run the demo:

pip install textual
python -m textual

In addition, one of Textual's developers maintains a list of Textual-based applications.

As a relatively new project (started in 2021), Textual still experiences occasional breaking changes in new releases. However, the developers are easily accessible for support on their Discord server and provide regular blog updates. Moreover, Textual has excellent documentation. I plan to use Textual again for new terminal user interfaces in Python.

For some more background, read my article Textual: a framework for terminal user interfaces on LWN.

Creating a diagnostics module for Python software

For Theengs Gateway we regularly got bug reports that were difficult to debug, as they depend on the operating system, Python version, application configuration, and Bluetooth adapter. After some back and forth we got an idea of the user's environment and started to identify the issue (or not). Then I discovered that Textual has a convenient solution for this: a diagnose command that prints information about the Textual library and its environment to help diagnose problems.

I borrowed this code from Textual and adapted it for Theengs Gateway's usage. So now we simply ask the user to run this command and include its output in the issue description on GitHub. Theengs Gateway's diagnose module looks like this:

theengs_gateway/diagnose.py (Source)

import asyncio
import json
import os
import platform
import re
import sys

from importlib_metadata import PackageNotFoundError, version

_conf_path = os.path.expanduser("~") + "/theengsgw.conf"
_ADDR_RE = re.compile(r"^(([0-9A-F]{2}:){3})([0-9A-F]{2}:){2}[0-9A-F]{2}$")


def _anonymize_strings(fields, config) -> None:
    for field in fields:
        if field in config:
            config[field] = "***"


def _anonymize_address(address) -> str:
    addr_parts = _ADDR_RE.match(address)
    if addr_parts:
        return f"{addr_parts.group(1)}XX:XX:XX"
    else:
        return "INVALID ADDRESS"


def _anonymize_addresses(field, config) -> None:
    try:
        config[field] = [
            _anonymize_address(address) for address in config[field]
        ]
    except KeyError:
        pass


# This function is taken from Textual
def _section(title, values) -> None:
    """Print a collection of named values within a titled section.
    Args:
        title: The title for the section.
        values: The values to print out.
    """
    max_name = max(map(len, values.keys()))
    max_value = max(map(len, [str(value) for value in values.values()]))
    print(f"## {title}")
    print()
    print(f"| {'Name':{max_name}} | {'Value':{max_value}} |")
    print(f"|-{'-' * max_name}-|-{'-'*max_value}-|")
    for name, value in values.items():
        print(f"| {name:{max_name}} | {str(value):{max_value}} |")
    print()


def _versions() -> None:
    """Print useful version numbers."""
    try:
        packages = {
            "Theengs Gateway": version("TheengsGateway"),
            "Theengs Decoder": version("TheengsDecoder"),
            "Bleak": version("bleak"),
            "Bluetooth Clocks": version("bluetooth-clocks"),
            "Bluetooth Numbers": version("bluetooth-numbers"),
            "Paho MQTT": version("paho-mqtt"),
        }
    except PackageNotFoundError as e:
        print(f"Package {e.name} not found. Please install it with:")
        print()
        print(f"    pip install {e.name}")
        print()

    if sys.version_info[:2] >= (3, 9):
        try:
            packages["Bluetooth Adapters"] = version("bluetooth-adapters")
        except PackageNotFoundError as e:
            print(f"Package {e.name} not found. Please install it with:")
            print()
            print(f"    pip install {e.name}")
            print()

    _section("Package Versions", packages)


def _python() -> None:
    """Print information about Python."""
    _section(
        "Python",
        {
            "Version": platform.python_version(),
            "Implementation": platform.python_implementation(),
            "Compiler": platform.python_compiler(),
            "Executable": sys.executable,
        },
    )


def _os() -> None:
    os_parameters = {
        "System": platform.system(),
        "Release": platform.release(),
        "Version": platform.version(),
        "Machine type": platform.machine(),
    }
    if platform.system() == "Linux" and sys.version_info[:2] >= (3, 10):
        os_parameters["Distribution"] = platform.freedesktop_os_release()[
            "PRETTY_NAME"
        ]

    _section("Operating System", os_parameters)


def _config() -> None:
    print("## Configuration")
    print()
    try:
        with open(_conf_path, encoding="utf-8") as config_file:
            config = json.load(config_file)
            _anonymize_strings(["user", "pass"], config)
            _anonymize_addresses("time_sync", config)
        print("```")
        print(json.dumps(config, sort_keys=True, indent=4))
        print("```")
        print()
    except FileNotFoundError:
        print(f"Configuration file not found: {_conf_path}")
        print()


async def _adapters() -> None:
    if sys.version_info[:2] >= (3, 9):
        from bluetooth_adapters import get_adapters

        print("## Bluetooth adapters")
        print()
        bluetooth_adapters = get_adapters()
        await bluetooth_adapters.refresh()
        print(f"Default adapter: {bluetooth_adapters.default_adapter}")
        print()

        for adapter, properties in sorted(bluetooth_adapters.adapters.items()):
            properties["address"] = _anonymize_address(properties["address"])
            print("#", end="")
            _section(adapter, properties)


async def diagnostics():
    print("# Theengs Gateway Diagnostics")
    print()
    _versions()
    _python()
    _os()
    _config()
    await _adapters()


if __name__ == "__main__":
    asyncio.run(diagnostics())

When you run this module, it prints a level one Markdown title (# Theengs Gateway Diagnostics) and then calls several functions. Each of these functions prints a level two Markdown title and some diagnostic information.

First, it displays the version numbers of the Python package for Theengs Gateway and some of its dependencies. This helps us immediately identify outdated versions, and we can suggest an update. Next, it shows information about the Python platform and the operating system. These functions are all borrowed from Textual's diagnose module, including the _section helper function to print a collection of named values within a titled section.

Since many Theengs Gateway issues depend on the exact configuration used, I also added a section that displays the contents of the configuration file (a JSON file). However, this configuration file contains some information that shouldn't be shared publicly, such as a username and password for an MQTT broker, or Bluetooth addresses. I could remove these fields in the code, but then we wouldn't know if the bug might be a result of a configuration file lacking one of these fields. So I created a simple function to anonymize specific fields:

def _anonymize_strings(fields, config) -> None:
    for field in fields:
        if field in config:
            config[field] = "***"

Then I can call this function on the configuration to anonymize the user and pass fields:

_anonymize_strings(["user", "pass"], config)

For Bluetooth addresses, I created a similar function. I want to keep the first three bytes of an address, which can point to the device manufacturer and be helpful for debugging purposes. Using a regular expression, I extract these bytes and add XX:XX:XX. This function looks like this:

_ADDR_RE = re.compile(r"^(([0-9A-F]{2}:){3})([0-9A-F]{2}:){2}[0-9A-F]{2}$")


def _anonymize_address(address) -> str:
    addr_parts = _ADDR_RE.match(address)
    if addr_parts:
        return f"{addr_parts.group(1)}XX:XX:XX"
    else:
        return "INVALID ADDRESS"

In the last part of the diagnostic information, where I display the information of the computer's Bluetooth adapters, I can call this function to anonymize the adapter's Bluetooth address:

properties["address"] = _anonymize_address(properties["address"])

Running the python -m TheengsGateway.diagnose command shows output like this:

# Theengs Gateway Diagnostics

## Package Versions

| Name               | Value  |
|--------------------|--------|
| Theengs Gateway    | 3.0    |
| Theengs Decoder    | 1.4.0  |
| Bleak              | 0.20.0 |
| Bluetooth Clocks   | 0.1.0  |
| Bluetooth Numbers  | 1.1.0  |
| Paho MQTT          | 1.6.1  |
| Bluetooth Adapters | 0.15.3 |

## Python

| Name           | Value           |
|----------------|-----------------|
| Version        | 3.10.6          |
| Implementation | CPython         |
| Compiler       | GCC 11.3.0      |
| Executable     | /usr/bin/python |

## Operating System

| Name         | Value                                               |
|--------------|-----------------------------------------------------|
| System       | Linux                                               |
| Release      | 6.2.0-10005-tuxedo                                  |
| Version      | #5 SMP PREEMPT_DYNAMIC Wed Mar 22 12:42:40 UTC 2023 |
| Machine type | x86_64                                              |
| Distribution | Ubuntu 22.04.1 LTS                                  |

## Configuration

```
{
    "adapter": "hci0",
    "ble_scan_time": 1000,
    "ble_time_between_scans": 5,
    "discovery": 1,
    "discovery_device_name": "TheengsGateway",
    "discovery_filter": [
        "IBEACON",
        "GAEN",
        "MS-CDP"
    ],
    "discovery_topic": "homeassistant/sensor",
    "hass_discovery": 1,
    "host": "rhasspy",
    "log_level": "DEBUG",
    "lwt_topic": "home/TheengsGateway/LWT",
    "pass": "***",
    "port": 1883,
    "presence": 0,
    "presence_topic": "home/TheengsGateway/presence",
    "publish_advdata": 1,
    "publish_all": 1,
    "publish_topic": "home/TheengsGateway/BTtoMQTT",
    "scanning_mode": "active",
    "subscribe_topic": "home/+/BTtoMQTT/undecoded",
    "time_format": 1,
    "time_sync": [
        "58:2D:34:XX:XX:XX",
        "E7:2E:00:XX:XX:XX",
        "BC:C7:DA:XX:XX:XX",
        "10:76:36:XX:XX:XX"
    ],
    "user": "***"
}
```

## Bluetooth adapters

Default adapter: hci0

### hci0

| Name         | Value               |
|--------------|---------------------|
| address      | 9C:FC:E8:XX:XX:XX   |
| sw_version   | tux                 |
| hw_version   | usb:v1D6Bp0246d0540 |
| passive_scan | True                |
| manufacturer | Intel Corporate     |
| product      | 0029                |
| vendor_id    | 8087                |
| product_id   | 0029                |

### hci1

| Name         | Value                   |
|--------------|-------------------------|
| address      | 00:01:95:XX:XX:XX       |
| sw_version   | tux #2                  |
| hw_version   | usb:v1D6Bp0246d0540     |
| passive_scan | True                    |
| manufacturer | Sena Technologies, Inc. |
| product      | 0001                    |
| vendor_id    | 0a12                    |
| product_id   | 0001                    |

In the repository's issue template for bug reports, we ask for the output of this command. The user simply has to copy the output, which is already formatted in Markdown syntax. This displays titles, subtitles, and even tables cleanly, providing us the necessary information:

/images/theengs-gateway-diagnose.png

Hopes and promises for open-source voice assistants

Paulus Schoutsen, founder of the open-source home-automation project Home Assistant, declared 2023 as "the year of voice" for the popular platform. The goal of the initiative is to enable users to control their homes through offline voice commands in their own language.

Voice control is a complex and computationally intensive task, which is usually delegated to the cloud. Companies like Google, Amazon and Apple make us believe that we need their cloud-based services to be able to use voice control. Of course, this comes with downsides: users don't have any control over what happens with their voice recordings, posing a significant privacy risk. But, fundamentally, the problem lies even deeper. It just makes no sense for users to have their voices make a long detour through the internet just to turn on a light in the same room.

In the past, projects like Snips and Mycroft attempted offline voice control but faced business challenges. Rhasspy, an independent open-source voice-assistant project that has been active for a few years now, was quite successful among the niche crowd of tinkerers and those who built their own voice assistants around the flexible services the project offered. [1] However, the core of Rhasspy was mainly developed by one person, and the project wasn't backed financially.

Last month, I wrote an article for LWN.net about these three projects: Hopes and promises for open-source voice assistants. I expressed the hope that Rhasspy would finally give us the ability to control our homes with a user-friendly voice assistant that is both privacy respecting and made from open-source software. Rhasspy's developer, Michael Hansen, has been hired by Nabu Casa, the company behind Home Assistant, and they're tightly integrating Rhasspy into their home-automation software.

In the mean time, OpenVoiceOS, a community that forked Mycroft, has published a FAQ about the future of Mycroft. I already alluded to Mycroft's revival in my article, but the plans were still vague at the time. By now, it looks like Mycroft has a real chance to live on in OpenVoiceOS.

Overall, these are exciting times for open-source voice control.