2022 was the year of Bluetooth
For me, 2022 was the year of Bluetooth. [1] With all the talk about Matter, the one protocol to connect all home automation devices, this can sound strange. However, it will take years for Matter to become adopted, and in the mean time Bluetooth devices are everywhere.
I wrote a book about Bluetooth
Elektor International Media published my book Develop your own Bluetooth Low Energy Applications for Raspberry Pi, ESP32 and nRF52 with Python, Arduino and Zephyr this year. Why did I decide to write a book about Bluetooth? It comes down to a unique combination of accessibility, ubiquity and easy basics of the technology.
Bluetooth Low Energy (BLE) is one of the most accessible wireless communication standards. There's no cost to access the official BLE specifications. Moreover, BLE chips are cheap, and the available development boards (based on an nRF5 or ESP32) and Raspberry Pis are quite affordable. [2] This means you can just start with BLE programming at minimal cost.
On the software side, BLE is similarly accessible. Many development platforms, most of them open source, offer an API (application programming interface) to assist you in developing your own BLE applications. The real-time operating system Zephyr is a powerful platform to develop BLE applications for Nordic Semiconductor nRF5 or equivalent SoCs, with Python and Bleak it's easy to decode BLE advertisemens, and NimBLE-Arduino makes it possible to create powerful firmware such as OpenMQTTGateway for the ESP32.
Another important factor is that BLE radio chips are ubiquitous. You can find them in smartphones, tablets, and laptops. This means that all those devices can talk to your BLE sensors or lightbulbs. Most manufacturers create mobile apps to control their BLE devices, which you can reverse engineer (as I explain in one of the chapters of my book).
You can also find BLE radios in many single-board computers, such as the Raspberry Pi, and in popular microcontroller platforms such as the ESP32. This makes it quite easy for you to create your own gateways for BLE devices. And platforms such as the Nordic Semiconductor nRF5 series of microcontrollers with BLE radio even make it possible to create your own battery-powered BLE devices.
Last but not least, while Bluetooth Low Energy is a complex technology with a comprehensive specification, getting started with the basics is relatively easy. I hope my book contributes to this by explaining the necessary groundwork and showing the right examples to create your own BLE applications.
I contributed to the Theengs project
I have been using OpenMQTTGateway at home for some time, which is a gateway for various wireless protocols that you can install on an ESP32 or other devices. This year OpenMQTTGateway spun out their BLE decoder to a separate project, Theengs Decoder. This is an efficient, portable and lightweight C++ library for BLE payload decoding.
I contributed to the Theengs Decoder project with some decoders for the following devices:
the RuuviTag environmental sensor (for the RAWv1 and RAWv2 format)
April Brother's ABTemp iBeacon with temperature sensor
devices broadcasting their battery percentage as service data
The Theengs project also created a gateway that you can run on a Linux machine such as a Raspberry Pi, Theengs Gateway. This leverages the same Theengs Decoder as OpenMQTTGateway. I quickly adopted this solution as an alternative to bt-mqtt-gateway (which I contributed to with RuuviTag support earlier), and I started contributing to the project. Amongst others, I:
added support for passive scanning on Windows and Linux
implemented time synchronization for LYWSD02 devices
packaged Theengs Gateway as a snap
created an Ubuntu Core appliance with Theengs Gateway
I also started Theengs Explorer under the Theengs umbrella. This is a text user interface to discover BLE devices and show their raw advertisement data and the data as decoded by Theengs Decoder. This project is still in early development, because I wrote this using a pre-0.2 release of Textual, and I still have to rewrite it.
I wrote some Python packages for Bluetooth
Outside the Theengs project, I created two Python packages related to Bluetooth this year. After I struggled with updating the Theengs Explorer code base to the new Textual 0.2 release, I decided to start from scratch with a 'simple' Bluetooth Low Energy scanner. This became HumBLE Explorer, which is a cross-platform, command-line and human-friendly Bluetooth Low Energy scanner, looking like this:
Textual is quite neat. It lets you create interactive applications for the terminal, with widgets such as checkboxes and input fields, a CSS-like layout language, and even mouse support. Moreover, it runs on Linux, Windows and macOS. Although I'm personally only using Linux at home, I find it important that my applications are cross-platform. That's also the reason why this application, as well as all my Bluetooth work with Python, is based on Bleak, which supports the same three operating systems.
Now that HumBLE Explorer is working, I'll revisit Theengs Explorer soon, and update it to the new Textual version with the knowledge that I gained.
A second Bluetooth project that I have been working on this year, even before HumBLE Explorer or Theengs Explorer, is bluetooth-numbers. It's a Python package with a wide set of numbers related to Bluetooth, so Python projects can easily use these numbers. The goal of this project is to provide a shared resource so various Python projects that deal with Bluetooth don't have to replicate this effort by rolling their own database and keeping it updated.
Luckily Nordic Semiconductor already maintains the Bluetooth Numbers Database for Company IDs, Service UUIDs, Characteristic UUIDs and Descriptor UUIDs. My bluetooth-numbers package started as a Python wrapper around this project, by generating Python modules with these data. In the mean time, I extended the package with some SDO Service UUIDs and Member Service UUIDs I extracted from the Bluetooth Assigned Numbers document (but which I'll probably upstream to the Bluetooth Numbers Database), as well as the IEEE database of OUIs for prefixes of Bluetooth addresses.
So you now can install the package from PyPI with pip:
Then you can get the description of a company ID in your Python code:
Get the description of a service UUID:
>>> from bluetooth_numbers import service >>> from uuid import UUID >>> service[0x180F] 'Battery Service' >>> service[UUID("6E400001-B5A3-F393-E0A9-E50E24DCCA9E")] 'Nordic UART Service'
Get the description of a characteristic UUID:
>>> from bluetooth_numbers import characteristic >>> from uuid import UUID >>> characteristic[0x2A37] 'Heart Rate Measurement' >>> characteristic[UUID("6E400002-B5A3-F393-E0A9-E50E24DCCA9E")] 'UART RX Characteristic'
Get the description of a descriptor UUID:
>>> from bluetooth_numbers import descriptor >>> descriptor[0x2901] 'Characteristic User Descriptor'
Get the description of an OUI:
I'm using bluetooth-numbers in HumBLE Explorer and Theengs Explorer to show human-readable descriptions of these numbers in the interface. I hope that other Python projects related to Bluetooth will adopt the package too, to prevent everyone from having to keep their own numbers database updated.
I'm keeping the package updated with its various sources, and it has a test suite with 100% code coverage. Contributions are welcome. The API documentation shows how to use it.
Bluetooth developments in Home Assistant and ESPHome
Although I'm personally advocating an MQTT-based approach to home automation, I'm a big fan of Home Assistant and ESPHome because they share my vision of home automation and make it rather easy to use. Both open-source home automation projects had some big improvements in their Bluetooth support this year.
When my book Getting Started with ESPHome: Develop your own custom home automation devices was published last year, BLE support in ESPHome was still quite limited. It only supported reading BLE advertisements, but not connecting to BLE devices. Support for using ESPHome as a BLE client was only added after the book was published.
The biggest BLE addition in 2022 was Bluetooth Proxy in ESPHome 2022.8. This allows you to use your ESP32 devices with ESPHome firmware as BLE extenders for Home Assistant. Each ESPHome Bluetooth proxy device forwards the BLE advertisements it receives to your Home Assistant installation. By strategically placing some devices in various places at home, you can expand the Bluetooth range of your devices this way. [3]
Starting from ESPHome 2022.9 the Bluetooth proxy also supports active connections: it lets Home Assistant connect to your devices that are out of reach of your home automation gateway, as long as one of your Bluetooth proxy devices are in reach of the device you want to connect to.
This feature was joined by a brand new Bluetooth integration in Home Assistant 2022.8, with automatic discovery of new devices and the ability to push device updates. Home Assistant 2022.9 then added support for ESPHome's Bluetooth proxies, and Home Assistant 2022.10 extended this support to active connections.
Another interesting initiative coming from the Home Assistant project is BTHome, a new open standard for broadcasting sensor data over BLE. Raphael Baron's open-source soil moisture sensor b-parasite already adopted the BTHome standard, as did the custom firmware ATC MiThermometer for various Xiaomi temperature sensors. With the BTHome integration in Home Assistant 2022.9, devices advertising in this format will be automatically discovered in your Home Assistant installation.
BTHome's data format is documented in detail, with various data types supported. There's even support for encryption, using AES in CCM mode with a pre-shared key. The bthome-ble project implements a parser for BTHome payloads you can use in your own Python projects. I applaud the initiative to create an open standard for BLE sensor advertisements, and I hope that many open-source devices will adopt BTHome. I will definitely use the format if I create a BLE broadcaster instead of coming up with my own data format.
I also found it interesting to see that Home Assistant decided to move to Bleak as their BLE library. They even sponsored the lead developer David Lechner to implement passive scanning in the Linux backend. This benefits the broader open-source community and allowed me to add passive scanning support to Theengs Gateway, Theengs Explorer and HumBLE Explorer. With Home Assistant as a big user of Bleak, we'll surely see it improving even more. And Bleak is already the best Python library for BLE...
What with BLE in 2023?
I expect that Bluetooth will still remain an important technology in 2023 and further, because I don't see anything changing about the unique combination of accessibility, ubiquity and easy basics. So I will keep contributing to the Theengs project and developing my own Bluetooth projects.
I still have a couple of BLE sensors at home that aren't supported yet by Theengs Decoder, and I'd like to change that! If you have a device that isn't on the list of supported devices, why don't you try adding a decoder? You don't need to be a developer to do this, as the decoders are specifications of the advertisement format in a JSON file.