What is Thread?

Thread is a low-power, wireless, and IPv6-based networking protocol designed specifically for IoT devices. Development of the protocol is managed by the Thread Group, an alliance founded in 2015. This is a consortium of major industry players, including Google, Apple, Amazon, Qualcomm, Silicon Labs, and Nordic Semiconductor.

You can download the Thread specification for free, although registration is required. The version history is as follows:

All versions of the Thread specification maintain backward compatibility.

Thread’s architecture

Thread employs a mesh networking architecture that allows devices to communicate directly with each other, eliminating the need for a central hub or gateway. This architecture offers inherent redundancy, as devices can relay data using multiple paths, ensuring increased reliability in case of any single node failure.

The Thread protocol stack is built upon the widely used IPv6 protocol, which simplifies the integration of Thread networks into existing IP infrastructures. Thread is designed to be lightweight, streamlined and efficient, making it an excellent protocol for IoT devices running in resource-constrained environments.

Before discussing Thread, it’s essential to understand its place in the network protocol environment. Most modern networks are based on the Internet Protocol suite, which uses a four-layer architecture. The layers of the Internet Protocol suite are, from bottom to top:

Link layer

Defines the device’s connection with a local network. Protocols like Ethernet and Wi-Fi operate within this layer. In the more complex Open Systems Interconnection (OSI) model, this layer includes the physical and data link layers.

Network layer

Enables communication across network boundaries, known as routing. The Internet Protocol (IP) operates within this layer and defines IP addresses.

Transport layer

Enables communication between two devices, either on the same network or on different networks with routers in between. The transport layer also defines the concept of a port. User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) operate within this layer.

Application layer

Facilitates communication between applications on the same device or different devices. Protocols like HyperText Transfer Protocol (HTTP), Domain Name System (DNS), Simple Mail Transfer Protocol (SMTP), and Message Queuing Telemetry Transport (MQTT) operate within this layer.

When data is transmitted across the network, the data of each layer is embedded in the layer below it, as shown in this diagram:

Thread operates in the network and transport layers. However, it’s important to know what’s going on in all layers.

Thread lacks an application layer

Unlike home automation protocols such as Zigbee, Z-Wave, or Matter, the Thread standard doesn’t define an application layer. A Thread network merely offers network infrastructure that applications can use to communicate. Just as your web browser and email client use TCP/IP over Ethernet or Wi-Fi, home automation devices can use Thread over IEEE 802.15.4 as their network infrastructure.

Several application layers exist that can make use of Thread:

Constrained Application Protocol (CoAP)

A simpler version of HTTP, designed for resource-constrained devices and using UDP instead of TCP.

MQTT for Sensor Networks (MQTT-SN)

A variant of MQTT designed to be used over UDP.

Apple HomeKit

Apple’s application protocol for home automation devices.

Matter

A new home automation standard developed by the Connectivity Standards Alliance (CSA).

Like your home network which simultaneously hosts a lot of application protocols including HTTP, DNS, and SMTP, a Thread network can also run all these application protocols concurrently.

Throughout this book, I will be using CoAP as an application protocol in a Thread network. Once you have a Thread network set up, you can also use it for Apple HomeKit and Matter devices.

Advantages of Thread

Some of Thread’s key benefits include:

Disadvantages of Thread

In life, they say there’s no such thing as a free lunch and Thread is no exception to this rule. Some of its limitations are:

Platforms used in this book

In this book I focus on building wireless sensor networks using the Thread protocol together with the following hardware and software environments:

Nordic Semiconductor’s nRF52840 SoC

A powerful yet energy-efficient and low cost hardware solution for Thread-based IoT devices

OpenThread

An open-source implementation of the Thread protocol, originally developed by Google, making it easy to incorporate Thread into your IoT projects

Zephyr

An open-source real-time operating system designed for resource-constrained devices

Armed with these tools and by using practical examples I will go on to guide you step-by-step through the details of the hardware and software required to build Thread networks and OpenThread-based applications.

Nordic Semiconductor’s nRF52840 SoC

Nordic Semiconductor’s nRF52840 is a SoC built around the 32-bit ARM Cortex-M4 CPU running at 64 MHz. This chip supports Bluetooth Low Energy (BLE), Bluetooth Mesh, Thread, Zigbee, IEEE 802.15.4, ANT and 2.4 GHz proprietary stacks. With its 1 MB flash storage and 256 KB RAM, it offers ample resources for advanced Thread applications.

This SoC is available for developers in the form of Nordic Semiconductor’s user-friendly nRF52840 Dongle. You can power and program this small, low-cost USB dongle via a computer USB port. The documentation lists comprehensive information about the dongle.

OpenThread

Thread is basically a networking protocol; in order to work with it you need an implementation of the Thread networking protocol. The industry’s reference implementation, used even by professional Thread device developers, is OpenThread. It’s a BSD-licensed implementation, with development ongoing via its GitHub repository. The license for this software allows for its use in both open-source and proprietary applications.

OpenThread provides an Application Programming Interface (API) that’s operating system and platform agnostic. It has a narrow platform abstraction layer to achieve this, and has a small memory footprint, making it highly portable. In this book, I will be using OpenThread in conjunction with Zephyr, but the same API can be used in other combinations, such as ESP-IDF for Espressif’s Thread SoCs.

Zephyr

Zephyr is an open-source real-time operating system (RTOS) designed for resource-constrained devices. It incorporates OpenThread as a module, simplifying the process of creating Thread applications based on Zephyr and OpenThread. Because of Zephyr’s hardware abstraction layer, these applications will run on all SoCs supported by Zephyr that have an IEEE 802.15.4 radio facility.

Recommended hardware

For the hardware, I recommend a Nordic Semiconductor nRF52840 Dongle, but you can use any microcontroller board with Thread radio supported by Zephyr. There are variants of the dongle by other manufacturers, such as the April USB Dongle 52840 (with an external antenna) or the nRF52840 MDK USB Dongle from makerdiary (with a case).

The more devices you have in a Thread network, the better it works, because Routers can relay messages via other devices. Having more devices also aids in understanding how Thread operates: you can turn devices on and off, run commands on them through Zephyr’s shell, and observe how the network reacts. So, if possible, I recommend purchasing a handful of nRF52840 Dongles or other IEEE 802.15.4 devices supported by Zephyr. A USB hub with several Thread devices plugged in makes an excellent test setup, as shown in this example:

If you’re serious about Thread development with Zephyr, I also recommend Nordic Semiconductor’s nRF52840 Development Kit. This is the bigger (and slightly less affordable) brother of the nRF52840 Dongle, offering more capabilities to debug code easily. With one of these, you can carry out initial development and debugging then flash the code to the nRF52840 Dongle once all the bugs have been ironed out.

Advertising data structures in Zephyr

Then, you see the definition of an array of struct bt_data elements. The bt_data struct describes an advertising data structure with a type, length, and a pointer to the data. Its definition (in zephyr/bluetooth/bluetooth.h) is:

/** Description of different data types that can be encoded into
 * advertising data. Used to form arrays that are passed to the
 * bt_le_adv_start() function.
 */
struct bt_data {
        uint8_t type;
        uint8_t data_len;
        const uint8_t *data;
};

In the iBeacon program, the array consists of two elements. This is consistent with the iBeacon specification, which shows that an iBeacon advertising packet consists of two advertising structures: flags and manufacturer-specific data.

In principle, you could fill the array with bt_data structs you create yourself, but Zephyr defines some helper macros. Their definition (again, from zephyr/bluetooth/bluetooth.h) looks like this:

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _data Pointer to the data field payload
 * @param _data_len Number of bytes behind the _data pointer
 */
#define BT_DATA(_type, _data, _data_len) \
        { \
                .type = (_type), \
                .data_len = (_data_len), \
                .data = (const uint8_t *)(_data), \
        }

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _bytes Variable number of single-byte parameters
 */
#define BT_DATA_BYTES(_type, _bytes...) \
        BT_DATA(_type, ((uint8_t []) { _bytes }), \
                sizeof((uint8_t []) { _bytes }))

So, with the BT_DATA macro, you construct a bt_data struct with the type, data, and data length you specify (using a pointer to the data). BT_DATA_BYTES is a macro that uses the BT_DATA macro and automatically fills it with the right length for data with a known size.

If you return to the iBeacon program, you see that it fills the array with two elements using the BT_DATA_BYTES macro.

The first element is of type BT_DATA_FLAGS and with data BT_LE_AD_NO_BREDR. You can find the definition of these constants in zephyr/bluetooth/gap.h (GAP stands for Generic Access Profile). For non-connectable advertising packets, the flags data structure is optional.

The second element is of type BT_DATA_MANUFACTURER_DATA. The data used to construct this element follows the format listed in the iBeacon specification: Apple's two-byte company ID, the type (0x02), and length (0x15 or 21 bytes) for the iBeacon data type, and then the UUID, major, minor, and measured power.

All in all, the result of all these macros is an array with advertising data structures that looks like this, schematically:

Enabling Bluetooth

Until now, the code has been all about setting up the right data structures to advertise. Let's have a look at the main() function. This actually does just one thing: calling the bt_enable() function. You have to call this function before any other calls that use the board's Bluetooth hardware. It initializes the Bluetooth subsystem and returns 0 on success and a negative error code otherwise.

The argument to the bt_enable() function is a callback function that's called when initializing Bluetooth is completed. It's in this callback function, bt_ready(), that the core of the program resides.

Advertising

The callback function is called with one argument -- an error code that is 0 on success, or an error code consisting of a negative value otherwise. So, in bt_ready(), you first check whether Bluetooth initialization failed, and, if not, you start advertising.

The definition of bt_le_adv_start() (still in zephyr/bluetooth/bluetooth.h), the function used to start advertising, is:

/** @brief Start advertising
 *
 * Set advertisement data, scan response data, advertisement parameters
 * and start advertising.
 *
 * When the advertisement parameter peer address has been set the advertising
 * will be directed to the peer. In this case advertisement data and scan
 * response data parameters are ignored. If the mode is high duty cycle
 * the timeout will be @ref BT_GAP_ADV_HIGH_DUTY_CYCLE_MAX_TIMEOUT.
 *
 * @param param Advertising parameters.
 * @param ad Data to be used in advertisement packets.
 * @param ad_len Number of elements in ad
 * @param sd Data to be used in scan response packets.
 * @param sd_len Number of elements in sd
 *
 * @return Zero on success or (negative) error code otherwise.
 * @return -ENOMEM No free connection objects available for connectable
 *                 advertiser.
 * @return -ECONNREFUSED When connectable advertising is requested and there
 *                       is already maximum number of connections established
 *                       in the controller.
 *                       This error code is only guaranteed when using Zephyr
 *                       controller, for other controllers code returned in
 *                       this case may be -EIO.
 */
int bt_le_adv_start(const struct bt_le_adv_param *param,
                    const struct bt_data *ad, size_t ad_len,
                    const struct bt_data *sd, size_t sd_len);

So, to call this function, you need to supply advertising parameters, the advertising data structure's array and its length, as well as the array of data structures for the scan response packet and its length.

The iBeacon example program doesn't use any scan response data, so the last two arguments are NULL and 0. The array of advertising data structures is ad, which you constructed in the beginning of the code. With ARRAY_SIZE, a macro defined in zephyr/sys/util.h, you compute the number of elements in the array. When using a C compiler, this macro is defined as:

#define ARRAY_SIZE(array) (sizeof(array) / sizeof((array)[0]))

The first argument to the bt_le_adv_start() function specifies the advertising parameters. In this example, this argument is BT_LE_ADV_NCONN. If you look up its definition in zephyr/bluetooth/bluetooth.h, you'll see that this defines non-connectable advertising with a private address:

/** Non-connectable advertising with private address */
#define BT_LE_ADV_NCONN BT_LE_ADV_PARAM(0, BT_GAP_ADV_FAST_INT_MIN_2, \
                                        BT_GAP_ADV_FAST_INT_MAX_2, NULL)

The macro BT_LE_ADV_PARAM is another helper macro:

/**
 * @brief Helper to declare advertising parameters inline
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM(_options, _int_min, _int_max, _peer) \
        ((struct bt_le_adv_param[]) { \
                BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
         })

This teaches you that BT_GAP_ADV_FAST_INT_MIN_2 and BT_GAP_ADV_FAST_INT_MAX_2 are the minimum and maximum advertising interval. You can find their definitions in zephyr/bluetooth/gap.h:

#define BT_GAP_ADV_FAST_INT_MIN_2               0x00a0  /* 100 ms   */
#define BT_GAP_ADV_FAST_INT_MAX_2               0x00f0  /* 150 ms   */

So, your iBeacon will use an advertising interval of between 100 ms and 150 ms.

BT_LE_ADV_PARAM uses another macro in the same header file, BT_LE_ADV_PARAM_INIT, and its definition is:

/**
 * @brief Initialize advertising parameters
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
{ \
        .id = BT_ID_DEFAULT, \
        .sid = 0, \
        .secondary_max_skip = 0, \
        .options = (_options), \
        .interval_min = (_int_min), \
        .interval_max = (_int_max), \
        .peer = (_peer), \
}

All in all, this means that the code starts non-connectable advertising with its default Bluetooth ID, a private address, no special advertising options, and with an advertising interval of between 100 ms and 150 ms.

Summary

When reading source code of Zephyr applications, you can definitely feel discouraged by all these macros at first. However, after you've seen a couple of Zephyr programs, you start to see how they all fit together. The beauty of Zephyr is that its source code is completely open. If you don't understand what a specific macro or function is doing, just look up its declaration in the header file or even in the corresponding C file where it's implemented.

This approach has thought me a lot about developing Bluetooth Low Energy applications with Zephyr. I hope it will be a useful approach to you too.

What is Bluetooth Low Energy?

Bluetooth is a wireless communication standard in the 2.4 GHz Industrial, Scientific, and Medical (ISM) frequency band. These days, if you hear about Bluetooth support in a product, this almost always is Bluetooth Low Energy (BLE). It's a radical departure from the original Bluetooth standard, which is now called Classic Bluetooth.

Bluetooth Low Energy and Classic Bluetooth are actually different protocols. Classic Bluetooth is essentially a wireless version of the traditional serial connection. If you want to print a document, transfer a file or stream audio, you want this to happen as fast as possible. Therefore, the focus of development in Classic Bluetooth was on attaining faster and faster speeds with every new version.

However, Classic Bluetooth wasn't a good fit for devices with low power consumption, for instance those powered by batteries. That's why Nokia adapted the Bluetooth standard to enable it to work in low-power scenarios. In 2006, they released their resulting technology onto the market, dubbed Wibree.

The Bluetooth Special Interest Group (SIG), the organization that maintains the Bluetooth specifications, showed interest in this new development. After consulting with Nokia, they decided to adopt Wibree as part of Bluetooth 4.0, with a new name, Bluetooth Low Energy. Classic Bluetooth remained available for high-throughput applications.

Layered architecture

The Bluetooth Core Specification is more than 3200 pages long. And this is only the core specification; there are many supplemental documents for BLE. However, BLE has a layered architecture. Many end-user applications only use the upper layers, so you don't need to know the details of the architecture's lower layers.

The BLE architecture consists of three main blocks: controller, host, and application.

Controller

This has the lower-level layers: the Physical Layer (PHY), Link Layer (LL) and Direct Test Mode (DTM). These are the layers where the Bluetooth radio does its work. The controller communicates with the outside world using the antenna, in a frequency band around 2.4 GHz. It communicates with the host using a standardized interface between the two blocks: the Host Controller Interface (HCI). [1]

Host

This is the block with which the end user or application developer comes in contact. The Logical Link Control and Adaptation Protocol (L2CAP) defines channels and signaling commands. On top of it, the Security Manager Protocol (SMP) handles secure connections (with authentication and encryption), and the Attribute Protocol (ATT) defines how to expose and access data as attributes. The Generic Attribute Profile (GATT) [2] builds on the Attribute Protocol to define how to discover services and their characteristics and how to read and write their values. The upper layer of the Host block is the Generic Access Profile (GAP), which defines how devices can discover other devices and connect, pair, and bond to them. The host communicates with the controller using its part of the host controller interface, and applications communicate with the host depending on the APIs exposed by the operating system.

Application

This layer builds on top of the Generic Attribute Profile to implement application-specific characteristics, services, and profiles. A characteristic defines a specific type of data, such as an Alert Level. A service defines a set of characteristics and their behaviors, such as the Link Loss Service. A profile is a specification that describes how two or more devices with one or more services communicate with each other. An example is the Proximity profile, which has two roles: Proximity Monitor and Proximity Reporter.

The three blocks don't have to run on the same processor. In fact, there are three common configurations --- one single-chip and two dual-chip:

Single-chip (SoC)

Controller, host and application code run on the same chip. The host and controller communicate through function calls and queues in the chip's RAM. Most simple devices such as BLE sensors use this configuration; it keeps the cost down. Some smartphones also use this configuration if they have a SoC with Bluetooth built in.

Dual-chip over HCI

A dual-chip solution with application and host on one chip, and the controller on another chip, communicates over HCI. Because HCI is a standardized interface, it lets you combine different platforms. For instance, on a Raspberry Pi, the Wi-Fi and BLE chip implements a BLE controller. If you connect a BLE dongle to an older Raspberry Pi, this dongle also implements a BLE controller. [3] BlueZ, the Raspberry Pi Linux kernel's Bluetooth stack, implements a BLE host. So BlueZ communicates with the BLE controller in the built-in BLE chip or the BLE dongle. In the former case, the HCI uses SDIO, and in the latter, UART over USB. [4] Many smartphones and tablets also use the dual-chip over HCI configuration, with a powerful processor running the host and a Bluetooth chip running the controller.

Dual-chip with connectivity device

Another dual-chip solution is one with the application running on one chip and the host and controller on another chip. The latter is then called the connectivity device because it adds BLE connectivity to the other device. This approach is useful if you have an existing hardware device that you want to extend with BLE connectivity. Because there's no standardized interface in this case, the communication between the application processor and the connectivity device needs to make use of a proprietary protocol implemented by the connectivity device.

A three-chip solution with controller, host, and application each running on its own chip is also possible. However, because of the associated cost, this is typically only done for development systems.

How to communicate with BLE devices?

Bluetooth Low Energy has two ways to communicate between devices: with and without a connection.

Without a connection

Without a connection means that the device just broadcasts information in an advertisement. Every BLE device in the neighborhood is able to receive this information.

Some examples of BLE devices broadcasting data are:

Proximity beacons

These devices, often following Apple's iBeacon standard, broadcast their ID. Receivers calculate their approximate distance to the beacons based on the advertisement's Received Signal Strength Indicator (RSSI).

Sensors

Many temperature and humidity sensors broadcast their sensor values. Most devices do this in an unencrypted fashion, but some of them encrypt the data to prevent it being read by every device in the neighborhood.

Mobile phones

After the COVID-19 pandemic started in 2020, Google and Apple collaborated on the Exposure Notifications standard for contact tracing. As part of this technology, Android phones and iPhones broadcast unique (but anonymous) numbers. Other phones can pick up these numbers and use them later to warn users that they have been in contact with someone who is known to have had COVID-19.

With a connection

The other way to communicate between BLE devices is with a connection. One device (the client) scans for BLE advertisements to find the device it wants to connect to. Then, optionally, it may do an active scan to ask the device (the server) which services are offered.

After the client connects to the server, the client can use the server's services. Each BLE service is a container of specific data from the server. You can read this data, or (with some services) write a value to the server.

Some examples of BLE devices using a connection are:

Fitness trackers

Your smartphone can connect to a fitness tracker and read your heart rate, the tracker's battery level, and other measurements.

Sensors

Some environmental sensors let you read their sensor values over a BLE connection.

Proximity reporters

These devices sound an alert when their connection to another device is lost.

Advantages of BLE

Disadvantages of BLE

Platforms used in this book

This book focuses on Bluetooth Low Energy programming on three platforms:

These choices were made in order to demonstrate a wide range of applications compatible with many software and hardware platforms.

Python/Bleak (Raspberry Pi, PC)

Python is an easy-to-use programming language that works on all major operating systems. There are a lot of Python Bluetooth Low Energy libraries, but many of them support only a single operating system. Bleak, which stands for Bluetooth Low Energy platform Agnostic Klient, is a welcome exception. It supports:

• Windows 10, version 16299 (Fall Creators Update) or higher

• Linux distributions with BlueZ 5.43 or higher (also on a Raspberry Pi)

• OS X 10.11 (El Capitan) or macOS 10.12+

Bleak is a GATT client: it's able to connect to BLE devices that act as GATT servers. It supports reading, writing, and getting notifications from GATT servers, and it's also able to discover BLE devices and read advertising data broadcast by them.

Bleak doesn't implement a GATT server. In practice this isn't a big limitation. GATT servers are typically implemented on constrained devices, so, for this purpose, the ESP32 and nRF52 hardware platforms are a better match. [8]

C++/NimBLE-Arduino (ESP32)

If you're looking at microcontrollers, the Arduino framework has become quite popular, not only for the original Arduino boards, which didn't have BLE functionality, but also on ESP32 development boards, which do.

Programming for the Arduino framework is done in a variant of C++, but the framework and many Arduino libraries hide much of C++'s complexity. Even if you only know some C (which is much less complex than C++), you'll be able to use the Arduino framework.

One of the more popular BLE libraries for Arduino on the ESP32 is NimBLE-Arduino. It's a fork of NimBLE, which is part of the Apache Mynewt real-time operating system. With NimBLE-Arduino, you can easily create your own GATT server or client.

C/Zephyr (nRF52)

For even more constrained devices, typically battery-powered, you need a specialized real-time operating system (RTOS). This book uses the Zephyr Project on nRF52840-based devices from Nordic Semiconductor. Zephyr has a completely open-source Bluetooth Low Energy stack.

Zephyr's BLE stack is highly configurable. You can build Zephyr firmware for three configuration types:

Combined build

Builds the BLE controller, BLE host, and your application for a one-chip configuration.

Host build

Builds the BLE host and your application, along with an HCI driver to let your device communicate with an external BLE controller on another chip. [9]

Controller build

Builds the BLE controller with an HCI driver to let your device communicate with an external BLE host on another chip.

With some basic knowledge of C, you can create your own BLE devices with Zephyr, such as BLE beacons, sensor boards, and proximity reporters. Zephyr has extensive documentation of its Bluetooth API, as well as a lot of ready-to-use examples that you can build upon.

Recommended hardware

You need some (affordable) hardware for the examples in this book:

• The Python examples are hardware-agnostic due to the use of Bleak. So, whether you want to use a Raspberry Pi or another Linux computer, a Windows computer, or a Mac, it won't make much difference.

• For the NimBLE-Arduino examples, any ESP32 development board will suffice. [10]

• For the Zephyr examples, I recommend a Nordic Semiconductor nRF52840 Dongle, but you can use any BLE microcontroller supported by Zephyr. Some other manufacturers have variants of the dongle, such as the April USB Dongle 52840 (with an external antenna) or the nRF52840 MDK USB Dongle from makerdiary (with a case).

If you're really serious about BLE development with Zephyr, I also recommend Nordic Semiconductor's nRF52840 Development Kit. This is the bigger (and slightly less affordable) brother of the nRF52840 Dongle, offering more capabilities to debug your code easily. If you have one of these, you can do your initial development on it, and move your code to the nRF52840 Dongle when you've solved the most complex issues.

So, at the minimum, you need your computer, one ESP32 development board and one BLE microcontroller supported by Zephyr.

Configuring instead of programming

The ESP8266 and its successor, the ESP32, are a series of low-cost microcontrollers with integrated Wi-Fi (for both series) and Bluetooth (for the ESP32), produced by Espressif Systems. The maker community quickly adopted these microcontrollers for tasks where an Arduino didn't suffice. [1]

You can program the ESP8266 and ESP32 using Espressif's ESP-IDF SDK, the ESP32 Arduino core, or MicroPython. Arduino and MicroPython lower the bar significantly, but it still takes some programming experience to build solutions with these microcontrollers.

One of the domains in which the ESP8266 and ESP32 have become popular is in the DIY (do-it-yourself) home automation scene. You just have to connect a sensor, switch, LED, or display to a microcontroller board, program it, and there you have it: your customised home automation device.

However, "program it" isn't that straightforward as it sounds. For instance, if you're using the Arduino environment, which has a lot of easy-to-use libraries, you still have to know your way around C++.

Luckily there are a couple of projects to make it easier to create firmware for ESP8266 or ESP32 devices for home automation. One of these is ESPHome.

On its homepage, the ESPHome developers describe it as: "ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems."

The fundamental idea of ESPHome is that you don't program your ESP8266 or ESP32 device, but configure it. Often you only have to configure which pins you have connected to a component, such as a sensor. You don't have to initialize the sensor, read its values in a loop, and process them.

Configuration is a completely different mindset than programming. It lowers the bar even more. With ESPHome, everyone can make home automation devices. [2]

Essentially ESPHome creates C++ code based on your configuration. The process looks like this:

So when you write a YAML file with your device's configuration, ESPHome generates C++ code from it. More specifically, ESPHome creates a PlatformIO project using the Arduino framework. PlatformIO then builds the C++ code, and esptool uploads the resulting firmware to your device.

You don't have to know anything about what's happening under the hood. You just have to write the configuration of your device in a YAML file and memorise a small number of commands to let ESPHome do the rest.

The advantages of ESPHome

Why use ESPHome? The first reason is clear from the project's description: because you don't need to be able to program. Even if you're a programmer, ESPHome offers many advantages:

Works completely locally

Many commercial Wi-Fi-based home automation devices need a connection to a cloud service of the manufacturer. In contrast, ESPHome devices work locally and can communicate with a local home automation system such as Home Assistant or an MQTT-based home automation system.

Offers on-device automations

Many home automation systems use a central gateway that contains all the logic, with automations like "if the sun goes down, close the blinds." In contrast, ESPHome offers powerful on-device automations. Your devices can work independently from a home automation gateway, so they keep working if they lose Wi-Fi access or if your home automation gateway crashes.

Offers over-the-air updates

ESPHome includes out-of-the-box over-the-air (OTA) update functionality. This makes it easy to centrally manage your ESPHome devices and update the firmware. This means you don't have to go around your house with your laptop to connect a serial cable to each device and flash the firmware.

Supports a lot of components

ESPHome supports many components out-of-the-box: several types of sensors, switches, and displays (even e-paper displays) are available with just a couple of configuration lines. The list of supported components is growing with every release.

Has extensive documentation

The developers have documented every component in ESPHome, and this documentation is quite good.

Is customisable

Although you create ESPHome firmware by writing a configuration file, ESPHome doesn't hide anything from you. It's still possible to add custom components that you write in C++. You can even look at the C++ code that ESPHome generates and change it.

What hardware do you need?

ESPHome creates custom firmware for the ESP8266 and ESP32 microcontrollers, so you need one of these. There are many types of boards for both microcontrollers, varying in the amount of flash memory, RAM, and available pins. Some of them even come with extras such as a built-in display (OLED, TFT, or e-paper), battery, or camera.

ESPHome doesn't support all features of all boards out-of-the-box. Technically, all ESP8266/ESP32 devices should be able to run ESPHome. Some features just aren't supported yet.

Your first choice is between the ESP8266 or ESP32. If you're buying a device at present, the choice is simple: the ESP32. It is much more capable than its predecessor and has a faster processor, more memory, more peripherals, and adds Bluetooth.

Then comes the choice of board. Espressif has some development boards. Many other companies are making them too. There are even complete kits such as the M5Stack series. These are ESP32 development boards ready to use in your living room in a case with a display, buttons, MicroSD card slot, and speaker. These are currently my favourite devices to use with ESPHome. If you don't need something with a nice finish as the M5Stack devices, the TTGO T-Display ESP32 made by LilyGO is my development board of choice: it has an integrated 1.14 inch TFT display.

Other interesting devices to run ESPHome on are devices from manufacturers such as Sonoff and Shelly. These come with firmware that works with the manufacturer's cloud services. You can however replace the firmware with ESPHome. This unlocks the full potential of these devices and lets you use them in your local home automation system without any link to a cloud system.

What software do you need?

You can use ESPHome to create a fully autonomous microcontroller project --- for example, a plant monitor that turns on an LED if the plant's soil is too dry. However, if you don't publish the plant's status over the network, this would be a waste of the ESP32's capabilities. The main usage cases of ESPHome are:

• To send a device's sensor measurements to a home automation gateway.

• To remotely control a device's lights or switches from a home automation gateway.

ESPHome supports two ways of communication between your device and the home automation gateway:

Native API

The ESPHome native API is a highly optimised network protocol using Google's protocol buffers. It's meant to be used with Home Assistant --- a popular open-source home automation system.

MQTT

MQTT (Message Queuing Telemetry Transport) is an OASIS standard messaging protocol designed with a lightweight publish/subscribe approach for messages. All your ESPHome devices then communicate with an MQTT broker such as Eclipse Mosquitto.

From the beginning (when it was still called esphomeyaml), the ESPHome project has been tightly integrated with Home Assistant, so the ESPHome developers prefer the native API. However, MQTT is fully supported, allowing your devices to communicate with many other home automation gateways, as MQTT is a popular standard. [3]

Do you need a development environment?

With ESPHome you don't program your devices but configure them. However, you still need something that looks like a "development" environment. When your device configurations are simple, you could do without, but the more complex they become, you'll need all the help you can get.

This doesn't mean you have to install a full-blown Integrated Development Environment (IDE). You should only need a couple of programs:

An editor

You could make do with a simple text editor such as Notepad (Windows), TextEdit (macOS), or the default text editor on your Linux distribution. However, having an editor with syntax highlighting for YAML files is easier. Some examples are Notepad++ and Sublime Text. If you're a command-line user on Linux, both vim and Emacs work fine. [4] Use whatever you like, because your editor is an important tool to work with ESPHome.

A YAML linter

A linter is a program that checks your file for the correct syntax. An editor with syntax highlighting has this linter built-in, but you can also run this standalone. A good YAML linter is the Python program yamllint. Not only does it check for syntax validity, but also weird things like key repetitions, as well as cosmetic problems such as line length, trailing spaces, and inconsistent indentation. ESPHome includes its own linter, specifically targeted at finding errors in ESPHome configurations. Both linters are complementary.

If you're used to developing in an IDE, an interesting alternative is the ESPHome plugin for Visual Studio Code. [5] This plugin provides validation and completion of what you type in an ESPHome YAML file. It also shows tooltips with help when you hover over keywords in the configuration.

Secure

Home automation and IoT devices are notoriously insecure. At the Usenix Security Conference 2019, the Czech security software company Avast and Stanford University presented their research of household IoT devices. Avast scanned 83 million IoT devices in 16 million homes around the world of people who agreed to share these data. The results of the study published in "All Things Considered: An Analysis of IoT Devices on Home Networks" were staggering:

• 7% percent of all IoT devices support an obsolete, insecure, and completely unencrypted protocol such as Telnet or FTP.

• Of these, 17% exhibit weak FTP passwords, and 2% have weak Telnet passwords.

• Surveillance cameras have the weakest Telnet profile, with more than 10% of them that support Telnet with weak credentials.

• 3% percent of the homes are externally visible on the internet and more than half of those have a known vulnerability or a weak password.

This is not an isolated study. Not a week goes by without some news items about insecure devices, most of the time because basic security measures such as strong passwords are not enforced by the manufacturer or basic programming errors have been made. To give you an idea about what can happen: in 2018 nude videos of the Dutch women's handball team appeared on a popular porn website because the surveillance cameras of the dressing room of a sauna were broken into. Imagine if someone can access your baby monitor with a camera or your security camera in your living room or bedroom...

So what can you do to secure your home automation system? If you choose an off-the-shelf system: not much. You fully rely on the manufacturer's ability to create a secure system and the goodwill to keep supplying patches that solve security issues that have been discovered. And the home automation and IoT industries have clearly shown they are not up to the task. This is one of the reasons why I prefer open source software. Not because it is always secure, but because the transparency of the open source development process forces developers to create more secure software.

Modular

There are many competing standards and communication protocols for home automation. Unfortunately, many off-the-shelf home automation gateways support only a small subset of these protocols or even use a proprietary protocol that locks you into using devices of the same manufacturer. That severely limits your choice of products.

You can't know which protocols will become popular in a few years, and maybe you like one product that uses Z-Wave and another product that uses Zigbee. It should be easy to interconnect these devices, even when they use different protocols.

So how can you do this? Many of the wireless communication protocols for home automation need a dedicated transceiver because they work on a specific radio frequency. That's where the Raspberry Pi shines: you can start with a basic Raspberry Pi setup supporting only IoT devices that are communicating over Wi-Fi and Bluetooth, add an RTL-SDR USB dongle to read the measurements of your 433.92 MHz weather sensors, later add a Z-Wave HAT on the board when you start adding Z-Wave sensors to your house and then add a Zigbee USB transceiver when you want to control some Zigbee lights.

Modularity is also important for software. There's a lot of user-friendly software to make your Raspberry Pi a home automation gateway. So you just install something like Home Assistant, openHAB or Domoticz on your Raspberry Pi and that's it: you have a gateway that supports tons of devices. Some of these systems are very modular and extensible, others aren't. Many of them support MQTT (Message Queuing Telemetry Transport), a common language to exchange messages.

MQTT has become the standard for interoperability between various home automation devices. For instance, if your home automation gateway of choice doesn't support Zigbee but it does support MQTT, then you only have to run the Zigbee2mqtt software, which translates the Zigbee protocol to MQTT messages. Your gateway can then talk to your Zigbee devices using MQTT.

Modularity also means that you don't have to have one gateway. You can perfectly have your main gateway in your basement, but install a second gateway with your 433.92 MHz receiver for your environmental sensors in your living room because that gives you better coverage to receive data from these wireless sensors. If you're using MQTT, that's very simple to implement: you just relay the sensor readings that your gateway in the living room receives to your MQTT broker, after which all the other components of your home automation setup can access the readings in the MQTT format.

In short: a good modular home automation system means that you can mix and match the devices that you like, irrespective of their protocol, and you can use the software and hardware components of your choice, in various locations in your house.

Open source

If you buy an off-the-shelf home automation gateway, you generally don't get access to its source code, so you cannot peek into it to see what it does or to assess its quality. You just have to believe the manufacturer on his word. Is that enough for you if it's about software that will get to know you intimately because it processes sensor readings and even camera images about you in your home? Not for me, so open source software is a hard requirement for my home automation setup.

But what when you're not a programmer and you wouldn't even understand the source code of your home automation system if you had access to it? Even then the use of open source software has a lot of advantages. It's not because you don't have the programming experience that others can't help. Most open source projects have a decentralized software development model that encourages open collaboration.

So if you find a bug in the software or see something wrong in its source code but don't have the programming experience needed to fix it, just report the bug on the issue tracker of the project, and hopefully, someone else in the project's community will step in and fix it. It all depends on the health of the project's community. But a good open source project has a vibrant community of developers and users who collaborate to continuously improve the software.

And open source doesn't just mean getting access to the source code, it's much more than that. For instance, when you know a specific software project is open source, you know that it doesn't arbitrarily restrict what you can do with it. The Open Source Definition even explicitly lists that the license of open source software must not discriminate against any person or group of persons, nor restrict anyone from making use of the program in a specific field of endeavour.

The decentralization and transparency of the open source development model gives power back to the users, where it belongs. For home automation that's even more important: I don't want a company having control over my house.

Self-sufficient

During the last ten years, there has been a worrying development in the computer industry: we all depend more and more on (centralized) cloud systems. Unfortunately, the home automation industry didn't escape this fate. Many popular home automation and IoT systems depend on a cloud server. Some examples:

• the Ring video doorbell with Wi-Fi camera;

• the Nest Learning Thermostat;

• so-called 'smart speakers' running voice assistants, like Amazon Echo and Google Home;

• the IFTTT service that links various other services.

This isn't without its problems. In the last couple of years some of these cloud services for home automation stopped working for their users:

• In 2014 Nest bought the company that was selling the Revolv Hub home automation system, not long after Nest itself was acquired by Google. In 2016 Nest shut down the servers Revolv Hub depended on, after announcing it with a quiet note on the website of Revolv a few months earlier. That meant that the $300 Revolv Hub ceased functioning entirely.

• At the end of 2019, Best Buy announced that several of their Insignia-branded smart devices would stop working because they decided to shut down the corresponding backend systems.

• In May 2020 Wink (with the catchphrase "A simpler way to a smarter home") announced with just a week's notice that it would start charging a monthly fee for the use of their services. Users that didn't want to pay were no longer be able to access their Wink devices and their automations were disabled. The Wink Hub was rendered useless, although it had been in stores with the clear description "no required monthly fees, ever". Ironically, the announcement ended with the message "Our user community is integral to Wink, and we want to continue to be your trusted smart home provider." Yeah, right.

But fundamentally, the problem lies even deeper. It just makes no sense to use cloud services to automate your home. Home automation comes down to: you want one device to be able to respond to another device in your home. For instance: the motion sensor in your bathroom detects motion at night, and this turns the bathroom light on for five minutes. If you use IFTTT to link both devices, the motion sensor has to send a message to the internet, IFTTT relays the message to your bathroom light (for instance a Philips Hue light), and the bathroom light turns on.

But there's no need for an internet service like IFTTT between these devices because both devices are in your home, so it makes much more sense to link them locally, using a server at home. This could be a Raspberry Pi running home automation software that doesn't need a cloud server to function, but does all its processing on-device (or on the edge, as it's called now). You can perfectly do this with Node-RED or Home Assistant. Then there's no way a company can render your home automation system useless by shutting down their service, or your bathroom light doesn't turn on at night because your internet connection or IFTTT's servers are down. You're fully in control of your home automation system.

There's another risk: using cloud services for your home automation system invades your privacy. Look at some of the privacy issues with the services I talked about above:

• If you use the Ring video doorbell, it sends a video of everyone that steps on your porch to the manufacturer. The Ring company (which has been bought by Amazon in 2018) has a questionable approach to privacy: in January 2019 it was uncovered that employees have access to the video recordings of all Ring devices and even that the data are stored unencrypted.

• If you use the Nest Learning Thermostat, Google knows precisely when you are home and when you aren't.

• If you run a smart speaker like the Amazon Echo, what you tell your house members will be sent countless times inadvertently to Amazon because the Echo thinks it has heard its wake word. Moreover, Amazon's employees listen to a part of all 'conversations' with the Echo to improve its algorithms.

• If you use the IFTTT service to link your various other services, you give one company access to all your home automation services, which is too much power concentrated in one company's hands: they can see exactly what you're doing.

After reading all this, do you still find it acceptable to use cloud services for your home automation?