Posts about book

Building wireless sensor networks with OpenThread, CoAP, and Zephyr

Last week my new book has been published, Building Wireless Sensor Networks with OpenThread: Developing CoAP Applications for Thread Networks with Zephyr.

Thread is a protocol for building efficient, secure and scalable wireless mesh networks for the Internet of Things (IoT), based on IPv6. OpenThread is an open-source implementation of the Thread protocol, originally developed by Google. It offers a comprehensive Application Programming Interface (API) that is both operating system and platform agnostic. OpenThread is the industry’s reference implementation and the go-to platform for professional Thread application developers.

This book uses OpenThread in conjunction with Zephyr, an open-source real-time operating system designed for use with resource-constrained devices. This allows you to develop Thread applications that work on various hardware platforms, without the need to delve into low-level details or to learn another API when switching hardware platforms.

With its practical approach, this book not only explains theoretical concepts, but also demonstrates Thread’s network features with the use of practical examples. It explains the code used for a variety of basic Thread applications based on Constrained Application Protocol (CoAP). This includes advanced topics such as service discovery and security. As you work through this book, you’ll build on both your knowledge and skills. By the time you finish the final chapter, you’ll have the confidence to effectively implement Thread-based wireless networks for your own IoT projects.

What is Thread?

Thread is a low-power, wireless, and IPv6-based networking protocol designed specifically for IoT devices. Development of the protocol is managed by the Thread Group, an alliance founded in 2015. This is a consortium of major industry players, including Google, Apple, Amazon, Qualcomm, Silicon Labs, and Nordic Semiconductor.

You can download the Thread specification for free, although registration is required. The version history is as follows:

Versions of the Thread specification

Version

Release year

Remarks

Thread 1.0

2015

Never implemented in commercial products

Thread 1.1

2017

  • The ability to automatically move to a clear channel on detecting interference (channel agility)

  • The ability to reset a master key and drive new rotating keys in the network

Thread 1.2

2019

  • Enhancements to scalability and energy efficiency

  • Support for large-scale networking applications, including the ability to integrate multiple Thread networks into one singular Thread domain

Thread 1.3

2023

  • Enhancements to scalability, reliability, and robustness of Thread networks

  • Standardization of Thread Border Routers, Matter support, and firmware upgrades for Thread devices

All versions of the Thread specification maintain backward compatibility.

Thread’s architecture

Thread employs a mesh networking architecture that allows devices to communicate directly with each other, eliminating the need for a central hub or gateway. This architecture offers inherent redundancy, as devices can relay data using multiple paths, ensuring increased reliability in case of any single node failure.

The Thread protocol stack is built upon the widely used IPv6 protocol, which simplifies the integration of Thread networks into existing IP infrastructures. Thread is designed to be lightweight, streamlined and efficient, making it an excellent protocol for IoT devices running in resource-constrained environments.

Before discussing Thread, it’s essential to understand its place in the network protocol environment. Most modern networks are based on the Internet Protocol suite, which uses a four-layer architecture. The layers of the Internet Protocol suite are, from bottom to top:

Link layer

Defines the device’s connection with a local network. Protocols like Ethernet and Wi-Fi operate within this layer. In the more complex Open Systems Interconnection (OSI) model, this layer includes the physical and data link layers.

Network layer

Enables communication across network boundaries, known as routing. The Internet Protocol (IP) operates within this layer and defines IP addresses.

Transport layer

Enables communication between two devices, either on the same network or on different networks with routers in between. The transport layer also defines the concept of a port. User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) operate within this layer.

Application layer

Facilitates communication between applications on the same device or different devices. Protocols like HyperText Transfer Protocol (HTTP), Domain Name System (DNS), Simple Mail Transfer Protocol (SMTP), and Message Queuing Telemetry Transport (MQTT) operate within this layer.

When data is transmitted across the network, the data of each layer is embedded in the layer below it, as shown in this diagram:

/images/internet-protocol-encapsulation.png

Network data is encapsulated in the four layers of the Internet Protocol suite (based on: Colin Burnett, CC BY-SA 3.0)

Thread operates in the network and transport layers. However, it’s important to know what’s going on in all layers.

Network layer: 6LoWPAN and IPv6

Thread’s network layer consists of IPv6 over Low-Power Wireless Access Networks (6LoWPAN). This is an IETF specification described in RFC 4944, "Transmission of IPv6 Packets over IEEE 802.15.4 Networks", with an update for the compression mechanism in RFC 6282, "Compression Format for IPv6 Datagrams over IEEE 802.15.4-Based Networks". 6LoWPAN’s purpose is to allow even the smallest devices with limited processing power and low energy requirements to be part of the Internet of Things.

6LoWPAN essentially enables you to run an IPv6 network over the IEEE 802.15.4 link layer. It acts as an ‘adaptation layer’ between IPv6 and IEEE 802.15.4 and is therefore sometimes considered part of the link layer. This adaptation poses some challenges. For example, IPv6 mandates that all links can handle datagram sizes of at least 1280 bytes, but IEEE 802.15.4, as explained in the previous subsection, limits a frame to 127 bytes. 6LoWPAN solves this by excluding information in the 6LoWPAN header that can already be derived from IEEE 802.15.4 frames and by using local, shortened addresses. If the payload is still too large, it’s divided into fragments.

The link-local IPv6 addresses of 6LoWPAN devices are derived from the IEEE 802.15.4 EUI-64 addresses and their shortened 16-bit addresses. The devices in a 6LoWPAN network can communicate directly with ‘normal’ IPv6 devices in a network if connected via an edge router. This means there is no need for translation via a gateway, in contrast to non-IP networks such as Zigbee or Z-Wave. Communication with non-6LoWPAN networks purely involves forwarding data packets at the network layer.

In this layer, Thread builds upon IEEE 802.15.4 to create an IPv6-based mesh network. In IEEE 802.15.4 only communication between devices that are in immediate radio range is possible, whereas routing allows devices that aren’t in immediate range to communicate.

If you want to delve into more details of Thread’s network layer, consult the Thread Group’s white paper Thread Usage of 6LoWPAN.

Transport layer: UDP

On top of 6LoWPAN and IPv6, Thread’s transport layer employs the User Datagram Protocol (UDP), which is the lesser known alternative to Transmission Control Protocol (TCP).

UDP has the following properties:

Connectionless

Unlike TCP, UDP doesn’t require establishing a connection before transferring data. It sends datagrams (packets) directly without any prior setup.

No error checking and recovery

UDP doesn’t provide built-in error checking, nor does it ensure the delivery of packets. If data is lost or corrupted during transmission, UDP doesn’t attempt to recover or resend it.

Speed

UDP is faster than TCP, as it doesn’t involve the overhead of establishing and maintaining a connection, error checking, or guaranteeing packet delivery.

UDP is ideal for use by Thread for several reasons:

Resource constraints

Thread devices often have limited processing power, memory, and energy. The simplicity and low overhead of UDP make it a better fitting choice for such resource-constrained devices compared to TCP.

Lower latency

UDP’s connectionless and lightweight nature guarantees data transmission with low latency, making it appropriate for wireless sensors.

Thread lacks an application layer

Unlike home automation protocols such as Zigbee, Z-Wave, or Matter, the Thread standard doesn’t define an application layer. A Thread network merely offers network infrastructure that applications can use to communicate. Just as your web browser and email client use TCP/IP over Ethernet or Wi-Fi, home automation devices can use Thread over IEEE 802.15.4 as their network infrastructure.

Several application layers exist that can make use of Thread:

Constrained Application Protocol (CoAP)

A simpler version of HTTP, designed for resource-constrained devices and using UDP instead of TCP.

MQTT for Sensor Networks (MQTT-SN)

A variant of MQTT designed to be used over UDP.

Apple HomeKit

Apple’s application protocol for home automation devices.

Matter

A new home automation standard developed by the Connectivity Standards Alliance (CSA).

Like your home network which simultaneously hosts a lot of application protocols including HTTP, DNS, and SMTP, a Thread network can also run all these application protocols concurrently.

/images/thread-stack.png

The Thread network stack supports multiple application protocols simultaneously.

Throughout this book, I will be using CoAP as an application protocol in a Thread network. Once you have a Thread network set up, you can also use it for Apple HomeKit and Matter devices.

Advantages of Thread

Some of Thread’s key benefits include:

Scalability

Thread’s mesh network architecture allows for large-scale deployment of IoT devices, supporting hundreds of devices within a single network. Thread accommodates up to 32 Routers per network and up to 511 End Devices per Router. In addition, from Thread 1.2, multiple Thread networks can be integrated into one single Thread domain, allowing thousands of devices within a mesh network.

Security

Devices can’t access a Thread network without authorization. Moreover, all communication on a Thread network is encrypted using IEEE 802.15.4 security mechanisms. As a result, an outsider without access to the network credentials can’t read network traffic.

Reliability

Thread networks are robust and support self-healing and self-organizing network properties at various levels, ensuring the network’s resilience against failures. This all happens transparently to the user; messages are automatically routed around any bad node via alternative paths.

For example, an End Device requires a parent Router to communicate with the rest of the network. If communication with this parent Router fails for any reason and it becomes unavailable the End Device will choose another parent Router in its neighborhood, after which communication resumes.

Thread Routers also relay packets from their End Devices to other Routers using the most efficient route they can find. In case of connection problems, the Router will immediately seek an alternate route. Router-Eligible End Devices can also temporarily upgrade their status to Routers if necessary. Each Thread network has a Leader who supervises the Routers and whose role is dynamically selected by the Routers. If the Leader fails, another Router automatically takes over as Leader.

Border Routers have the same built-in resilience. In a Thread network with multiple Border Routers, communication between Thread devices and devices on another IP network (such as a home network) occurs along multiple routes. If one Border Router loses connectivity, communications will be rerouted via the other Border Routers. If your Thread network only has a single Border Router, this will become a single point of failure for communication with the outside network.

Low power consumption

Thread is based on the power-efficient IEEE 802.15.4 link layer. This enables devices to operate on batteries for prolonged periods. Sleepy End Devices will also switch off their radio during idle periods and only wake up periodically to communicate with their parent Router, thereby giving even longer battery life.

Interoperability

As the protocol is built upon IPv6, Thread devices are straightforward to incorporate into existing networks, both in industrial and home infrastructure, and for both local networks and cloud connections. You don’t need a proprietary gateway; every IP-based device is able to communicate with Thread devices, as long as there’s a route between both devices facilitated by a Border Router.

Disadvantages of Thread

In life, they say there’s no such thing as a free lunch and Thread is no exception to this rule. Some of its limitations are:

Limited range

Thread’s emphasis on low power consumption can lead to a restricted wireless range compared with other technologies such as Wi-Fi. Although this can be offset by its mesh architecture, where Routers relay messages, it still means that devices in general can’t be placed too far apart.

Low data rates

Thread supports lower data rates than some rival IoT technologies. This makes Thread unsuitable for applications requiring high throughput.

Complexity

The Thread protocol stack may be more challenging to implement compared to other IoT networking solutions, potentially ramping up development costs and time.

Platforms used in this book

In this book I focus on building wireless sensor networks using the Thread protocol together with the following hardware and software environments:

Nordic Semiconductor’s nRF52840 SoC

A powerful yet energy-efficient and low cost hardware solution for Thread-based IoT devices

OpenThread

An open-source implementation of the Thread protocol, originally developed by Google, making it easy to incorporate Thread into your IoT projects

Zephyr

An open-source real-time operating system designed for resource-constrained devices

Armed with these tools and by using practical examples I will go on to guide you step-by-step through the details of the hardware and software required to build Thread networks and OpenThread-based applications.

Nordic Semiconductor’s nRF52840 SoC

Nordic Semiconductor’s nRF52840 is a SoC built around the 32-bit ARM Cortex-M4 CPU running at 64 MHz. This chip supports Bluetooth Low Energy (BLE), Bluetooth Mesh, Thread, Zigbee, IEEE 802.15.4, ANT and 2.4 GHz proprietary stacks. With its 1 MB flash storage and 256 KB RAM, it offers ample resources for advanced Thread applications.

This SoC is available for developers in the form of Nordic Semiconductor’s user-friendly nRF52840 Dongle. You can power and program this small, low-cost USB dongle via a computer USB port. The documentation lists comprehensive information about the dongle.

/images/nrf52840-dongle.png

Nordic Semiconductor’s nRF52840 Dongle is a low-cost USB dongle ideal for experimenting with Thread. (image source: Nordic Semiconductor)

OpenThread

Thread is basically a networking protocol; in order to work with it you need an implementation of the Thread networking protocol. The industry’s reference implementation, used even by professional Thread device developers, is OpenThread. It’s a BSD-licensed implementation, with development ongoing via its GitHub repository. The license for this software allows for its use in both open-source and proprietary applications.

OpenThread provides an Application Programming Interface (API) that’s operating system and platform agnostic. It has a narrow platform abstraction layer to achieve this, and has a small memory footprint, making it highly portable. In this book, I will be using OpenThread in conjunction with Zephyr, but the same API can be used in other combinations, such as ESP-IDF for Espressif’s Thread SoCs.

/images/openthread-website.png

The BSD-licensed OpenThread project is the industry’s standard implementation of the Thread protocol.

Zephyr

Zephyr is an open-source real-time operating system (RTOS) designed for resource-constrained devices. It incorporates OpenThread as a module, simplifying the process of creating Thread applications based on Zephyr and OpenThread. Because of Zephyr’s hardware abstraction layer, these applications will run on all SoCs supported by Zephyr that have an IEEE 802.15.4 radio facility.

/images/zephyr-website.png

Zephyr is an open-source real-time operating system (RTOS) with excellent support for OpenThread.

Developing Bluetooth Low Energy applications with Zephyr RTOS: Dissecting an iBeacon example

When I started learning how to develop Bluetooth Low Energy (BLE) applications with Zephyr RTOS, I was immediately impressed by the excellent documentation and the wealth of sample code of this open source real-time operating system. But what really got me understand the way of working with BLE on Zephyr was just going through each line of code in a sample and find out the corresponding definition in Zephyr's header files, which are surprisingly readable.

As an exercise in this approach, I want to go through the iBeacon sample application with you. You'll find that this deceptively simple application already hides a lot of details that are important to know before starting with more complex Zephyr code.

Here's the sample code to broadcast an iBeacon advertisement:

/*
 * Copyright (c) 2018 Henrik Brix Andersen <henrik@brixandersen.dk>
 *
 * SPDX-License-Identifier: Apache-2.0
 */

#include <zephyr/types.h>
#include <stddef.h>
#include <zephyr/sys/printk.h>
#include <zephyr/sys/util.h>

#include <zephyr/bluetooth/bluetooth.h>
#include <zephyr/bluetooth/hci.h>

#ifndef IBEACON_RSSI
#define IBEACON_RSSI 0xc8
#endif

/*
 * Set iBeacon demo advertisement data. These values are for
 * demonstration only and must be changed for production environments!
 *
 * UUID:  18ee1516-016b-4bec-ad96-bcb96d166e97
 * Major: 0
 * Minor: 0
 * RSSI:  -56 dBm
 */
static const struct bt_data ad[] = {
    BT_DATA_BYTES(BT_DATA_FLAGS, BT_LE_AD_NO_BREDR),
    BT_DATA_BYTES(BT_DATA_MANUFACTURER_DATA,
              0x4c, 0x00, /* Apple */
              0x02, 0x15, /* iBeacon */
              0x18, 0xee, 0x15, 0x16, /* UUID[15..12] */
              0x01, 0x6b, /* UUID[11..10] */
              0x4b, 0xec, /* UUID[9..8] */
              0xad, 0x96, /* UUID[7..6] */
              0xbc, 0xb9, 0x6d, 0x16, 0x6e, 0x97, /* UUID[5..0] */
              0x00, 0x00, /* Major */
              0x00, 0x00, /* Minor */
              IBEACON_RSSI) /* Calibrated RSSI @ 1m */
};

static void bt_ready(int err)
{
    if (err) {
        printk("Bluetooth init failed (err %d)\n", err);
        return;
    }

    printk("Bluetooth initialized\n");

    /* Start advertising */
    err = bt_le_adv_start(BT_LE_ADV_NCONN, ad, ARRAY_SIZE(ad),
                  NULL, 0);
    if (err) {
        printk("Advertising failed to start (err %d)\n", err);
        return;
    }

    printk("iBeacon started\n");
}

void main(void)
{
    int err;

    printk("Starting iBeacon Demo\n");

    /* Initialize the Bluetooth Subsystem */
    err = bt_enable(bt_ready);
    if (err) {
        printk("Bluetooth init failed (err %d)\n", err);
    }
}

First, this includes a couple of header files. Then the code defines the IBEACON_RSSI constant, which sets the measured power in the iBeacon advertisement. The value 0xc8 is equal to decimal value 200. As this field in the iBeacon specification's advertising data structure is encoded as a signed integer, this is interpreted as 200 - 256 = -56 dBm.

Advertising data structures in Zephyr

Then, you see the definition of an array of struct bt_data elements. The bt_data struct describes an advertising data structure with a type, length, and a pointer to the data. Its definition (in zephyr/bluetooth/bluetooth.h) is:

/** Description of different data types that can be encoded into
 * advertising data. Used to form arrays that are passed to the
 * bt_le_adv_start() function.
 */
struct bt_data {
        uint8_t type;
        uint8_t data_len;
        const uint8_t *data;
};

In the iBeacon program, the array consists of two elements. This is consistent with the iBeacon specification, which shows that an iBeacon advertising packet consists of two advertising structures: flags and manufacturer-specific data.

In principle, you could fill the array with bt_data structs you create yourself, but Zephyr defines some helper macros. Their definition (again, from zephyr/bluetooth/bluetooth.h) looks like this:

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _data Pointer to the data field payload
 * @param _data_len Number of bytes behind the _data pointer
 */
#define BT_DATA(_type, _data, _data_len) \
        { \
                .type = (_type), \
                .data_len = (_data_len), \
                .data = (const uint8_t *)(_data), \
        }

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _bytes Variable number of single-byte parameters
 */
#define BT_DATA_BYTES(_type, _bytes...) \
        BT_DATA(_type, ((uint8_t []) { _bytes }), \
                sizeof((uint8_t []) { _bytes }))

So, with the BT_DATA macro, you construct a bt_data struct with the type, data, and data length you specify (using a pointer to the data). BT_DATA_BYTES is a macro that uses the BT_DATA macro and automatically fills it with the right length for data with a known size.

If you return to the iBeacon program, you see that it fills the array with two elements using the BT_DATA_BYTES macro.

The first element is of type BT_DATA_FLAGS and with data BT_LE_AD_NO_BREDR. You can find the definition of these constants in zephyr/bluetooth/gap.h (GAP stands for Generic Access Profile). For non-connectable advertising packets, the flags data structure is optional.

The second element is of type BT_DATA_MANUFACTURER_DATA. The data used to construct this element follows the format listed in the iBeacon specification: Apple's two-byte company ID, the type (0x02), and length (0x15 or 21 bytes) for the iBeacon data type, and then the UUID, major, minor, and measured power.

All in all, the result of all these macros is an array with advertising data structures that looks like this, schematically:

/images/zephyr-ibeacon-bt-data.png

An iBeacon packet's advertisement data in Zephyr

Enabling Bluetooth

Until now, the code has been all about setting up the right data structures to advertise. Let's have a look at the main() function. This actually does just one thing: calling the bt_enable() function. You have to call this function before any other calls that use the board's Bluetooth hardware. It initializes the Bluetooth subsystem and returns 0 on success and a negative error code otherwise.

The argument to the bt_enable() function is a callback function that's called when initializing Bluetooth is completed. It's in this callback function, bt_ready(), that the core of the program resides.

Advertising

The callback function is called with one argument -- an error code that is 0 on success, or an error code consisting of a negative value otherwise. So, in bt_ready(), you first check whether Bluetooth initialization failed, and, if not, you start advertising.

The definition of bt_le_adv_start() (still in zephyr/bluetooth/bluetooth.h), the function used to start advertising, is:

/** @brief Start advertising
 *
 * Set advertisement data, scan response data, advertisement parameters
 * and start advertising.
 *
 * When the advertisement parameter peer address has been set the advertising
 * will be directed to the peer. In this case advertisement data and scan
 * response data parameters are ignored. If the mode is high duty cycle
 * the timeout will be @ref BT_GAP_ADV_HIGH_DUTY_CYCLE_MAX_TIMEOUT.
 *
 * @param param Advertising parameters.
 * @param ad Data to be used in advertisement packets.
 * @param ad_len Number of elements in ad
 * @param sd Data to be used in scan response packets.
 * @param sd_len Number of elements in sd
 *
 * @return Zero on success or (negative) error code otherwise.
 * @return -ENOMEM No free connection objects available for connectable
 *                 advertiser.
 * @return -ECONNREFUSED When connectable advertising is requested and there
 *                       is already maximum number of connections established
 *                       in the controller.
 *                       This error code is only guaranteed when using Zephyr
 *                       controller, for other controllers code returned in
 *                       this case may be -EIO.
 */
int bt_le_adv_start(const struct bt_le_adv_param *param,
                    const struct bt_data *ad, size_t ad_len,
                    const struct bt_data *sd, size_t sd_len);

So, to call this function, you need to supply advertising parameters, the advertising data structure's array and its length, as well as the array of data structures for the scan response packet and its length.

The iBeacon example program doesn't use any scan response data, so the last two arguments are NULL and 0. The array of advertising data structures is ad, which you constructed in the beginning of the code. With ARRAY_SIZE, a macro defined in zephyr/sys/util.h, you compute the number of elements in the array. When using a C compiler, this macro is defined as:

#define ARRAY_SIZE(array) (sizeof(array) / sizeof((array)[0]))

The first argument to the bt_le_adv_start() function specifies the advertising parameters. In this example, this argument is BT_LE_ADV_NCONN. If you look up its definition in zephyr/bluetooth/bluetooth.h, you'll see that this defines non-connectable advertising with a private address:

/** Non-connectable advertising with private address */
#define BT_LE_ADV_NCONN BT_LE_ADV_PARAM(0, BT_GAP_ADV_FAST_INT_MIN_2, \
                                        BT_GAP_ADV_FAST_INT_MAX_2, NULL)

The macro BT_LE_ADV_PARAM is another helper macro:

/**
 * @brief Helper to declare advertising parameters inline
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM(_options, _int_min, _int_max, _peer) \
        ((struct bt_le_adv_param[]) { \
                BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
         })

This teaches you that BT_GAP_ADV_FAST_INT_MIN_2 and BT_GAP_ADV_FAST_INT_MAX_2 are the minimum and maximum advertising interval. You can find their definitions in zephyr/bluetooth/gap.h:

#define BT_GAP_ADV_FAST_INT_MIN_2               0x00a0  /* 100 ms   */
#define BT_GAP_ADV_FAST_INT_MAX_2               0x00f0  /* 150 ms   */

So, your iBeacon will use an advertising interval of between 100 ms and 150 ms.

BT_LE_ADV_PARAM uses another macro in the same header file, BT_LE_ADV_PARAM_INIT, and its definition is:

/**
 * @brief Initialize advertising parameters
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
{ \
        .id = BT_ID_DEFAULT, \
        .sid = 0, \
        .secondary_max_skip = 0, \
        .options = (_options), \
        .interval_min = (_int_min), \
        .interval_max = (_int_max), \
        .peer = (_peer), \
}

All in all, this means that the code starts non-connectable advertising with its default Bluetooth ID, a private address, no special advertising options, and with an advertising interval of between 100 ms and 150 ms.

Summary

When reading source code of Zephyr applications, you can definitely feel discouraged by all these macros at first. However, after you've seen a couple of Zephyr programs, you start to see how they all fit together. The beauty of Zephyr is that its source code is completely open. If you don't understand what a specific macro or function is doing, just look up its declaration in the header file or even in the corresponding C file where it's implemented.

This approach has thought me a lot about developing Bluetooth Low Energy applications with Zephyr. I hope it will be a useful approach to you too.

Develop your own Bluetooth Low Energy applications

This week my new book has been published, Develop your own Bluetooth Low Energy Applications for Raspberry Pi, ESP32 and nRF52 with Python, Arduino and Zephyr.

Bluetooth Low Energy (BLE) is one of the most accessible wireless communication standards. You don't need any expensive equipment to develop BLE devices such as wireless sensor boards, proximity beacons, or heart rate monitors. All you need is a computer or a Raspberry Pi, an ESP32 microcontroller board, or a development board with a Nordic Semiconductor nRF5 (or an equivalent BLE SoC from another manufacturer).

On the software side, BLE is similarly accessible. Many development platforms, most of them open source, offer an API (application programming interface) to assist you in developing your own BLE applications. This book shows you the ropes of Bluetooth Low Energy programming with Python and the Bleak library on a Raspberry Pi or PC, with C++ and NimBLE-Arduino on Espressif's ESP32 development boards, and with C on one of the development boards supported by the Zephyr real-time operating system, such as Nordic Semiconductor's nRF52 boards.

While Bluetooth Low Energy is a complex technology with a comprehensive specification, getting started with the basics is relatively easy. This book takes a practical approach to BLE programming to make the technology even more approachable. With a minimal amount of theory, you'll develop code right from the start. After you've completed this book, you'll know enough to create your own BLE applications.

What is Bluetooth Low Energy?

Bluetooth is a wireless communication standard in the 2.4 GHz Industrial, Scientific, and Medical (ISM) frequency band. These days, if you hear about Bluetooth support in a product, this almost always is Bluetooth Low Energy (BLE). It's a radical departure from the original Bluetooth standard, which is now called Classic Bluetooth.

Bluetooth Low Energy and Classic Bluetooth are actually different protocols. Classic Bluetooth is essentially a wireless version of the traditional serial connection. If you want to print a document, transfer a file or stream audio, you want this to happen as fast as possible. Therefore, the focus of development in Classic Bluetooth was on attaining faster and faster speeds with every new version.

However, Classic Bluetooth wasn't a good fit for devices with low power consumption, for instance those powered by batteries. That's why Nokia adapted the Bluetooth standard to enable it to work in low-power scenarios. In 2006, they released their resulting technology onto the market, dubbed Wibree.

The Bluetooth Special Interest Group (SIG), the organization that maintains the Bluetooth specifications, showed interest in this new development. After consulting with Nokia, they decided to adopt Wibree as part of Bluetooth 4.0, with a new name, Bluetooth Low Energy. Classic Bluetooth remained available for high-throughput applications.

Layered architecture

The Bluetooth Core Specification is more than 3200 pages long. And this is only the core specification; there are many supplemental documents for BLE. However, BLE has a layered architecture. Many end-user applications only use the upper layers, so you don't need to know the details of the architecture's lower layers.

/images/ble-stack.png

The BLE architecture consists of three main blocks: controller, host, and application.

Controller

This has the lower-level layers: the Physical Layer (PHY), Link Layer (LL) and Direct Test Mode (DTM). These are the layers where the Bluetooth radio does its work. The controller communicates with the outside world using the antenna, in a frequency band around 2.4 GHz. It communicates with the host using a standardized interface between the two blocks: the Host Controller Interface (HCI). [1]

Host

This is the block with which the end user or application developer comes in contact. The Logical Link Control and Adaptation Protocol (L2CAP) defines channels and signaling commands. On top of it, the Security Manager Protocol (SMP) handles secure connections (with authentication and encryption), and the Attribute Protocol (ATT) defines how to expose and access data as attributes. The Generic Attribute Profile (GATT) [2] builds on the Attribute Protocol to define how to discover services and their characteristics and how to read and write their values. The upper layer of the Host block is the Generic Access Profile (GAP), which defines how devices can discover other devices and connect, pair, and bond to them. The host communicates with the controller using its part of the host controller interface, and applications communicate with the host depending on the APIs exposed by the operating system.

Application

This layer builds on top of the Generic Attribute Profile to implement application-specific characteristics, services, and profiles. A characteristic defines a specific type of data, such as an Alert Level. A service defines a set of characteristics and their behaviors, such as the Link Loss Service. A profile is a specification that describes how two or more devices with one or more services communicate with each other. An example is the Proximity profile, which has two roles: Proximity Monitor and Proximity Reporter.

The three blocks don't have to run on the same processor. In fact, there are three common configurations --- one single-chip and two dual-chip:

Single-chip (SoC)

Controller, host and application code run on the same chip. The host and controller communicate through function calls and queues in the chip's RAM. Most simple devices such as BLE sensors use this configuration; it keeps the cost down. Some smartphones also use this configuration if they have a SoC with Bluetooth built in.

Dual-chip over HCI

A dual-chip solution with application and host on one chip, and the controller on another chip, communicates over HCI. Because HCI is a standardized interface, it lets you combine different platforms. For instance, on a Raspberry Pi, the Wi-Fi and BLE chip implements a BLE controller. If you connect a BLE dongle to an older Raspberry Pi, this dongle also implements a BLE controller. [3] BlueZ, the Raspberry Pi Linux kernel's Bluetooth stack, implements a BLE host. So BlueZ communicates with the BLE controller in the built-in BLE chip or the BLE dongle. In the former case, the HCI uses SDIO, and in the latter, UART over USB. [4] Many smartphones and tablets also use the dual-chip over HCI configuration, with a powerful processor running the host and a Bluetooth chip running the controller.

Dual-chip with connectivity device

Another dual-chip solution is one with the application running on one chip and the host and controller on another chip. The latter is then called the connectivity device because it adds BLE connectivity to the other device. This approach is useful if you have an existing hardware device that you want to extend with BLE connectivity. Because there's no standardized interface in this case, the communication between the application processor and the connectivity device needs to make use of a proprietary protocol implemented by the connectivity device.

A three-chip solution with controller, host, and application each running on its own chip is also possible. However, because of the associated cost, this is typically only done for development systems.

How to communicate with BLE devices?

Bluetooth Low Energy has two ways to communicate between devices: with and without a connection.

Without a connection

Without a connection means that the device just broadcasts information in an advertisement. Every BLE device in the neighborhood is able to receive this information.

/images/ble-broadcaster-observers.png

Some examples of BLE devices broadcasting data are:

Proximity beacons

These devices, often following Apple's iBeacon standard, broadcast their ID. Receivers calculate their approximate distance to the beacons based on the advertisement's Received Signal Strength Indicator (RSSI).

Sensors

Many temperature and humidity sensors broadcast their sensor values. Most devices do this in an unencrypted fashion, but some of them encrypt the data to prevent it being read by every device in the neighborhood.

Mobile phones

After the COVID-19 pandemic started in 2020, Google and Apple collaborated on the Exposure Notifications standard for contact tracing. As part of this technology, Android phones and iPhones broadcast unique (but anonymous) numbers. Other phones can pick up these numbers and use them later to warn users that they have been in contact with someone who is known to have had COVID-19.

With a connection

The other way to communicate between BLE devices is with a connection. One device (the client) scans for BLE advertisements to find the device it wants to connect to. Then, optionally, it may do an active scan to ask the device (the server) which services are offered.

After the client connects to the server, the client can use the server's services. Each BLE service is a container of specific data from the server. You can read this data, or (with some services) write a value to the server.

/images/ble-peripheral-central.png

Some examples of BLE devices using a connection are:

Fitness trackers

Your smartphone can connect to a fitness tracker and read your heart rate, the tracker's battery level, and other measurements.

Sensors

Some environmental sensors let you read their sensor values over a BLE connection.

Proximity reporters

These devices sound an alert when their connection to another device is lost.

Advantages of BLE

Low power consumption

As its name implies, Bluetooth Low Energy is optimized for low-power applications. Its whole architecture is designed to reduce power consumption. For instance, setting up a connection, reading or writing data, and disconnecting happens in a couple of milliseconds. The radio is often the most energy-consuming part of a device. Therefore, the idea is to turn on the Bluetooth radio, create a connection, read or write data, disconnect, and turn off the radio again until the next time the device has to communicate.

This way, a well-designed BLE temperature sensor is able to work on a coin cell for ten years or more. You can use the same approach with other wireless technologies, such as Wi-Fi, but they require more power and more time to set up a connection.

Ubiquitous

BLE radio chips are ubiquitous. You can find them in smartphones, tablets, and laptops. This means that all those devices can talk to your BLE sensors or lightbulbs. Most manufacturers create mobile apps to control their BLE devices.

You can also find BLE radios in many single-board computers, such as the Raspberry Pi, and in popular microcontroller platforms such as the ESP32. [5] This makes it quite easy for you to create your own gateways for BLE devices. And, platforms such as the Nordic Semiconductor nRF5 series of microcontrollers with BLE radio even make it possible to create your own battery-powered BLE devices.

Low cost

There's no cost to access the official BLE specifications. Moreover, BLE chips are cheap, and the available development boards (based on an nRF5 or ESP32) and Raspberry Pis are quite affordable. This means you can just start with BLE programming at minimal cost.

Disadvantages of BLE

Short range

BLE has a short range (for most devices, less than 10 meters) compared to other wireless networks, such as Zigbee, Z-Wave, and Thread. It's not a coincidence that these competitors all have a mesh architecture, in which devices can forward their neighbors' messages in order to improve range. Low-power wide area networks (LPWANs), such as LoRaWAN, Sigfox, and NB-IoT, have even longer ranges.

In 2017, the Bluetooth SIG added Bluetooth Mesh, a mesh protocol. This builds upon BLE's physical and link layers with a whole new stack above them. However, Bluetooth Mesh isn't as well-established as the core BLE protocol, at least not for home use.

Limited speed

The BLE radio has a limited transmission speed. For Bluetooth 4.2 and earlier, this is 1 Mbps, while for Bluetooth 5 and later, this can be up to 2 Mbps. This makes BLE unsuitable for high-bandwidth applications.

You need a gateway

Wi-Fi devices have their own IP addresses, so you can communicate with them directly from other IP-based devices, and they're integrated in your LAN (local area network). Bluetooth doesn't have this: to integrate your BLE devices with other network devices, you need a gateway. This device has to translate Bluetooth packets to IP-based protocols such as MQTT (Message Queuing Telemetry Transport). That's why many BLE device manufacturers have smartphone apps that function as device gateways. [6]

Platforms used in this book

This book focuses on Bluetooth Low Energy programming on three platforms:

BLE platforms used in this book

Programming language

Library

Software platform

Hardware platform

Python

Bleak

Windows, Linux, macOS

Raspberry Pi or PC

C++

NimBLE-Arduino

Arduino framework

ESP32

C

/ [7]

Zephyr

nRF52

These choices were made in order to demonstrate a wide range of applications compatible with many software and hardware platforms.

Python/Bleak (Raspberry Pi, PC)

Python is an easy-to-use programming language that works on all major operating systems. There are a lot of Python Bluetooth Low Energy libraries, but many of them support only a single operating system. Bleak, which stands for Bluetooth Low Energy platform Agnostic Klient, is a welcome exception. It supports:

  • Windows 10, version 16299 (Fall Creators Update) or higher

  • Linux distributions with BlueZ 5.43 or higher (also on a Raspberry Pi)

  • OS X 10.11 (El Capitan) or macOS 10.12+

/images/rpi4.jpg

Bleak is a GATT client: it's able to connect to BLE devices that act as GATT servers. It supports reading, writing, and getting notifications from GATT servers, and it's also able to discover BLE devices and read advertising data broadcast by them.

Bleak doesn't implement a GATT server. In practice this isn't a big limitation. GATT servers are typically implemented on constrained devices, so, for this purpose, the ESP32 and nRF52 hardware platforms are a better match. [8]

C++/NimBLE-Arduino (ESP32)

If you're looking at microcontrollers, the Arduino framework has become quite popular, not only for the original Arduino boards, which didn't have BLE functionality, but also on ESP32 development boards, which do.

/images/esp32-pico-kit-v4.1.jpg

Programming for the Arduino framework is done in a variant of C++, but the framework and many Arduino libraries hide much of C++'s complexity. Even if you only know some C (which is much less complex than C++), you'll be able to use the Arduino framework.

One of the more popular BLE libraries for Arduino on the ESP32 is NimBLE-Arduino. It's a fork of NimBLE, which is part of the Apache Mynewt real-time operating system. With NimBLE-Arduino, you can easily create your own GATT server or client.

C/Zephyr (nRF52)

For even more constrained devices, typically battery-powered, you need a specialized real-time operating system (RTOS). This book uses the Zephyr Project on nRF52840-based devices from Nordic Semiconductor. Zephyr has a completely open-source Bluetooth Low Energy stack.

/images/nrf52840-dongle.png

Zephyr's BLE stack is highly configurable. You can build Zephyr firmware for three configuration types:

Combined build

Builds the BLE controller, BLE host, and your application for a one-chip configuration.

Host build

Builds the BLE host and your application, along with an HCI driver to let your device communicate with an external BLE controller on another chip. [9]

Controller build

Builds the BLE controller with an HCI driver to let your device communicate with an external BLE host on another chip.

With some basic knowledge of C, you can create your own BLE devices with Zephyr, such as BLE beacons, sensor boards, and proximity reporters. Zephyr has extensive documentation of its Bluetooth API, as well as a lot of ready-to-use examples that you can build upon.

Getting started with ESPHome

Last week my new book has been published, Getting Started with ESPHome: Develop your own custom home automation devices. It demonstrates how to create your own home automation devices with ESPHome on an ESP32 microcontroller board.

I always like to look at the big picture. That's why I want to take some time to talk about what ESPHome is, why you should use it and what you need.

Configuring instead of programming

The ESP8266 and its successor, the ESP32, are a series of low-cost microcontrollers with integrated Wi-Fi (for both series) and Bluetooth (for the ESP32), produced by Espressif Systems. The maker community quickly adopted these microcontrollers for tasks where an Arduino didn't suffice. [1]

You can program the ESP8266 and ESP32 using Espressif's ESP-IDF SDK, the ESP32 Arduino core, or MicroPython. Arduino and MicroPython lower the bar significantly, but it still takes some programming experience to build solutions with these microcontrollers.

One of the domains in which the ESP8266 and ESP32 have become popular is in the DIY (do-it-yourself) home automation scene. You just have to connect a sensor, switch, LED, or display to a microcontroller board, program it, and there you have it: your customised home automation device.

However, "program it" isn't that straightforward as it sounds. For instance, if you're using the Arduino environment, which has a lot of easy-to-use libraries, you still have to know your way around C++.

Luckily there are a couple of projects to make it easier to create firmware for ESP8266 or ESP32 devices for home automation. One of these is ESPHome.

On its homepage, the ESPHome developers describe it as: "ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems."

The fundamental idea of ESPHome is that you don't program your ESP8266 or ESP32 device, but configure it. Often you only have to configure which pins you have connected to a component, such as a sensor. You don't have to initialize the sensor, read its values in a loop, and process them.

Configuration is a completely different mindset than programming. It lowers the bar even more. With ESPHome, everyone can make home automation devices. [2]

Essentially ESPHome creates C++ code based on your configuration. The process looks like this:

/images/esphome-yaml-to-firmware.png

So when you write a YAML file with your device's configuration, ESPHome generates C++ code from it. More specifically, ESPHome creates a PlatformIO project using the Arduino framework. PlatformIO then builds the C++ code, and esptool uploads the resulting firmware to your device.

You don't have to know anything about what's happening under the hood. You just have to write the configuration of your device in a YAML file and memorise a small number of commands to let ESPHome do the rest.

The advantages of ESPHome

Why use ESPHome? The first reason is clear from the project's description: because you don't need to be able to program. Even if you're a programmer, ESPHome offers many advantages:

Works completely locally

Many commercial Wi-Fi-based home automation devices need a connection to a cloud service of the manufacturer. In contrast, ESPHome devices work locally and can communicate with a local home automation system such as Home Assistant or an MQTT-based home automation system.

Offers on-device automations

Many home automation systems use a central gateway that contains all the logic, with automations like "if the sun goes down, close the blinds." In contrast, ESPHome offers powerful on-device automations. Your devices can work independently from a home automation gateway, so they keep working if they lose Wi-Fi access or if your home automation gateway crashes.

Offers over-the-air updates

ESPHome includes out-of-the-box over-the-air (OTA) update functionality. This makes it easy to centrally manage your ESPHome devices and update the firmware. This means you don't have to go around your house with your laptop to connect a serial cable to each device and flash the firmware.

Supports a lot of components

ESPHome supports many components out-of-the-box: several types of sensors, switches, and displays (even e-paper displays) are available with just a couple of configuration lines. The list of supported components is growing with every release.

Has extensive documentation

The developers have documented every component in ESPHome, and this documentation is quite good.

Is customisable

Although you create ESPHome firmware by writing a configuration file, ESPHome doesn't hide anything from you. It's still possible to add custom components that you write in C++. You can even look at the C++ code that ESPHome generates and change it.

What hardware do you need?

ESPHome creates custom firmware for the ESP8266 and ESP32 microcontrollers, so you need one of these. There are many types of boards for both microcontrollers, varying in the amount of flash memory, RAM, and available pins. Some of them even come with extras such as a built-in display (OLED, TFT, or e-paper), battery, or camera.

ESPHome doesn't support all features of all boards out-of-the-box. Technically, all ESP8266/ESP32 devices should be able to run ESPHome. Some features just aren't supported yet.

Your first choice is between the ESP8266 or ESP32. If you're buying a device at present, the choice is simple: the ESP32. It is much more capable than its predecessor and has a faster processor, more memory, more peripherals, and adds Bluetooth.

Then comes the choice of board. Espressif has some development boards. Many other companies are making them too. There are even complete kits such as the M5Stack series. These are ESP32 development boards ready to use in your living room in a case with a display, buttons, MicroSD card slot, and speaker. These are currently my favourite devices to use with ESPHome. If you don't need something with a nice finish as the M5Stack devices, the TTGO T-Display ESP32 made by LilyGO is my development board of choice: it has an integrated 1.14 inch TFT display.

/images/ttgo-t-display.jpg

Other interesting devices to run ESPHome on are devices from manufacturers such as Sonoff and Shelly. These come with firmware that works with the manufacturer's cloud services. You can however replace the firmware with ESPHome. This unlocks the full potential of these devices and lets you use them in your local home automation system without any link to a cloud system.

What software do you need?

You can use ESPHome to create a fully autonomous microcontroller project --- for example, a plant monitor that turns on an LED if the plant's soil is too dry. However, if you don't publish the plant's status over the network, this would be a waste of the ESP32's capabilities. The main usage cases of ESPHome are:

  • To send a device's sensor measurements to a home automation gateway.

  • To remotely control a device's lights or switches from a home automation gateway.

ESPHome supports two ways of communication between your device and the home automation gateway:

Native API

The ESPHome native API is a highly optimised network protocol using Google's protocol buffers. It's meant to be used with Home Assistant --- a popular open-source home automation system.

MQTT

MQTT (Message Queuing Telemetry Transport) is an OASIS standard messaging protocol designed with a lightweight publish/subscribe approach for messages. All your ESPHome devices then communicate with an MQTT broker such as Eclipse Mosquitto.

From the beginning (when it was still called esphomeyaml), the ESPHome project has been tightly integrated with Home Assistant, so the ESPHome developers prefer the native API. However, MQTT is fully supported, allowing your devices to communicate with many other home automation gateways, as MQTT is a popular standard. [3]

Do you need a development environment?

With ESPHome you don't program your devices but configure them. However, you still need something that looks like a "development" environment. When your device configurations are simple, you could do without, but the more complex they become, you'll need all the help you can get.

This doesn't mean you have to install a full-blown Integrated Development Environment (IDE). You should only need a couple of programs:

An editor

You could make do with a simple text editor such as Notepad (Windows), TextEdit (macOS), or the default text editor on your Linux distribution. However, having an editor with syntax highlighting for YAML files is easier. Some examples are Notepad++ and Sublime Text. If you're a command-line user on Linux, both vim and Emacs work fine. [4] Use whatever you like, because your editor is an important tool to work with ESPHome.

A YAML linter

A linter is a program that checks your file for the correct syntax. An editor with syntax highlighting has this linter built-in, but you can also run this standalone. A good YAML linter is the Python program yamllint. Not only does it check for syntax validity, but also weird things like key repetitions, as well as cosmetic problems such as line length, trailing spaces, and inconsistent indentation. ESPHome includes its own linter, specifically targeted at finding errors in ESPHome configurations. Both linters are complementary.

If you're used to developing in an IDE, an interesting alternative is the ESPHome plugin for Visual Studio Code. [5] This plugin provides validation and completion of what you type in an ESPHome YAML file. It also shows tooltips with help when you hover over keywords in the configuration.

/images/esphome-vscode-plugin.png

The four pillars of a good home automation system

I have been running a home automation system on a Raspberry Pi almost as long as this popular single-board computer family exists. In all those years, the setup has changed a lot: I have replaced the Raspberry Pi 1 by a Raspberry Pi 4, I switched from Domoticz to Home Assistant and more recently to a highly modular setup of home automation services running in Docker containers and communicating by MQTT.

Looking back at my first setup, the seeds were already there for what I now consider the four pillars of a good home automation system:

  • be secure, so you don't risk someone else controlling your house or spying on you at home;

  • be modular, to make it easy to plug in other protocols or applications;

  • only use open source software;

  • be self-sufficient, not relying on cloud systems from Google, Amazon, or other parties.

Let's look into these one by one.

Secure

Home automation and IoT devices are notoriously insecure. At the Usenix Security Conference 2019, the Czech security software company Avast and Stanford University presented their research of household IoT devices. Avast scanned 83 million IoT devices in 16 million homes around the world of people who agreed to share these data. The results of the study published in "All Things Considered: An Analysis of IoT Devices on Home Networks" were staggering:

  • 7% percent of all IoT devices support an obsolete, insecure, and completely unencrypted protocol such as Telnet or FTP.

  • Of these, 17% exhibit weak FTP passwords, and 2% have weak Telnet passwords.

  • Surveillance cameras have the weakest Telnet profile, with more than 10% of them that support Telnet with weak credentials.

  • 3% percent of the homes are externally visible on the internet and more than half of those have a known vulnerability or a weak password.

This is not an isolated study. Not a week goes by without some news items about insecure devices, most of the time because basic security measures such as strong passwords are not enforced by the manufacturer or basic programming errors have been made. To give you an idea about what can happen: in 2018 nude videos of the Dutch women's handball team appeared on a popular porn website because the surveillance cameras of the dressing room of a sauna were broken into. Imagine if someone can access your baby monitor with a camera or your security camera in your living room or bedroom...

So what can you do to secure your home automation system? If you choose an off-the-shelf system: not much. You fully rely on the manufacturer's ability to create a secure system and the goodwill to keep supplying patches that solve security issues that have been discovered. And the home automation and IoT industries have clearly shown they are not up to the task. This is one of the reasons why I prefer open source software. Not because it is always secure, but because the transparency of the open source development process forces developers to create more secure software.

Modular

There are many competing standards and communication protocols for home automation. Unfortunately, many off-the-shelf home automation gateways support only a small subset of these protocols or even use a proprietary protocol that locks you into using devices of the same manufacturer. That severely limits your choice of products.

You can't know which protocols will become popular in a few years, and maybe you like one product that uses Z-Wave and another product that uses Zigbee. It should be easy to interconnect these devices, even when they use different protocols.

So how can you do this? Many of the wireless communication protocols for home automation need a dedicated transceiver because they work on a specific radio frequency. That's where the Raspberry Pi shines: you can start with a basic Raspberry Pi setup supporting only IoT devices that are communicating over Wi-Fi and Bluetooth, add an RTL-SDR USB dongle to read the measurements of your 433.92 MHz weather sensors, later add a Z-Wave HAT on the board when you start adding Z-Wave sensors to your house and then add a Zigbee USB transceiver when you want to control some Zigbee lights.

Modularity is also important for software. There's a lot of user-friendly software to make your Raspberry Pi a home automation gateway. So you just install something like Home Assistant, openHAB or Domoticz on your Raspberry Pi and that's it: you have a gateway that supports tons of devices. Some of these systems are very modular and extensible, others aren't. Many of them support MQTT (Message Queuing Telemetry Transport), a common language to exchange messages.

MQTT has become the standard for interoperability between various home automation devices. For instance, if your home automation gateway of choice doesn't support Zigbee but it does support MQTT, then you only have to run the Zigbee2mqtt software, which translates the Zigbee protocol to MQTT messages. Your gateway can then talk to your Zigbee devices using MQTT.

Modularity also means that you don't have to have one gateway. You can perfectly have your main gateway in your basement, but install a second gateway with your 433.92 MHz receiver for your environmental sensors in your living room because that gives you better coverage to receive data from these wireless sensors. If you're using MQTT, that's very simple to implement: you just relay the sensor readings that your gateway in the living room receives to your MQTT broker, after which all the other components of your home automation setup can access the readings in the MQTT format.

In short: a good modular home automation system means that you can mix and match the devices that you like, irrespective of their protocol, and you can use the software and hardware components of your choice, in various locations in your house.

Open source

If you buy an off-the-shelf home automation gateway, you generally don't get access to its source code, so you cannot peek into it to see what it does or to assess its quality. You just have to believe the manufacturer on his word. Is that enough for you if it's about software that will get to know you intimately because it processes sensor readings and even camera images about you in your home? Not for me, so open source software is a hard requirement for my home automation setup.

But what when you're not a programmer and you wouldn't even understand the source code of your home automation system if you had access to it? Even then the use of open source software has a lot of advantages. It's not because you don't have the programming experience that others can't help. Most open source projects have a decentralized software development model that encourages open collaboration.

So if you find a bug in the software or see something wrong in its source code but don't have the programming experience needed to fix it, just report the bug on the issue tracker of the project, and hopefully, someone else in the project's community will step in and fix it. It all depends on the health of the project's community. But a good open source project has a vibrant community of developers and users who collaborate to continuously improve the software.

And open source doesn't just mean getting access to the source code, it's much more than that. For instance, when you know a specific software project is open source, you know that it doesn't arbitrarily restrict what you can do with it. The Open Source Definition even explicitly lists that the license of open source software must not discriminate against any person or group of persons, nor restrict anyone from making use of the program in a specific field of endeavour.

The decentralization and transparency of the open source development model gives power back to the users, where it belongs. For home automation that's even more important: I don't want a company having control over my house.

Self-sufficient

During the last ten years, there has been a worrying development in the computer industry: we all depend more and more on (centralized) cloud systems. Unfortunately, the home automation industry didn't escape this fate. Many popular home automation and IoT systems depend on a cloud server. Some examples:

  • the Ring video doorbell with Wi-Fi camera;

  • the Nest Learning Thermostat;

  • so-called 'smart speakers' running voice assistants, like Amazon Echo and Google Home;

  • the IFTTT service that links various other services.

This isn't without its problems. In the last couple of years some of these cloud services for home automation stopped working for their users:

  • In 2014 Nest bought the company that was selling the Revolv Hub home automation system, not long after Nest itself was acquired by Google. In 2016 Nest shut down the servers Revolv Hub depended on, after announcing it with a quiet note on the website of Revolv a few months earlier. That meant that the $300 Revolv Hub ceased functioning entirely.

  • At the end of 2019, Best Buy announced that several of their Insignia-branded smart devices would stop working because they decided to shut down the corresponding backend systems.

  • In May 2020 Wink (with the catchphrase "A simpler way to a smarter home") announced with just a week's notice that it would start charging a monthly fee for the use of their services. Users that didn't want to pay were no longer be able to access their Wink devices and their automations were disabled. The Wink Hub was rendered useless, although it had been in stores with the clear description "no required monthly fees, ever". Ironically, the announcement ended with the message "Our user community is integral to Wink, and we want to continue to be your trusted smart home provider." Yeah, right.

The home automation system at the left is cloud-based: a simple motion detection message first goes to a server over the internet before returning to your light. The self-hosted system on the right makes much more sense: a Raspberry Pi on your network relays the message without using the internet detour.

But fundamentally, the problem lies even deeper. It just makes no sense to use cloud services to automate your home. Home automation comes down to: you want one device to be able to respond to another device in your home. For instance: the motion sensor in your bathroom detects motion at night, and this turns the bathroom light on for five minutes. If you use IFTTT to link both devices, the motion sensor has to send a message to the internet, IFTTT relays the message to your bathroom light (for instance a Philips Hue light), and the bathroom light turns on.

But there's no need for an internet service like IFTTT between these devices because both devices are in your home, so it makes much more sense to link them locally, using a server at home. This could be a Raspberry Pi running home automation software that doesn't need a cloud server to function, but does all its processing on-device (or on the edge, as it's called now). You can perfectly do this with Node-RED or Home Assistant. Then there's no way a company can render your home automation system useless by shutting down their service, or your bathroom light doesn't turn on at night because your internet connection or IFTTT's servers are down. You're fully in control of your home automation system.

There's another risk: using cloud services for your home automation system invades your privacy. Look at some of the privacy issues with the services I talked about above:

  • If you use the Ring video doorbell, it sends a video of everyone that steps on your porch to the manufacturer. The Ring company (which has been bought by Amazon in 2018) has a questionable approach to privacy: in January 2019 it was uncovered that employees have access to the video recordings of all Ring devices and even that the data are stored unencrypted.

  • If you use the Nest Learning Thermostat, Google knows precisely when you are home and when you aren't.

  • If you run a smart speaker like the Amazon Echo, what you tell your house members will be sent countless times inadvertently to Amazon because the Echo thinks it has heard its wake word. Moreover, Amazon's employees listen to a part of all 'conversations' with the Echo to improve its algorithms.

  • If you use the IFTTT service to link your various other services, you give one company access to all your home automation services, which is too much power concentrated in one company's hands: they can see exactly what you're doing.

After reading all this, do you still find it acceptable to use cloud services for your home automation?