Getting started with ESPHome

Last week my new book has been published, Getting Started with ESPHome: Develop your own custom home automation devices. It demonstrates how to create your own home automation devices with ESPHome on an ESP32 microcontroller board.

I always like to look at the big picture. That's why I want to take some time to talk about what ESPHome is, why you should use it and what you need.

Configuring instead of programming

The ESP8266 and its successor, the ESP32, are a series of low-cost microcontrollers with integrated Wi-Fi (for both series) and Bluetooth (for the ESP32), produced by Espressif Systems. The maker community quickly adopted these microcontrollers for tasks where an Arduino didn't suffice. 1

You can program the ESP8266 and ESP32 using Espressif's ESP-IDF SDK, the ESP32 Arduino core, or MicroPython. Arduino and MicroPython lower the bar significantly, but it still takes some programming experience to build solutions with these microcontrollers.

One of the domains in which the ESP8266 and ESP32 have become popular is in the DIY (do-it-yourself) home automation scene. You just have to connect a sensor, switch, LED, or display to a microcontroller board, program it, and there you have it: your customised home automation device.

However, "program it" isn't that straightforward as it sounds. For instance, if you're using the Arduino environment, which has a lot of easy-to-use libraries, you still have to know your way around C++.

Luckily there are a couple of projects to make it easier to create firmware for ESP8266 or ESP32 devices for home automation. One of these is ESPHome.

On its homepage, the ESPHome developers describe it as: "ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems."

The fundamental idea of ESPHome is that you don't program your ESP8266 or ESP32 device, but configure it. Often you only have to configure which pins you have connected to a component, such as a sensor. You don't have to initialize the sensor, read its values in a loop, and process them.

Configuration is a completely different mindset than programming. It lowers the bar even more. With ESPHome, everyone can make home automation devices. 2

Essentially ESPHome creates C++ code based on your configuration. The process looks like this:

/images/esphome-yaml-to-firmware.png

So when you write a YAML file with your device's configuration, ESPHome generates C++ code from it. More specifically, ESPHome creates a PlatformIO project using the Arduino framework. PlatformIO then builds the C++ code, and esptool uploads the resulting firmware to your device.

You don't have to know anything about what's happening under the hood. You just have to write the configuration of your device in a YAML file and memorise a small number of commands to let ESPHome do the rest.

The advantages of ESPHome

Why use ESPHome? The first reason is clear from the project's description: because you don't need to be able to program. Even if you're a programmer, ESPHome offers many advantages:

Works completely locally

Many commercial Wi-Fi-based home automation devices need a connection to a cloud service of the manufacturer. In contrast, ESPHome devices work locally and can communicate with a local home automation system such as Home Assistant or an MQTT-based home automation system.

Offers on-device automations

Many home automation systems use a central gateway that contains all the logic, with automations like "if the sun goes down, close the blinds." In contrast, ESPHome offers powerful on-device automations. Your devices can work independently from a home automation gateway, so they keep working if they lose Wi-Fi access or if your home automation gateway crashes.

Offers over-the-air updates

ESPHome includes out-of-the-box over-the-air (OTA) update functionality. This makes it easy to centrally manage your ESPHome devices and update the firmware. This means you don't have to go around your house with your laptop to connect a serial cable to each device and flash the firmware.

Supports a lot of components

ESPHome supports many components out-of-the-box: several types of sensors, switches, and displays (even e-paper displays) are available with just a couple of configuration lines. The list of supported components is growing with every release.

Has extensive documentation

The developers have documented every component in ESPHome, and this documentation is quite good.

Is customisable

Although you create ESPHome firmware by writing a configuration file, ESPHome doesn't hide anything from you. It's still possible to add custom components that you write in C++. You can even look at the C++ code that ESPHome generates and change it.

What hardware do you need?

ESPHome creates custom firmware for the ESP8266 and ESP32 microcontrollers, so you need one of these. There are many types of boards for both microcontrollers, varying in the amount of flash memory, RAM, and available pins. Some of them even come with extras such as a built-in display (OLED, TFT, or e-paper), battery, or camera.

ESPHome doesn't support all features of all boards out-of-the-box. Technically, all ESP8266/ESP32 devices should be able to run ESPHome. Some features just aren't supported yet.

Your first choice is between the ESP8266 or ESP32. If you're buying a device at present, the choice is simple: the ESP32. It is much more capable than its predecessor and has a faster processor, more memory, more peripherals, and adds Bluetooth.

Then comes the choice of board. Espressif has some development boards. Many other companies are making them too. There are even complete kits such as the M5Stack series. These are ESP32 development boards ready to use in your living room in a case with a display, buttons, MicroSD card slot, and speaker. These are currently my favourite devices to use with ESPHome. If you don't need something with a nice finish as the M5Stack devices, the TTGO T-Display ESP32 made by LilyGO is my development board of choice: it has an integrated 1.14 inch TFT display.

/images/ttgo-t-display.jpg

Other interesting devices to run ESPHome on are devices from manufacturers such as Sonoff and Shelly. These come with firmware that works with the manufacturer's cloud services. You can however replace the firmware with ESPHome. This unlocks the full potential of these devices and lets you use them in your local home automation system without any link to a cloud system.

What software do you need?

You can use ESPHome to create a fully autonomous microcontroller project --- for example, a plant monitor that turns on an LED if the plant's soil is too dry. However, if you don't publish the plant's status over the network, this would be a waste of the ESP32's capabilities. The main usage cases of ESPHome are:

  • To send a device's sensor measurements to a home automation gateway.

  • To remotely control a device's lights or switches from a home automation gateway.

ESPHome supports two ways of communication between your device and the home automation gateway:

Native API

The ESPHome native API is a highly optimised network protocol using Google's protocol buffers. It's meant to be used with Home Assistant --- a popular open-source home automation system.

MQTT

MQTT (Message Queuing Telemetry Transport) is an OASIS standard messaging protocol designed with a lightweight publish/subscribe approach for messages. All your ESPHome devices then communicate with an MQTT broker such as Eclipse Mosquitto.

From the beginning (when it was still called esphomeyaml), the ESPHome project has been tightly integrated with Home Assistant, so the ESPHome developers prefer the native API. However, MQTT is fully supported, allowing your devices to communicate with many other home automation gateways, as MQTT is a popular standard. 3

Do you need a development environment?

With ESPHome you don't program your devices but configure them. However, you still need something that looks like a "development" environment. When your device configurations are simple, you could do without, but the more complex they become, you'll need all the help you can get.

This doesn't mean you have to install a full-blown Integrated Development Environment (IDE). You should only need a couple of programs:

An editor

You could make do with a simple text editor such as Notepad (Windows), TextEdit (macOS), or the default text editor on your Linux distribution. However, having an editor with syntax highlighting for YAML files is easier. Some examples are Notepad++ and Sublime Text. If you're a command-line user on Linux, both vim and Emacs work fine. 4 Use whatever you like, because your editor is an important tool to work with ESPHome.

A YAML linter

A linter is a program that checks your file for the correct syntax. An editor with syntax highlighting has this linter built-in, but you can also run this standalone. A good YAML linter is the Python program yamllint. Not only does it check for syntax validity, but also weird things like key repetitions, as well as cosmetic problems such as line length, trailing spaces, and inconsistent indentation. ESPHome includes its own linter, specifically targeted at finding errors in ESPHome configurations. Both linters are complementary.

If you're used to developing in an IDE, an interesting alternative is the ESPHome plugin for Visual Studio Code. 5 This plugin provides validation and completion of what you type in an ESPHome YAML file. It also shows tooltips with help when you hover over keywords in the configuration.

/images/esphome-vscode-plugin.png
1

Basic Arduino models don't have network connectivity, which limits their use for home automation and IoT applications.

2

Note that you can still add your own C++ code to program ESPHome devices if you like.

3

I prefer MQTT, and my previous book, Control Your Home with Raspberry Pi: Secure, Modular, Open-Source and Self-Sufficient explains how to create a home automation system centered around MQTT. ESPHome fits perfectly in this approach, and this is how I'm using it at home.

4

I'm an avid vim user.

5

If you're worried about the telemetry that Microsoft has enabled by default in Visual Studio Code, download VSCodium. This project builds Microsoft's vscode source code without the telemetry.

Building ESP32 firmware with Espressif's IDF Docker image

If you're only occasionally building ESP32 firmware with Espressif's IoT Development Framework (ESP-IDF), your current build probably fails because the installation from last time is outdated. Then you have to reinstall the newest version of the framework and its dependencies, and you probably still encounter a dependency issue. If it finally works and you have successfully built your firmware, your excitement makes up for the frustrating experience, until the next time you want to build some other ESP32 firmware, and the process repeats.

But there's another way: use the IDF Docker image. It contains ESP-IDF, all the required Python packages, and all build tools such as CMake, make, ninja, and cross-compiler toolchains. In short: a ready-to-use IDF build system.

Building ESP32 firmware with CMake in the container is as easy as:

docker run --rm -v $PWD:/project -w /project espressif/idf idf.py build

This mounts the current directory on the host ($PWD) as the directory /project in the container and runs the command idf.py build on the code. After the build is finished, you can find your firmware in the current directory.

Note

If you're building a project that needs a specific version of ESP-IDF, just use a tag such as espressif/idf:v4.2.1 instead of espressif/idf. You can find the available tags on Docker Hub.

The same works if the project is using make for its build:

docker run --rm -v $PWD:/project -w /project espressif/idf make defconfig all -j4

You can also use the container interactively.

How to upgrade the Adafruit nRF52 bootloader

The Adafruit nRF52 bootloader is a USB-enabled CDC/DFU/UF2 bootloader for nRF52 boards. An advantage compared to Nordic Semiconductor's default bootloader is that you can just drag and drop your application firmware from your operating system's file explorer, without having to install any programming tools. For nRF52840 boards, you hold the reset button while sliding the USB connector in the USB port of your computer, or you tap the reset button twice within 500 ms. The bootloader then starts in DFU (device firmware upgrade) mode and behaves like a removable flash drive.

This device shows three virtual files:

INFO_UF2.TXT

contains information about the bootloader build and the board on which it's running

INDEX.HTM

redirects to a page that contains an IDE or other information about the board

CURRENT.UF2

the contents of the entire flash storage of the device

Flashing the device with new firmware is as easy as copying a UF2 file to the drive. After the file is copied, the drive is unmounted and the new firmware is running on the board. 1

However, the bootloader itself can't be upgraded like this. Today I had some problems with the April USB Dongle 52840 2 that interrupted the UF2 file transfer before it was finished. The dmesg command showed a call trace and some errors. As a result, the device was useless: I couldn't put any new firmware on it.

I was puzzled, but then I looked at the bootloader's version in INFO_UF2.TXT, and this was quite old: a 0.2.x version from 2018. I hoped that upgrading the bootloader would solve the problem.

Upgrading the Adafruit nRF52 bootloader is quite easy:

  1. Download the latest release of the bootloader. For the April USB Dongle 52840 and other devices based on Nordic Semiconductor's nRF52840 Dongle 3, the firmware file is pca10059_bootloader-0.5.0.zip for the bootloader and SoftDevice wireless protocol stack. PCA10059 is the official name of Nordic Semiconductor's nRF52840 Dongle.

  2. Unpack the ZIP file. You need the file pca10059_bootloader-0.5.0_s140_6.1.1.zip (yes, another ZIP file) in it.

  3. Install adafruit-nrfutil: pip3 install --user adafruit-nrfutil.

  4. Connect the board to your computer's USB port and flash the new bootloader package:

$ adafruit-nrfutil dfu serial --package pca10059_bootloader-0.5.0_s140_6.1.1.zip --port /dev/ttyACM0
Upgrading target on /dev/ttyACM0 with DFU package /home/koan/pca10059_bootloader-0.5.0_s140_6.1.1.zip. Flow control is disabled, Dual bank, Touch disabled
########################################
########################################
########################################
########################################
########################################
########################################
########################################
########################################
########################################
############
Activating new firmware
Device programmed.

After this, the INFO_UF2.TXT file contains UF2 Bootloader 0.5.0 lib/nrfx (v2.0.0) lib/tinyusb (0.9.0-22-g7cdeed54) lib/uf2 (remotes/origin/configupdate-9-gadbb8c7).

Luckily the upgraded bootloader solved my problem: I was able to flash the board with new UF2 application firmware.

1

The UF2 format for firmware has become quite popular in recent years. For instance, the Raspberry Pi Pico also has a bootloader that accepts UF2 files.

2

If you're looking for an nRF52840 device with a longer range than similar devices with a PCB-based antenna, I can definitely recommend the April USB Dongle 52840: in my experiments with the dongle as a Bluetooth Low Energy and a 802.15.4/Zigbee sniffer, the external antenna makes a big difference.

3

Another interesting nRF52840 board is the nRF52840 MDK USB Dongle from makerdiary. This is essentially a Nordic Semiconductor nRF52840 Dongle with the Adafruit nRF52 bootloader, sold in a case.