The four pillars of a good home automation system

I have been running a home automation system on a Raspberry almost as long as this popular single-board computer family exists. In all those years, the setup has changed a lot: I have replaced the Raspberry Pi 1 by a Raspberry Pi 4, I switched from Domoticz to Home Assistant and more recently to a highly modular setup of home automation services running in Docker containers and communicating by MQTT.

Looking back at my first setup, the seeds were already there for what I now consider the four pillars of a good home automation system:

  • be secure, so you don't risk someone else controlling your house or spying on you at home;

  • be modular, to make it easy to plug in other protocols or applications;

  • only use open source software;

  • be self-sufficient, not relying on cloud systems from Google, Amazon, or other parties.

Let's look into these one by one.

Secure

Home automation and IoT devices are notoriously insecure. At the Usenix Security Conference 2019, the Czech security software company Avast and Stanford University presented their research of household IoT devices. Avast scanned 83 million IoT devices in 16 million homes around the world of people who agreed to share these data. The results of the study published in "All Things Considered: An Analysis of IoT Devices on Home Networks" were staggering:

  • 7% percent of all IoT devices support an obsolete, insecure, and completely unencrypted protocol such as Telnet or FTP.

  • Of these, 17% exhibit weak FTP passwords, and 2% have weak Telnet passwords.

  • Surveillance cameras have the weakest Telnet profile, with more than 10% of them that support Telnet with weak credentials.

  • 3% percent of the homes are externally visible on the internet and more than half of those have a known vulnerability or a weak password.

This is not an isolated study. Not a week goes by without some news items about insecure devices, most of the time because basic security measures such as strong passwords are not enforced by the manufacturer or basic programming errors have been made. To give you an idea about what can happen: in 2018 nude videos of the Dutch women's handball team appeared on a popular porn website because the surveillance cameras of the dressing room of a sauna were broken into. Imagine if someone can access your baby monitor with a camera or your security camera in your living room or bedroom...

So what can you do to secure your home automation system? If you choose an off-the-shelf system: not much. You fully rely on the manufacturer's ability to create a secure system and the goodwill to keep supplying patches that solve security issues that have been discovered. And the home automation and IoT industries have clearly shown they are not up to the task. This is one of the reasons why I prefer open source software. Not because it is always secure, but because the transparency of the open source development process forces developers to create more secure software.

Modular

There are many competing standards and communication protocols for home automation. Unfortunately, many off-the-shelf home automation gateways support only a small subset of these protocols or even use a proprietary protocol that locks you into using devices of the same manufacturer. That severely limits your choice of products.

You can't know which protocols will become popular in a few years, and maybe you like one product that uses Z-Wave and another product that uses Zigbee. It should be easy to interconnect these devices, even when they use different protocols.

So how can you do this? Many of the wireless communication protocols for home automation need a dedicated transceiver because they work on a specific radio frequency. That's where the Raspberry Pi shines: you can start with a basic Raspberry Pi setup supporting only IoT devices that are communicating over Wi-Fi and Bluetooth, add an RTL-SDR USB dongle to read the measurements of your 433.92 MHz weather sensors, later add a Z-Wave HAT on the board when you start adding Z-Wave sensors to your house and then add a Zigbee USB transceiver when you want to control some Zigbee lights.

Modularity is also important for software. There's a lot of user-friendly software to make your Raspberry Pi a home automation gateway. So you just install something like Home Assistant, openHAB or Domoticz on your Raspberry Pi and that's it: you have a gateway that supports tons of devices. Some of these systems are very modular and extensible, others aren't. Many of them support MQTT (Message Queuing Telemetry Transport), a common language to exchange messages.

MQTT has become the standard for interoperability between various home automation devices. For instance, if your home automation gateway of choice doesn't support Zigbee but it does support MQTT, then you only have to run the Zigbee2mqtt software, which translates the Zigbee protocol to MQTT messages. Your gateway can then talk to your Zigbee devices using MQTT.

Modularity also means that you don't have to have one gateway. You can perfectly have your main gateway in your basement, but install a second gateway with your 433.92 MHz receiver for your environmental sensors in your living room because that gives you better coverage to receive data from these wireless sensors. If you're using MQTT, that's very simple to implement: you just relay the sensor readings that your gateway in the living room receives to your MQTT broker, after which all the other components of your home automation setup can access the readings in the MQTT format.

In short: a good modular home automation system means that you can mix and match the devices that you like, irrespective of their protocol, and you can use the software and hardware components of your choice, in various locations in your house.

Open source

If you buy an off-the-shelf home automation gateway, you generally don't get access to its source code, so you cannot peek into it to see what it does or to assess its quality. You just have to believe the manufacturer on his word. Is that enough for you if it's about software that will get to know you intimately because it processes sensor readings and even camera images about you in your home? Not for me, so open source software is a hard requirement for my home automation setup.

But what when you're not a programmer and you wouldn't even understand the source code of your home automation system if you had access to it? Even then the use of open source software has a lot of advantages. It's not because you don't have the programming experience that others can't help. Most open source projects have a decentralized software development model that encourages open collaboration.

So if you find a bug in the software or see something wrong in its source code but don't have the programming experience needed to fix it, just report the bug on the issue tracker of the project, and hopefully, someone else in the project's community will step in and fix it. It all depends on the health of the project's community. But a good open source project has a vibrant community of developers and users who collaborate to continuously improve the software.

And open source doesn't just mean getting access to the source code, it's much more than that. For instance, when you know a specific software project is open source, you know that it doesn't arbitrarily restrict what you can do with it. The Open Source Definition even explicitly lists that the license of open source software must not discriminate against any person or group of persons, nor restrict anyone from making use of the program in a specific field of endeavour.

The decentralization and transparency of the open source development model gives power back to the users, where it belongs. For home automation that's even more important: I don't want a company having control over my house.

Self-sufficient

During the last ten years, there has been a worrying development in the computer industry: we all depend more and more on (centralized) cloud systems. Unfortunately, the home automation industry didn't escape this fate. Many popular home automation and IoT systems depend on a cloud server. Some examples:

  • the Ring video doorbell with Wi-Fi camera;

  • the Nest Learning Thermostat;

  • so-called 'smart speakers' running voice assistants, like Amazon Echo and Google Home;

  • the IFTTT service that links various other services.

This isn't without its problems. In the last couple of years some of these cloud services for home automation stopped working for their users:

  • In 2014 Nest bought the company that was selling the Revolv Hub home automation system, not long after Nest itself was acquired by Google. In 2016 Nest shut down the servers Revolv Hub depended on, after announcing it with a quiet note on the website of Revolv a few months earlier. That meant that the $300 Revolv Hub ceased functioning entirely.

  • At the end of 2019, Best Buy announced that several of their Insignia-branded smart devices would stop working because they decided to shut down the corresponding backend systems.

  • In May 2020 Wink (with the catchphrase "A simpler way to a smarter home") announced with just a week's notice that it would start charging a monthly fee for the use of their services. Users that didn't want to pay were no longer be able to access their Wink devices and their automations were disabled. The Wink Hub was rendered useless, although it had been in stores with the clear description "no required monthly fees, ever". Ironically, the announcement ended with the message "Our user community is integral to Wink, and we want to continue to be your trusted smart home provider." Yeah, right.

The home automation system at the left is cloud-based: a simple motion detection message first goes to a server over the internet before returning to your light. The self-hosted system on the right makes much more sense: a Raspberry Pi on your network relays the message without using the internet detour.

But fundamentally, the problem lies even deeper. It just makes no sense to use cloud services to automate your home. Home automation comes down to: you want one device to be able to respond to another device in your home. For instance: the motion sensor in your bathroom detects motion at night, and this turns the bathroom light on for five minutes. If you use IFTTT to link both devices, the motion sensor has to send a message to the internet, IFTTT relays the message to your bathroom light (for instance a Philips Hue light), and the bathroom light turns on.

But there's no need for an internet service like IFTTT between these devices because both devices are in your home, so it makes much more sense to link them locally, using a server at home. This could be a Raspberry Pi running home automation software that doesn't need a cloud server to function, but does all its processing on-device (or on the edge, as it's called now). You can perfectly do this with Node-RED or Home Assistant. Then there's no way a company can render your home automation system useless by shutting down their service, or your bathroom light doesn't turn on at night because your internet connection or IFTTT's servers are down. You're fully in control of your home automation system.

There's another risk: using cloud services for your home automation system invades your privacy. Look at some of the privacy issues with the services I talked about above:

  • If you use the Ring video doorbell, it sends a video of everyone that steps on your porch to the manufacturer. The Ring company (which has been bought by Amazon in 2018) has a questionable approach to privacy: in January 2019 it was uncovered that employees have access to the video recordings of all Ring devices and even that the data are stored unencrypted.

  • If you use the Nest Learning Thermostat, Google knows precisely when you are home and when you aren't.

  • If you run a smart speaker like the Amazon Echo, what you tell your house members will be sent countless times inadvertently to Amazon because the Echo thinks it has heard its wake word. Moreover, Amazon's employees listen to a part of all 'conversations' with the Echo to improve its algorithms.

  • If you use the IFTTT service to link your various other services, you give one company access to all your home automation services, which is too much power concentrated in one company's hands: they can see exactly what you're doing.

After reading all this, do you still find it acceptable to use cloud services for your home automation?

How to do this?

This is an abridged version of the introduction of my book Control Your Home with Raspberry Pi. You can read the full introduction for free in the Elektor store, where you can also find the table of contents. In this book you will learn how to implement the four pillars of a good home automation system, by installing and configuring your Raspberry Pi as a highly flexible home automation gateway for protocols of your choice, and link various services with MQTT to make it your own system, including fully offline voice commands in your own language, and all secured with TLS.

Exploring the PinePhone with the multi-distro demo image

The PinePhone, powered by Linux

A few days ago my PinePhone arrived. It came with postmarketOS, which is based on Alpine Linux. By default it has a fairly limited set of apps. And if you look in the application manager, there are not many apps available to install either. With apk, Alpine's package manager, you can install a lot of other software. I installed Firefox this way, it seems to work, but you often need to zoom out to 50% to get the whole website on the screen.

But what I find most impressive is how this low-cost Linux phone has garnered extensive support from all major mobile Linux distributions. Who would have thought a few years ago that 'distro hopping' on a phone would be a thing?

Trying out some Linux distributions on the PinePhone

The easiest way to explore a lot of the supported distributions is the PinePhone multi-distro demo image, which is a 5 GB image with 13 distributions and a neat bootloader, p-boot.

The bootloader p-boot, here with 13 Linux distributions for the PinePhone

So installed the multi-distro demo image on a spare microSD card with USBImager, resized the partition to the microSD card's full size and then I have been trying all 13 distributions for the last few days. These are my notes for the 2020-09-14 version of the image, not in any way systematic or complete: 1

Arch Linux

Boots into a command line. I don't even see a virtual keyboard appearing to log in. This is not what I expect from a phone, but it's very understandable coming from the Arch Linux philosophy :-)

LuneOS

Has a bit more apps installed by default: maps, email, a PDF viewer, ... The maps app uses Google Maps data, but it doesn't work. I can't find general settings. It seems somewhat limited and basic things such as rebooting or shutting down don't work. LuneOS is based on webOS.

Maemo Leste

Boots into landscape mode, and I can't find how to change it to portrait mode. It has Vim and Htop installed by default, but they crash. There's no browser installed by default. It has an app manager, but it doesn't have a lot of apps available to install. The settings are fairly limited. The virtual keyboard is barely usable. Maemo Leste is the successor of Nokia's Maemo and it's currently an ARM64 port of Devuan (Debian without systemd).

Mobian

Has a good set of default apps. It has both Web and Firefox installed as web browsers. There's a To Do app, Processor usage, Telegram (the Desktop version), Maps (using OpenStreetMap), Geary for email. There's also an Authenticator app, but it doesn't show anything. There's no terminal installed by default. 2 Software shows a lot of apps, and you get access to a lot of GNOME settings, including accounts, for instance to set up your email account (Geary uses the accounts you have set up here). It definitely feels Linux-like, but at the same time feels like a real mobile operating system. It's essentially just Debian for ARM64 with the Phosh user interface developed by Purism for the Librem 5, and it works surprisingly well.

KDE Neon

Looks nice, there are but not many apps installed. I'm more familiar with the GNOME world, so I feel a bit lost here. There's Spacebar (to chat), Koko (music), Okular (document viewer), Konsole. Contrary to all other distributions I tried, there's a screenshot app in the drawer. But how does it work? I don't think I will get used to the Plasma Mobile interface. So while it looks like a perfectly usable system, it's not for me.

postmarketOS with fbkeyboard

Boots into a command line. The virtual keyboard needed to log in is quite usable, but this is not what I expect from a phone.

postmarketOS with GNOME

I hear a high-pitched sound when the PinePhone boots into the graphical environment. The virtual keyboard has awkwardly small buttons. There's a terminal installed. But the apps are not easy to access: if you open the application drawer, the icons suddenly fly off the screen, so the only way to start the apps is by typing an app's name in the search box. There are a lot of GNOME settings. However, the system doesn't seem to be adapted to a mobile screen: windows are shown only partially, even the GNOME initial setup window. All in all, this is not very usable in this state.

postmarketOS with Phosh

Also gives the high-pitched sound, but less long. There are not many apps preinstalled. There are also not many apps to install from Software. There are a lot of GNOME settings. Not all windows are adapted to the mobile screen size.

postmarketOS with Plasma Mobile

I hear a high-pitched sound when the PinePhone boots into the graphical environment. A limited set of preinstalled apps. The system doesn't seem to be very responsive. I couldn't even enter my Wi-Fi settings because the interface hangs when I want to open the settings window. Unusable in this state.

postmarketOS with sxmo

A nice lightweight alternative using suckless programs. Sxmo (Simple X Mobile) is actually a collection of simple programs and scripts to create a fully functional mobile user interface adhering to the Unix philosophy. It's a bit too minimalistic for me, but it works nicely to my surprise, controlling the user interface largely through the PinePhone's volume and power buttons and swipe gestures.

PureOS

Has a limited set of apps. It seems to be better adapted to the mobile screen size than some of the other distributions. The default web browser is GNOME Web. There's no Firefox to install from the Software app. There seems to be an Iceweasel in the app, but it hasn't been updated since the version equivalent to Firefox 55! The available GNOME settings are rather limited.

Sailfish OS

Looks very nice. It has a very user-friendly user onboarding experience. There's a fairly limited set of installed apps. The user interface is closed source, so this is no option for me.

Ubuntu Touch

Looks very nice, it also a nice user onboarding experience. The default web browser is Morph. It also has a media player, weather app, music, terminal, navigation (uNav, but it didn't work for me, as it keeps "waiting for GPS"). It seems to be a bit slower to boot and less responsive than some of the other distributions. I also experienced some weird behaviour with the window placement.

Powered by Linux

Mobian is a surprisingly useful Debian distribution on the PinePhone.

The PinePhone supports even more Linux distributions: Fedora, Manjaro, Nemo Mobile (an open source build of Sailfish OS), NixOS, openSUSE and AVMultiPhone (basically postmarketOS with MATE). These are not (yet) available from the multi-distro demo image, and I haven't tried them yet.

As you see in my notes above, there are still a lot of issues if you want to use the PinePhone on a daily basis. 3 Most of them will probably be solved eventually: things seem to move very fast in this domain. Also, the fact that these are all just Linux distributions is extremely helpful. If you encounter a problem, you can have a look at dmesg, run systemctl list-units --state=failed and so on and you have the same powerful toolbox at your disposal as on the Linux desktop.

All in all, Mobian impressed me most, with PureOS as my second choice. So after distro hopping for the last few days, I'm going to do a clean install with Mobian and explore it further. The feeling of having freedom and control on my phone in a familiar environment is amazing. Powered by Linux!

1

I haven't tested phone calls or SMS messages, as I haven't put my SIM card yet in the PinePhone.

2

Apparently there is. It's called King's Cross. For some reason I never bothered to click on that weirdly looking icon.

3

For instance, on none of the distributions I tested I was able to get the camera working, but this is probably due to the kernel used in the multi-distro demo image, as it seems to work in the "official" Mobian image that I installed later. I also couldn't take a screenshot on any system, even with scrot. Later I learned that on Phosh based distributions you can take screenshots with the command-line tool grim.

Fixing the Arduino IDE for the ESP32/ESP8266 on Ubuntu 20.04

If you want to use the Arduino IDE with an ESP32 or ESP8266 microcontroller on Ubuntu 20.04, you get the following error when compiling your sketch:

Traceback (most recent call last):
  File "/home/koan/.arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 37, in <module>
      import serial
      ImportError: No module named serial

This is because esptool.py is called as a Python 2 program, and Python 2 is deprecated. Ubuntu 20.04 doesn't even have pip anymore for Python 2, so you can't even install the serial module to solve this error.

There are two solutions to this problem: using Python 3 or Python 2.

Installing the module for Python 3

One solution is to create a symlink from /usr/bin/python to /usr/bin/python3. Ubuntu even has a package for this:

$ sudo apt install python-is-python3

After this install the pyserial module:

$ pip3 install pyserial

If you already have the serial module, you have to remove it first:

$ pip3 install serial

After this, try compiling your sketch again. This time the Arduino IDE calls esptool.py with Python 3, and thus finds the module you just installed with pip3.

Note

This approach makes python behave like python3 globally. So if you rely on other legacy Python software, this may not be the best approach.

Installing the module for Python 2

Another solution is to keep using Python 2 and thus installing the missing module with pip2. First enable the universe repository, where Python 2 has been moved to in Ubuntu 20.04:

$ sudo add-apt-repository universe

Update your package list, install Python 2 if you don't have it (it's not included anymore by default in new Ubuntu installations, but the Arduino IDE would have shown you another error in that case) and then pip2:

$ sudo apt update
$ sudo apt install python2
$ curl https://bootstrap.pypa.io/get-pip.py --output get-pip.py
$ python2 get-pip.py

Then finally install the missing module:

$ pip2 install pyserial

After this, try compiling your sketch again. The Arduino IDE still calls esptool.py with Python 2, but this time it finds the module you just installed with pip2, so the error disappears.

Using the M5Core2 for ESP32 development

A Wi-Fi enabled microcontroller such as the ESP32 or ESP8266 has a lot of applications, but if you use a development board, the result doesn't really look appealing in your living room, unless you spend some effort in creating your own case.

Enter M5Stack, a young Chinese company that offers a lot of ESP32-based products that don't look like development boards but like finished products. I first encountered them two years ago when I discovered their M5Stack Core BASIC Kit: a modular, stackable, ESP32 board with a 2 inch LCD screen, all in a package that doesn't look bad in your living room. I bought one and turned it into a dashboard for my MQTT-based home automation system: it could read the temperature and humidity of various sensors in my home by subscribing to the corresponding MQTT messages and showing them on the screen.

M5Core2, the newest ESP32 product of M5Stack

Now M5Stack has sent me their newest product: the M5Core2. It has more RAM, a touch screen, a real-time clock and even a microphone. This looks like an even better dashboard for my home automation system, adding a lot of possibilities.

Although from time to time you still bump into Chinese data sheets or code comments, I noticed that M5Stack's documentation and code has improved considerably since I last checked it for the first generation of this product. For the new M5Core2 there's:

I'm definitely going to experiment with this device and see how I can integrate this in my home automation system. I can reuse a lot of existing Arduino libraries, and together with the M5Core2 Arduino library it should be fairly straightforward to come up with a prototype.

Some specifications:

ESP32-D0WD-V3

240 MHz dual core, 600 DMIPS, 520 KB SRAM, Wi-Fi, dual mode Bluetooth

Flash

16 MB

PSRAM

8 MB

Input Voltage

5 V @ 500 mA

Interface

TypeC x 1, GROVE (I²C + I/O + UART) x 1

IPS LCD Screen

2.0"@320*240 ILI9342C

Touch Screen

FT6336U

Speaker

1W-0928

LED

Green power indicator light

Button

Power button, RST button, Virtual screen button x 3

Vibration reminder

Vibration motor

MIC

SPM1423

I2S Power Amplifier

NS4168

6-axis IMU

MPU6886

RTC

BM8563

PMU

AXP192

USB Chip

CP2104

DC-DC Boost

SY7088

TF card slot

16 GB max

Lithium Battery

390m Ah @ 3.7 V

Antenna

2.4 GHz 3D antenna

Operating temperature

0°C to 40°C

Net Weight

52 g

Gross Weight

70 g

Product Size

54 mm x 54 mm x 16 mm

Package Size

75 mm x 60 mm x 20 mm

Case Material

Plastic

How to test a Python program with command-line arguments in pytest

Recently I wanted to test how a Python program behaved with various combinations of command-line arguments.

Thanks to pytest's mocker fixture, this is easy to do:

def test_arguments_from_cli(mocker):
    """Test whether arguments from the command line are set up correctly in a HermesApp object."""
    mocker.patch(
        "sys.argv",
        [
            "rhasspy-hermes-app-test",
            "--host",
            "rhasspy.home",
            "--port",
            "8883",
            "--tls",
            "--username",
            "rhasspy-hermes-app",
            "--password",
            "test",
        ],
    )
    app = HermesApp("Test arguments in init", mqtt_client=mocker.MagicMock())

    assert app.args.host == "rhasspy.home"
    assert app.args.port == 8883
    assert app.args.tls is True
    assert app.args.username == "rhasspy-hermes-app"
    assert app.args.password == "test"

You just patch the sys.argv variable with the command-line arguments you want to test your program with. This variable is a list of strings representing the arguments as separated by a space, and with the command's name as the first element. So this example function patches sys.argv to behave as if the test function is running in a command you started as follows:

$ rhasspy-hermes-app-test --host rhasspy.home --port 8883 --tls --username rhasspy-hermes-app --password test

In this case the HermesApp object uses the argparse library to populate some attributes with values from sys.argv, such as host, port, tls, username and password. The function test_arguments_from_cli tests whether these attributes are initialized correctly. But you can use the same approach if you have a main function for your command that reads your command-line arguments.

If you want to test other combinations of command-line arguments, you just add another test function that patches sys.argv with another list of arguments. All these test functions are executed independently with their own patched argument list.

Automatically save the volume of your ReSpeaker 2-Mics Pi HAT

For my Rhasspy voice assistant I use the Seeed ReSpeaker 2-Mics Pi HAT on a Raspberry Pi: it's cheap and has two on-board microphones, so you only need to a add a speaker to use it for voice projects.

The drivers are open source and they can be installed easily from the respeaker/seeed-voicecard repository together with supporting systemd services. However, every time you reboot your Raspberry Pi, the playback volume is set to its maximal value, so the feedback sounds and replies from your voice assistant echo through the whole house.

I never bothered to change this behaviour, but when someone on the Rhasspy forum asked about it yesterday (Rhasspy with respeaker 2 hat starts with highest playback volume) I decided to dive into the details.

It turns out that the seeed-voicecard.service systemd script removes your system's asound.conf and asound.state at each boot, replacing them by its default files.

If you know this and read the /usr/bin/seeed-voicecard shell script, the solution for the state file (containing your volume settings) is simple: let the ALSA service store to and restore from this file installed by the Seeed driver, /etc/voicecard/wm8960_asound.state. Have a look at the alsa-restore.service file:

$ systemctl cat alsa-restore.service
# /lib/systemd/system/alsa-restore.service
#
# Note that two different ALSA card state management schemes exist and they
# can be switched using a file exist check - /etc/alsa/state-daemon.conf .
#

[Unit]
Description=Save/Restore Sound Card State
Documentation=man:alsactl(1)
ConditionPathExists=!/etc/alsa/state-daemon.conf
ConditionPathExistsGlob=/dev/snd/control*
After=alsa-state.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=-/usr/sbin/alsactl -E HOME=/run/alsa restore
ExecStop=-/usr/sbin/alsactl -E HOME=/run/alsa store

You need to change the latter two lines. So edit the service file to override these lines:

$ sudo systemctl edit alsa-restore.service

Then add the following lines containing the filename of the Seeed driver's asound state file:

[Service]
ExecStart=
ExecStart=-/usr/sbin/alsactl -E HOME=/run/alsa -f /etc/voicecard/wm8960_asound.state restore
ExecStop=
ExecStop=-/usr/sbin/alsactl -E HOME=/run/alsa -f /etc/voicecard/wm8960_asound.state store

The empty ExecStart and ExecStop commands are needed because you want to replace the original commands instead of adding extra commands.

Reload the systemd configuration after your changes:

$ sudo systemctl daemon-reload

Then reboot. After this, every change you make to the ALSA mixer (for instance with the alsamixer command) will be saved when your Raspberry Pi shuts down and reloaded when it boots, so it always uses the volume you have set.

Note

If you don't need your changes to the ALSA mixer to be stored automatically, you can ignore the previous steps and just store the state manually every time after you change it: sudo alsactl -f /etc/voicecard/wm8960_asound.state store.

Flashing a Shelly RGBW2 with crocodile clips and cut resistor leads

The Shelly RGBW2 is a nice Wi-Fi RGBW LED controller. 1 But I want to have open source firmware on as much devices as possible at home. So after playing with the original firmware for a while (for my book about self-hosted home automation) I decided to flash ESPHome on the device.

ESPHome doesn't list the Shelly RGBW2 as an officially supported device, but as it's built around an ESP8266, it should run the firmware fine. It's just a matter of finding the right firmware configuration and the right pins to connect the USB to TTL adapter cable to flash the firmware.

Finding the device pinout isn't an issue. Shelly has extensive documentation about the RGBW2 and neatly shows the pins under the header Flash/Debug:

/images/shelly-rgbw2-pins.png

However, connecting to the pin header is something else. While the Shelly 1 has a standard-size 2.54 mm pitch header that fits normal (Dupont) jumper wires, the RGBW2 (as well as the Shelly 2.5) has a 1.27 mm pitch header, which is considerably smaller. So you need some kind of adapter. 2

I found a number of solutions for this:

The latter looks like the nicest solution, but I had to order the 1.27 mm pitch header and I wanted to flash the device now, so I thought about the easiest way to create an adapter with the stuff I had lying at home.

After giving it some thought, I MacGyvered the following solution:

/images/shelly-rgbw2-crocodile.jpg

Yes, that's right, these are four crocodile clips and some thin wires put into the header.

The wires are leads I cut off two resistors. This gives you four wires, but you need to connect five pins. However, while flashing you have to pull GPIO0 to ground to start the boot loader in flash mode, so you can just fold one of the resistor leads and put one end in the GPIO0 hole and the other one in the GND hole. The folded wire is then connected to the white crocodile clip you see at the top.

The crocodile clips then connect to jumper wires that go into the breadboard, where the USB to TTL adapter is plugged in, and then connected to my computer to flash the device with the firmware.

This works, but you have to make sure that the resistor leads fit right into the header holes. After some wiggling, the esphome flasher recognized the serial connection, and the result was ESPHome on my RGBW2, which integrates nicely with Home Assistant.

And after this, you can apply OTA (over-the-air) updates of the ESPHome firmware, so you don't need the MacGyver adapter anymore.

1

The nice thing about Shelly is that they give you the choice: use their devices with their own cloud services, or use them completely self-hosted with your own home automation gateway on your local network, such as Home Assistant. Shelly's firmware has an HTTP API and also supports a subset of the features with MQTT.

2

Trust me, I tried to cram the jumper wires in the header. They really don't fit.

Create a table without a header in reStructuredText and Markdown

Have you ever written a table without a header in Markdown? It turns out that most Markdown parsers don't support tables without headers.

When I wrote the page about my book Control Your Home with Raspberry Pi on this website, I wanted to list some specifications (title, publication date, number of pages, ...) in a table without a header. In reStructuredText, which I'm using for this website, the code for the table looks like this:

+----------------------+-------------------------------------+
| **Title**            | Control Your Home with Raspberry Pi |
+----------------------+-------------------------------------+
| **Author**           | Koen Vervloesem                     |
+----------------------+-------------------------------------+
| **Publication date** | 2020-08-17                          |
+----------------------+-------------------------------------+
| **Number of pages**  | 331                                 |
+----------------------+-------------------------------------+
| **Price**            | € 37.50                             |
+----------------------+-------------------------------------+
| **ISBN-13**          | 978-1-907920-94-3                   |
+----------------------+-------------------------------------+
| **ISBN-10**          | 1-907920-94-3                       |
+----------------------+-------------------------------------+
| **Publisher**        | Elektor International Media (EIM)   |
+----------------------+-------------------------------------+

This is rendered by Nikola, the static site generator I'm using, as:

Title

Control Your Home with Raspberry Pi

Author

Koen Vervloesem

Publication date

2020-08-17

Number of pages

331

Price

€ 37.50

ISBN-13

978-1-907920-94-3

ISBN-10

1-907920-94-3

Publisher

Elektor International Media (EIM)

This looks fine as a simple table without a header.

Now I wanted to do the same in Markdown in the corresponding GitHub repository with code examples. I could have written the README as a reStructuredText file, but I already created a README.md out of habit, so I tried to create the same table without header in Markdown. But apparently GitHub-flavoured Markdown and many other Markdown flavours don't support tables without headers.

That StackOverflow post linked above shows a hack that seems to work in many Markdown parsers, including in GitHub:

|    <!-- -->          |        <!-- -->                     |
|----------------------|-------------------------------------|
| **Title**            | Control Your Home with Raspberry Pi |
| **Author**           | Koen Vervloesem                     |
| **Publication date** | 2020-08-17                          |
| **Number of pages**  | 331                                 |
| **Price**            | € 37.50                             |
| **ISBN-13**          | 978-1-907920-94-3                   |
| **ISBN-10**          | 1-907920-94-3                       |
| **Publisher**        | Elektor International Media (EIM)   |

This adds HTML comment blocks in the header cells, which essentially adds an empty header row to the table. Unfortunately in GitHub the result looks a bit odd, with that compressed empty header row:

/images/markdown-table-without-header.png

This is literally an ugly hack. Of course I can just create an HTML table without a header in the Markdown file, as shown in one of the StackOverflow answers, but that defeats the purpose of using a more human-centered markup language. So now I have converted the README file from Markdown to reStructuredText. The result is rendered by GitHub as:

/images/rst-table-without-header.png

Just like I wanted. It's these small quibbles with Markdown all the time that strenghten my preference for reStructuredText as a markup language.

Heavy reading

/images/heavy_reading.jpg

I knew I had some heavy reading on my bookshelf, but I was just surprised by this view in my home office. Coincidentally, it was one of the mathematics shelves. 1

1

Actually, the shelf broke by what you don't see in this picture: the stack of magazines that stood in front of the row of books and is now lying on the floor all around the room.

Automating society

Earlier this year, I wrote two articles for AlgorithmWatch:

Both articles were part of AlgorithmWatch's upcoming report Automating Society 2020, which investigates the applications of automated decision-making in the public sector in various European countries.

The full report will be published this fall, but the abridged versions of both stories that will come in the report have now been published on the website Ethics of Algorithms of the Bertelsmann Stiftung:

Especially the Dutch story about SyRI, an algorithm cross-referencing personal data from citizens in various databases, makes it clear that algorithmic fraud detection has a lot of pitfalls: the way it was used in the Netherlands there was no transparency (the exact algorithm or risk model was never disclosed), it exacerbated biases and discrimination, and it was used for purposes it wasn't designed for.

The best argument against such systems is actually quite simple: you don't need them. As Ronald Huissen from the Platform Bescherming Burgerrechten said, the government doesn't need this kind of mass surveillance to prevent fraud:

The government already has information about who owns which house, so it could check this before granting the person a rental allowance. For all big fraud scandals in social security we have seen in the past decades it became clear afterwards that they could have been prevented with simple checks beforehand. That happens far too little. It is tempting to look for solutions in secret algorithms analyzing big data sets, but often the solution is far simpler.

It's an issue I see all too often: there's a problem (social fraud, fiscal fraud, crime, a virus outbreak, ...) and the government thinks it has to solve this with a big centralized database, massive surveillance and some technological voodoo (big data, AI, an app, or the next big thing). The solution is often much simpler: everyone just has to do their job instead of outsourcing it to technology.