One weird trick to re-enable MQTT messages for uplink fields in The Things Network with mqttwarn

I use The Things Network to get sensor measurements from a couple of LoRaWAN sensors in the garden and the garden shed. In a previous blog post, I already showed how I bridged The Things Network to my local MQTT broker.

The Things Network publishes MQTT messages with a JSON payload on the topic APPID/devices/DEVID/up. This looks like:

{
  "app_id": "APPID",
  "dev_id": "dragino-lht65-1",
  "hardware_serial": "AAAAAAAAAAAAAAAA",
  "port": 2,
  "counter": 22608,
  "payload_raw": "z18FWhLpQZMly/5=",
  "payload_fields": {
    "BatV": 2.95,
    "Hum_SHT": "91.0",
    "TempC_DS": "1.56",
    "TempC_SHT": "1.15"
  },
  "metadata": {
    "time": "2020-12-04T19:38:02.270923288Z",
    "frequency": 867.1,
    "modulation": "LORA",
    "data_rate": "SF7BW125",
    "airtime": 61696000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-9999999999999999",
        "timestamp": 1184233211,
        "time": "2020-12-04T19:38:02.214832067Z",
        "channel": 0,
        "rssi": -91,
        "snr": 9.5,
        "rf_chain": 0
      }
    ]
  }
}

I have a couple of ESP32 devices around the house that extract sensor measurements from MQTT messages and show them on a display. On these microcontroller-based devices I like to simplify [1] the parsing, so I prefer MQTT messages with just a number as the payload.

In the past The Things Network not only published the full JSON payload, but also published the values from the payload fields to individual MQTT topics [AppID]/devices/[DevID]/up/[Field]. This made it much easier to read the values. However, at the end of 2019 The Things Network disabled uplink fields on MQTT for performance reasons.

But if you're bridging The Things Network to your own local MQTT broker, you can parse the JSON payload and republish the payload fields as separate MQTT messages yourself.

Mqttwarn to the rescue

It turns out that this is quite easy to do with mqttwarn, which I was already running as a notification system for my MQTT-based home automation system. It's a bit of a hack, but it works well.

In your mqttwarn.ini, define the following topic section:

[ttn/+/devices/+/up]
targets = log:info
format = the_things_network_uplink_fields()

This supposes that all incoming messages from the Things Network are translated by your MQTT bridge to subtopics of ttn, and that you have a log service defined and enabled in the launch line.

Now in your funcs.py, define the the_things_network_uplink_fields() function:

def the_things_network_uplink_fields(data, srv=None):
    """Extract application ID, device ID and uplink fields
    from JSON payload of The Things Network messages on topic
    ttn/+/devices/+/up and publish the uplink fields on separate
    MQTT topics ttn/+/devices/+/up/[Field]."""
    if type(data) == dict and "app_id" in data and "dev_id" in data and "payload_fields" in data:
        app_id = data["app_id"]
        dev_id = data["dev_id"]
        base_topic = "/".join(["ttn", app_id, "devices", dev_id, "up"])
        if type(data["payload_fields"]) == dict and srv is not None:
            for field, value in data["payload_fields"].items():
                srv.mqttc.publish("/".join([base_topic, field]), value, qos=0, retain=False)
            return "Published uplink fields of " + base_topic

    return None

So what's happening here? If mqttwarn receives a ttn/+/devices/+/up topic, it logs this. To define the log message, it calls the function defined in the format line. This functions has access to the data from the MQTT message as a dict. So in this function I decode the app ID and device ID, extract all uplink fields from the payload_fields dict and then for every field publish the corresponding value to the MQTT topic consisting of the original topic with the field's name as a subtopic.

The result? Not only do I get the JSON payload of above on ttn/APPID/devices/DEVID/up, but a mosquitto_sub -t 'ttn/#' -v also gives the following messages:

ttn/APPID/devices/DEVID/up/BatV 2.95
ttn/APPID/devices/DEVID/up/Hum_SHT 91.0
ttn/APPID/devices/DEVID/up/TempC_DS 1.56
ttn/APPID/devices/DEVID/up/TempC_SHT 1.15

I can now easily let my ESP32 code subscribe to the previous MQTT topics to get the sensor measurements directly without having to parse a JSON payload.

The function the_things_network_uplink_fields is general enough that it works for all The Things Network payloads. So you don't need to add anything if you add new devices to The Things Network. [2]

Stop oversharing messages in the bridge

I also changed one thing in the original bridge configuration in mosquitto.conf from my previous blog post. Instead of configuring two-way sharing of messages (topic # both 0 ttn/ APPID/devices/), I made sure to only share the uplink and events messages from The Things Network to Mosquitto and only the downlink messages from Mosquitto to The Things Network:

topic +/devices/+/up in 0 ttn/ ""
topic +/devices/+/events/# in 0 ttn/ ""
topic +/devices/+/down out 0 ttn/ ""

These topics can be found in the MQTT API documentation from The Things Network.

I don't think the MQTT messages published by mqttwarn would result in a loop, but better safe than sorry: there's no need for Mosquitto to forward the payload field messages to The Things Network.