"Meshtrics:" A Nosy Neighbor's Guide to Meshtastic Airtime Metrics in Grafana

Meshtastic garners a lot of interest, a lot of criticism, and a lot of experimentation. I love things that provide people with an opportunity to experiment, and this tech is right up my alley. I am undecided on whether or not it’s the right choice for low-power, portable, unlicensed, distributed mesh communication, but it is worth tinkering with for any radio enthusiast, given the availability and relatively low cost of hardware.

What is Meshtastic?

For the uninitiated, Meshtastic is an open-source software suite built to utilize inexpensive radios operating in the 915MHz Industrial, Scientific, and Medical (ISM) Radio Band. This is handy, because this band does not require licensing to use. To quote the project directly:

Meshtastic utilizes LoRa, a long-range radio protocol, which is widely accessible in most regions without the need for additional licenses or certifications, unlike ham radio operations.

These radios are designed to rebroadcast messages they receive, forming a mesh network. This setup ensures that every group member, including those at the furthest distance, can receive messages.

Additionally, Meshtastic radios can be paired with a single phone, allowing friends and family to send messages directly to your specific radio. It’s important to note that each device is capable of supporting a connection from only one user at a time.

So great… who cares? Well, given that you can pick up LoRa devices based on ESP32 or NRF52 chipsets for as low as around $30, a lot of hobbiests care!

Other Meshes

I would encourage everyone to check out both Meshtastic and its most popular competitor MeshCore, which addresses some of the systemic configuration-based issues that plague meshtastic. I won’t say one is better than the other, primarily because the MeshCore is licensed with MIT, so the developers can monetize (fine), through closed-source premium offerings (not fine). Additionally, MeshCore’s mobile app is not open-source. 😑

Meshtastic is GPLv3 licensed, which makes me happy and is in the spirit of FLOSS software! However, as it doesn’t allow granular control of packet paths, people that setup repeater type nodes in poor locations can have a negative effect on the overall mesh, and you’re at the whim of their poor choices.

Why Metrics?

Why metrics…? Well, because we can! Also, this will help you determine if those unfortunate-minded individuals who have unknowingly hurt the mesh with poorly configured and placed repeaters are potentially affecting you. Start being a nosy neighbor and figure out how healthy your mesh is. Or, just keep track of your telemetry in your Grafana instance instead of being limited to the Meshtastic app!

Requirements

Assuming you’re up and running with at least one node on Meshtastic at this point, you’ll need a few other things to get this up and running.

1. An ESP32-based node to use as the hardware collector.
2. A working Grafana installation.
3. A working Prometheus installation.

Hardware Setup

First, flash your ESP32-based node with the latest and greatest Meshtastic firmware. I’m using a Heltec v3 for this purpose. Grab the firmware from the Meshtastic GitHub Releases Page. I would recommend the latest “beta” version.

Second, after you have a working node, configure it to connect to your wifi network and record its IP address. We’ll use that shortly for our Prometheus exporter.

As hardware and general Meshtastic config varies pretty widely, I’m leaving this as an exercise for the reader for the most part. Just know that the end state is a device you have routable connectivity to at a known IP address.

Software Setup

Now comes the fun part.

We have one critical piece of software that will help us with our journey, and it is the Prometheus exporter written by artiommocrenco: https://github.com/artiommocrenco/meshtastic-prometheus-exporter

The way I have structured my docker-compose manifest for this, and a lot of tweaking has led me to building the Docker image for this locally, so that’s what I’ll demonstrate here. I’ve also rewritten the Dockerfile for better layering.

Let’s first examine the docker-compose.yml file that we’ll be using for this. Feel free to make adjustments for your deployment strategy (or rewrite it for Podman!)

networks:
  meshtastic:
    driver: bridge
  metrics_metricsserver:
    external: true

volumes:
  redis_data:

services:
  exporter:
    build:
      context: ./meshtastic-prometheus-exporter
    restart: unless-stopped
    environment:
      - MESHTASTIC_INTERFACE=TCP
      - INTERFACE_TCP_ADDR=127.255.43.166 #Replace this IP with your network-connected meshtastic node's IP
      - REDIS_URL=redis://redis:6379
    networks:
      - meshtastic
      - metrics_metricsserver
  redis:
    image: docker.io/valkey/valkey:8
    restart: unless-stopped
    volumes:
      - redis_data:/data
    networks:
      - meshtastic

Pretty simple so far. Again, this is just going to get our collector running, I’m assuming you already have Prometheus and Grafana running elsewhere. My inclusion of the external metrics network in this example is an artifact of running those two services in a different compose manifest.

Also note the use of Valkey instead of Redis in our K/V container. We like free-as-in-speech software here.

The keen observer here will notice a build: key in the compose manifest. Now we need to come up with the code that it’s going to build. I’ll point you to my fork here, so we can capture the differences in Dockerfile, but this is effectively the unedited work of the original author. I spent a lot of time tweaking trying to get things more stable and eventually got back to the unmodified codebase.

From wherever you have your docker-compose.yml file saved, we need our build code in a subdirectory of that, so cd over to the folder your docker-compose.yml file is in then run:

git clone https://git.zarn.cloud/StarkZarn/meshtastic-prometheus-exporter

Test It!

Assuming you have edited network names and any other external dependencies or reverse proxies you need to to get this working with your Grafana and Prometheus installation, we should be ready to fire it up!

First stage your images:

sudo docker compose pull;
sudo docker compose build --pull;

Then start the stack and watch for errors. There will likely be some, but that’s not necessarily the end of the world. Did I mention this is still pretty brittle…?

sudo docker compose up -d

Speaking of errors… If you see something like this, no big deal. The exporter isn’t perfect and neither is all Meshtastic traffic. This is an example of what I most frequently see, a packet causing a python error, but the container keeps running.

You should be on the look out for errors involving radio connection — effectively any error that causes the container to restart.

Troubleshooting

If you run into issues with a restart loop:

• Power cycle the meshtastic node then restart the docker-compose stack
• Ensure you don’t have ANY other devices connecting to the IP of the meshtastic node. It does not handle multiple connections gracefully…
• Update your Meshtastic node to the latest firmware
• Update the Prometheus exporter to the latest version:
  • cd ./meshtastic-prometheus-exporter && git pull

Prometheus Config

I imagine you’re familiar with Prometheus already, but let’s look at the required snippet to get this Meshtastic data into our Prometheus instance.

The following should appear under the scrape_configs: key in your Prometheus config file.

scrape_configs:

  - job_name: 'meshtastic'
    static_configs:
      - targets: ['meshtastic-exporter-1:9464']

Edit container names as necessary, and restart your Prometheus instance and you should be collecting data!

Grafana Config

Okay, let’s see what we have to look forward to. These are just a couple examples, but likely the two panels that I use the most while trying to figure out what nearby nodes are contributing to congestion. Contending with the masses putting up repeaters in bad places or spamming telemetry that just sucks up airtime is the bane of Meshtastic.

Assuming you’ve got your Prometheus data source already plugged into your Grafana instance, all you need now are some dashboards! Feel free to tweak and build at your whimsy, but these are what I landed on (for now).

All six edited versions of the dashboards from the original project are included in the git repo in the grafana-dashboards folder. You should be able to import them to your instance. I made a “Meshtastic” folder to hold the dashboards in my Grafana instance. I find myself most often using the Telemetry (net) one.

If you’re looking for just the dashboards, the link is here: https://git.zarn.cloud/StarkZarn/meshtastic-prometheus-exporter/tree/main/grafana-dashboards

Conclusion

Okay, that’s it for now. This one is abstract, fast, easy, and short. And brittle. Did I mention it was brittle? I have been sitting on this draft for quite some time now working out stability issues with the TCP/Meshtastic interconnect. Honestly, I think just time helped more than any tweaking. After updating my node several times it seems to have gotten better. Don’t rely on this for anything terribly important though, that’s for sure. Hopefully with time it’ll improve.

Use those stats to find the offenders in your network, coordinate contacting people to help improve node configs, and make the network better! Or, maybe it’ll convince you that something like MeshCore is better even with its regretable licensing.

Happy meshing!

Support

If you’ve enjoyed this, consider helping support the infrastructure to run it.

Buy me a coffee or three with the button below.