Aoife put her finger on it. We run AGVs in a pharma warehouse and had the same failure until we changed three things.

Mechanical: let the dock do the final alignment. A V-shaped funnel or two angled guide rails that steer the robot into position in the last 15 cm, with spring-loaded contacts that have some travel. IR only has to get it into the funnel. Michael’s wider contacts help, but the funnel is what fixes the crooked case.

Logic: “docked” means charging current flows, not “IR is strongest”. After stopping, wait 2 s and read the charge current. If it’s not there, back off 30 cm, approach again, up to 3 tries. After 3 failures stop trying and report it.

Twin side: place each dock (if it reports anything) on the room asset it sits in, and place the robots on the site, since they move between zones and report their zone in telemetry. Then derive a status key for the robots, either on the Pi or in the root flow: critical after 3 failed docking tries or battery under 15 %, warning while it’s retrying, ok otherwise. Site status takes the worst status of everything placed under it, so a stuck robot turns the hall red on the site view without anyone opening telemetry. Also store a numeric copy (0/1/2), because text keys have no chart history. How the status rolls up: Site status and the status key.

Jessica nailed the main issue: at 15-20 cm you’re inside the blind zone. Mount it higher, a short PVC pipe section as a stand-off works, so full tank is at least 30 cm away.

Second issue is the Python timing. Busy-wait loops on a Pi 3 miss edges whenever the scheduler steps in. Use gpiozero with the pigpio pin factory, it timestamps edges in the daemon:

from gpiozero import DistanceSensor
from gpiozero.pins.pigpio import PiGPIOFactory
import statistics, time

s = DistanceSensor(echo=24, trigger=23, max_distance=4.5,
                   pin_factory=PiGPIOFactory())

def level_cm():
    vals = []
    for _ in range(7):
        vals.append(s.distance * 100); time.sleep(0.1)
    return statistics.median(vals)

Median of 7 throws away the odd echo off the tank wall. Then send {"distance_cm": d, "level_pct": ...} as numbers and put an alarm rule on level_pct below 20 with a debounce of 3 readings, so one bad echo doesn’t wake the warden.

The String is gone before you use it. (String("barn-") + WiFi.macAddress()) makes a temporary that’s destroyed at the end of that line, so clientId points into freed memory. Whatever the heap does next ends up in your client ID. Bit me on a sensor board years ago and I wasted a day too.

Same trap with setServer(host.c_str(), ...): PubSubClient keeps the pointer, not a copy, so if host is a local String in setup() you’ll get odd reconnect failures later.

Use buffers that live as long as the client:

static char clientId[32];
static char brokerHost[64];

void setupMqtt() {
  uint8_t mac[6];
  WiFi.macAddress(mac);
  snprintf(clientId, sizeof(clientId), "barn-%02x%02x%02x", mac[3], mac[4], mac[5]);
  strlcpy(brokerHost, settings.host.c_str(), sizeof(brokerHost));
  mqtt.setServer(brokerHost, 8883);
}

// later
mqtt.connect(clientId, settings.token.c_str(), "");

settings.token.c_str() is fine inside the call as long as settings lives globally.

Aisha and Ciaran have the two causes. I did this for a Nano in an air handler once, here’s the whole recipe for the ZS-040 board.

1. Put the HC-05 in AT mode: hold the little button while powering it, the LED blinks slowly (about every 2 s). Talk to it at 38400 from a USB-serial adapter. 2. Set it up (115200 if your Nano works with “ATmega328P”, 57600 if only “Old Bootloader” works):

AT+ROLE=0
AT+UART=115200,0,0
AT+POLAR=1,0

AT+POLAR=1,0 makes PIO11 go low while a link is up. 3. Reset wiring. The ZS-040 doesn’t bring PIO11 out to a header pin, so solder a thin wire to pin 32 of the module. Connect it to the Nano’s RESET through a 0.1 uF capacitor, the same way the USB chip’s DTR does it. When avrdude opens the port, the link comes up, PIO11 drops, the capacitor turns that into a short reset pulse, and the bootloader is listening. 4. Disconnect STATE from RESET.

It’s fiddly, and the first sync after pairing sometimes fails, so try twice before you start debugging again.

Aisha has it. Hermetry stores a body that isn’t JSON as a single raw key, which is why you see your template text. Build the JSON with printf, and send all sites in one message, cheaper and tidier:

#!/bin/bash
json="{"
while read -r ip name; do
  if ping -c 3 -W 2 "$ip" > /dev/null 2>&1; then ok=1; else ok=0; fi
  json+=$(printf '"%s.up":%d,' "$name" "$ok")
done < /home/pi/sites.txt
json="${json%,}}"
mosquitto_pub -h "$BROKER" -p 8883 --capath /etc/ssl/certs -u "$TOKEN" -q 1 
  -t v1/telemetry -m "$json"

Three pings, not one: counts as up if any one comes back.

For the alert, an alarm rule on the profile per site key with a count debounce, so it needs several bad readings in a row before it opens and several good ones to clear:

{ "type": "pump3_unreachable",
  "title": "Pump 3 modem not answering",
  "severities": [ { "severity": "MAJOR", "condition": { "key": "pump3.up", "op": "==", "value": 0 } } ],
  "debounce": { "mode": "count", "raise": 5, "clear": 2 },
  "enabled": true }

Five minutes of failed pings before anyone hears about it. Add a Device offline rule for the Pi too, otherwise a dead Pi looks like “everything fine”. Debounce options: Debounce, hysteresis and dampening.