We run a Go service on an edge box at a solar site that reads inverters and publishes to Hermetry over MQTT with QoS 1. Link is a 4G router that drops a few times a day, usually for 30 s to a few minutes.
My mental model came from running our own Mosquitto years ago: the broker persists QoS 1 to disk, client reconnects with clean session = false, everything queued gets delivered. So I set CleanSession false on the client and assumed Hermetry holds things for us both ways.
Looking at the history after a drop I see gaps of exactly the outage length. Is Hermetry’s broker not persisting? Do I need to switch some setting on, or is the buffer my job?
What I checked so far: the device’s Log tab shows the disconnect with keepalive_timeout and a reconnect a few minutes later, as expected. Our client logs say the publishes during the outage returned without error, which is what made me think the broker had them. Inverter readings are every 30 s with a ts we take from the inverter’s own clock (NTP synced), sent as {“ts”: …, “values”: {…}}. We need the history to be complete because the site owner bills from it monthly, so a few minutes missing per day actually costs money.
Gaps of exactly the outage length sounds like nothing was published at all during the outage, not like the broker lost them. Is your client even queuing while disconnected?
Good point. Paho Go with the default in-memory store. I’d assumed it queues while offline.
From the management side: our integrator told us the same thing about gaps, they ended up adding storage on the box. Can’t tell you the details, sorry.
Clean session false won’t help on Hermetry anyway. I’m fairly sure the docs say it’s served as a clean session.
Kevin’s right, and that’s half of it. We hit the same thing on a wind site.
Hermetry’s MQTT is 3.1.1 and a CONNECT with clean session false is served as clean. Nothing is queued for a device while it’s offline, apart from commands that were sent as persistent. So no session state to resume, in either direction for telemetry.
What Hermetry does promise is on the way in: the PUBACK goes out only after the message is safely queued on their side. If it can’t be queued you get no PUBACK and the session closes, and the client is expected to resend after reconnecting. Duplicates of the same device, key and timestamp are stored once, so resending is safe.
So the buffer is yours, and the pattern is:
- Write every reading to a local store (we use a SQLite file) with its own ts in ms.
- Publish from the store, delete a row only when its PUBACK arrives.
- After reconnect: subscribe, ask for shared attributes, then send the backlog before live data.
Send the backlog as arrays of {ts, values} so it lands at the right times. The profile has a separate Replay messages per second limit for messages whose records are all older than a minute, so a long backlog drains over a while. Don’t panic when it doesn’t all appear at once.
The MQTT page has the acknowledgement section, worth reading once.
Yep. The Paho memory store was throwing everything away on disconnect, so Maria’s guess was spot on too. Put a bbolt file in front of the publisher, delete on PUBACK, and pulled the 4G antenna for 5 minutes as a test: backlog showed up in history with the right timestamps about 20 s after reconnect. Removed the CleanSession=false line as well since it did nothing.
One thing to add: cap the local store. We had a site offline for 9 days once and the disk filled up. Readings older than your retention get refused anyway.
- You must be logged in to reply to this topic.
Join the conversation
Sign in to ask and answer, like helpful posts and mark the reply that solved your problem. Every contribution earns points and levels.