ops_keisha
@kstewart · Joined March 2025
Plant manager · food and beverage
- Topics
- 8
- Replies
- 64
- Points
- 0
Forum Replies Created
Plant manager, not a security person, but we lost a board out of a remote pump house last year and I had to clean up after it, so this is what we do now.
One credential per device, never shared. If two boards ever run the same token, the second login takes over the first one’s session and you spend a week chasing “flapping” devices. Tokens and the PSK go into NVS at the bench, like Aoife says, so the binary carries no secrets. Flash encryption on production boards once you’ve practised on a couple of spares.
On the Hermetry side, the profile has an Allowed networks field. Put the public address range your cabinet routers use in it. A token dumped from a stolen board is then refused from anywhere else (CONNACK 5, and the reason shows on the device’s Log tab). That one setting made our IT people much calmer.
When a board goes missing, revoke its credential (the API has DELETE on the device’s credentials) or just delete the device, and the old token stops working at once, with every open session cut. Admins also get a notice when a device logs in from a network it hasn’t used before, so watch for those.
For the PSK, put the sensors on their own SSID or VLAN on the router so a leaked PSK only gets you onto a network with nothing else on it.
Most of this is on the device credentials page.
No, kept them separate on purpose, that’s why the opto is there.
We bought finished trackers, so no help sorry. Following though, we’re thinking about our own boards.
Separate question, can you get an alarm when only one of the two temperature keys stops arriving? We had a board die quietly and the device stayed online.
As the one who signs off on the depot visits: please check this before the next batch gets flashed.
On Forge: there’s no doc page for it, so I can’t tell you whether it counts rejects or how you’d switch it on. What Lewis describes is what we do on our packing lines: counters as telemetry and a bar chart per hour with the increase aggregate. The supervisors look at it at shift change.
Then the full image is fine, I’d stop worrying about delta. We pushed a 1.6 MB image to about 200 controllers at our plant last spring. The rollout is what made it calm.
On the package assignment pick Every device of the profile and turn on Send in waves. The usual plan is a canary of 1% (at least one, at most ten), then 10%, then 50%, then the rest, with 15-30 minute waits between, and it pauses itself if a wave has more than 5% failures. For a hotel I’d edit the waves: first wave 3 rooms you can walk to, wait a couple of hours, then 20, then the rest overnight. Nobody downloads before their wave, so the Wi-Fi only ever sees a few at once.
Other things that helped us:
- Test with One device assignments on two units first.
- The download link expires after 15 minutes but the device can ask again and resume with an HTTP Range request, so a flaky room isn’t a disaster.
- A device that doesn’t come back on the new version shows as Went back to the old version, that’s your rollback working.
Details: Rolling out firmware in waves.
We had this with old shields on a bottling line, two problems stacked on top of each other.
Forcing the switch port to 100/full is what killed traffic after you got a link. The W5100 still auto-negotiates. When the other side is forced, auto-negotiation can’t find out the duplex and falls back to half duplex. Switch on full, shield on half, and you get a duplex mismatch: link up, almost nothing passes. Put the ports back to auto, or force both ends (and you can’t force the W5100 from the Arduino library).
The original no-link problem is very often the reset on those clone shields. The W5100 comes out of reset before its supply is stable and its PHY sometimes never negotiates properly. The old switch was forgiving, the new one isn’t. Fix that worked for us:
// shield RESET pin wired to D8 with a jumper wire
pinMode(8, OUTPUT);
digitalWrite(8, LOW);
delay(50);
digitalWrite(8, HIGH);
delay(500); // let the PHY settle
Ethernet.begin(mac, ip);And if one still refuses, a cheap unmanaged 10/100 switch between the shield and the managed one ends the argument. Long term, W5500 shields don’t have this problem.
A 9 V block won’t last a week with the Nano’s regulator and LED. I’d not go there.
Is 8883 even open on that network? Some places only allow 80 and 443 out.
Plant manager view: I’d be nervous letting a cloud rule move a furnace setpoint. What happens if the internet drops halfway?
Aisha’s right, and I’d add a few things from running Pi gateways on a production floor.
Buy a splitter that says 802.3af/at on it and lets the switch negotiate, never a passive injector. Check the cable run, long thin patch leads drop enough voltage that the Pi throttles. You can see that from the Pi itself:
$ vcgencmd get_throttled
throttled=0x50005 # bit 0 = under-voltage now, bit 16 = has happened since boot
I’d send that value to Hermetry with the probe readings. The gateway can report its own telemetry on v1/telemetry as a normal device, while the probes go on v1/gw/telemetry as children. Then an alarm rule on the gateway’s profile for pi.undervolt == 1 tells you the supply is marginal before the SD card gets corrupted.
Remember that when the gateway goes offline, every child goes offline with it. So the gateway’s offline alarm is the one that matters, and the children’s offline alarms will all fire too unless their delay is longer. We put the gateway on 300 s and the children on 900 s. The child behaviour is described here: Gateways and devices behind a gateway.
And keep the SQLite buffer you’d use anyway, a PoE switch reboot during a firmware update is a 2-minute outage.
Nice. We added an “are you sure” prompt to ours for the dock lights after someone switched them off with a driver still reversing in.
The EN capacitor saved us too. Our line counter boards would boot-loop maybe one power-up in twenty without it.
Did you try ifdown first and then ifup?