New: the AI firewall watches the whole platform and explains each incident in plain words. See how it works

After a power outage, 60 ESP32s enter an endless reconnect loop

After a power outage, 60 ESP32s enter an endless reconnect loop

Solved 4 3

Translated from Türkçe by AI ·

Hello, sorry in advance for the long post.We are planning to set up about 200 small ESP32-based measurement boxes at the water treatment plant where I work, for level, flow and pump status. All of them will be in the same building, connect to the same access point, and send data to Hermetry over MQTT. We first piloted with 60 of them.During normal operation there is no problem. But last week the building had a 20-minute power outage, and when power came back, most of the boxes could not connect properly. On the serial port I am watching one of them, there is a constant connect-drop cycle, and on the dashboard devices keep switching between online and offline. I checked the Log tab of the devices:

Connected via mqtts
Disconnected  Reason: takeover
Connected via mqtts
Disconnected  Reason: takeover

Also, the administrators received a notification called “takeover fight”.The first thing that comes to mind is the access point. We are using a home-type AP, it may be struggling with 60 clients, and I don’t know what will happen with 200. Meanwhile, to speed up the pilot, we flashed the same firmware and the same access token onto all the boxes. I’ll mention that too, I don’t know if it’s related.

♥ 0

Translated from Türkçe by AI ·

Most consumer APs get clogged at 30-50 clients, I think that’s where the problem is. We need an enterprise AP.

Sahadan selamlar.
♥ 0

Translated from Türkçe by AI ·

İlgaz is right about the AP, but the takeover in the log is saying something else, and the main cause is the last sentence you wrote.In Hermetry, each device gets a single session. When a second connection from the same device arrives, it closes the first one and takes over. When 60 boxes connect with the same token, Hermetry sees them as one device: each new connection drops the previous one, the dropped one reconnects, and it loops endlessly. A device that keeps doing this from two networks is held back for a while, and that is the “takeover fight” notification. Normally the connections are spread out so you wouldn’t have noticed, but when power came back and all 60 went down and up at the same time, it blew up.What to do:1. Give each box its own credentials. Instead of adding 200 devices one by one, you can do a bulk CSV import from the Imports page (up to 1,000 rows per file). It shows a dry run first, and the tokens for new devices come in a file that can be downloaded only once. Alternatively, you can use a provisioning key in the profile so each device gets its own token on first boot.2. Add a random delay before reconnecting. If CONNACK 3 comes back, wait and retry, but don’t let them all retry in the same second:

uint32_t backoff = 2000 + (esp_random() % 8000);   // 2-10 sn
if (!mqtt.connected()) { delay(backoff); mqtt.connect(id, token, ""); }

3. Keepalive between 30 and 300 seconds. 60 is good.4. Don’t rely on a single home-type AP for 200 boxes; split them across 2-3 enterprise APs.Session and CONNACK details are on the MQTT page.

Kayıtta yoksa olmamıştır.
Accepted solution♥ 5

Translated from Türkçe by AI ·

Esra, thanks a lot, that’s exactly how it was. I imported the 60 devices via CSV, and we loaded the tokens from the file I downloaded into the boxes by serial number. Yesterday we deliberately pulled out the fuse and reinserted it, and all of them connected cleanly within 15 seconds, with no single takeover in the Log.We also changed the AP: as we scale to 200, we’re splitting it across three ceiling-type APs.

♥ 0

Translated from Türkçe by AI ·

+1 for jitter. We also had a field case where 40 sensors trying to connect at the same time caused some of them to hit address blocking; it was resolved with a random delay.

Sahadan selamlar.
♥ 2
Viewing 5 posts – 1 through 5 (of 5 total)
  • You must be logged in to reply to this topic.

Join the conversation

Sign in to ask and answer, like helpful posts and mark the reply that solved your problem. Every contribution earns points and levels.

Share