New: the AI firewall watches the whole platform and explains each incident in plain words. See how it works

Reconnection and back-off

For firmware authors: when Hermetry refuses or ends a connection, what the device receives, and how long it should wait before it tries again.

This page is for people who write device firmware or gateway software. It gathers in one place what Hermetry does when it refuses or ends a connection and how a device should come back. Connection settings, logins and the disconnect reasons shown on the Log tab are in MQTT; the general device rules are in Protocol versions and device rules.

The CONNACK codes Hermetry sends

CodeWhen Hermetry sends itSchedule
0The login is acceptedReset after 60 seconds connected (below)
1The CONNECT is for a protocol level Hermetry does not speak, such as MQTT 5Stop. Fix the client.
3The service is starting, stopping or busy; the device’s address is blocked for a short time after failed logins; one MQTT username is held back after repeated wrong passwords; the device is held back after a takeover fight; the platform is under load and holds back your organisationNormal
4Unknown access token, wrong MQTT username or password, or a CONNECT without a usernameLong
5Not authorised: the device is disabled, its certificate is revoked or expired, the organisation is suspended or at its device or connected-device limit, or the profile does not allow the login type, the network or the portLong

Hermetry does not send CONNACK 2 over MQTT 3.1.1. The client ID is never a reason to refuse a login: any client ID is accepted, and the session is recorded under the device’s serial number (or its device ID when it has none). The reason behind a code is never sent to the device; it is written to the device’s Log tab.

Connections closed without a CONNACK

Some connections are closed before a CONNACK can be sent. The device sees only a closed socket and follows the normal schedule.

SituationDefault
The TLS handshake and the CONNECT are not done in time5 seconds from the TCP connection
Too many connections from the same public address are logging in at the same moment64 per address on each broker node
The address is blocked after failed loginsMore than 10 failed logins within 5 minutes block the address for 60 seconds

Sessions Hermetry ends

Hermetry has no reason code on a disconnect: the device sees the connection close. What it should do next:

Why the session endedWhat to do
Nothing arrived for 1.5 times the keepaliveReconnect on the normal schedule. Check that the device sends PINGREQ when idle.
Another connection of the same device took overReconnect on the normal schedule. If this repeats, two devices share one credential.
The credential was rotated or revoked, the device disabled or deleted, the organisation suspended (re-checked about every 60 seconds)Reconnect on the normal schedule. The next CONNACK (4 or 5) moves the device to the long schedule.
The device kept breaking its rate limitReconnect on the normal schedule, then send less. See Device profiles.
A broker node restartsIts connections are closed over about 30 seconds in random order. Reconnect on the normal schedule; a CONNECT that reaches a node that is shutting down gets CONNACK 3.
A message could not be queued (it got no PUBACK)The session is closed about 5 seconds later. Reconnect, then send the unacknowledged messages again.

Keepalive

  • Connect with a keepalive of 60 seconds and send PINGREQ when there is nothing else to send.
  • Hermetry closes a session when nothing arrives for 1.5 times the keepalive: 90 seconds at 60.
  • A keepalive of 0, or one above the broker’s maximum (300 seconds by default), is served at the maximum.
  • A Will message is accepted and ignored. Hermetry decides the connection state itself.

The two back-off schedules

Normal scheduleLong schedule
Used afterA lost network, a closed socket, CONNACK 3The first CONNACK 4 or 5
First wait1 second (a device may start at up to 5 seconds)5 minutes (a device may start at up to 15 minutes)
Next waitsDoubled each timeDoubled each time
Longest wait5 minutes (up to 30 minutes while the device’s active uplink is metered)1 hour
JitterEvery wait multiplied by a random factor between 0.5 and 1.5The same

Rules that go with the schedules:

  • Reset the schedule after a connection that lasted at least 60 seconds. A connection that drops sooner does not reset it, so a device the server keeps closing does not reconnect in a tight loop.
  • The normal schedule also starts again when the device switches its active uplink, for example from Ethernet to mobile data.
  • After a boot or a firmware update, wait a random 0 to 30 seconds before the first attempt. A wave of updated devices then does not reconnect in one spike.
  • Stop on CONNACK 1 until the client is fixed.

The long schedule brings a device back within about 90 minutes of the problem being fixed, for example after the device is enabled again or the plan is raised.

/* Wait before reconnect attempt n (n = 0, 1, 2 ...), in seconds. */
double backoff_s(int n, int long_schedule)
{
    double first = long_schedule ? 300.0 : 1.0;
    double cap   = long_schedule ? 3600.0 : 300.0;
    double wait  = first * pow(2.0, n);
    if (wait > cap)
        wait = cap;
    return wait * (0.5 + (double)rand() / RAND_MAX);   /* factor 0.5 to 1.5 */
}

Tip

Seed the random generator differently on every device, for example from its serial number and a hardware random source. Devices that share a seed pick the same waits and still arrive together.

Repeated wrong passwords

After more than 5 failed logins (CONNACK 4) for one MQTT username from one network within 15 minutes, that username is answered CONNACK 3 for 15 seconds, doubling with each further failure up to 5 minutes. A device that has had a CONNACK 4 and then gets CONNACK 3 is usually still using the wrong password: keep it on the long schedule until a connection succeeds.

Over the device HTTP API

StatusMeaningWhat to do
429 rate_limitedOver the device’s or the organisation’s rate, or over the request rate of the client address (50 a second, bursts of 200, by default)The request was not stored. Wait the Retry-After seconds, then send it again.
429 blockedThe address is blocked after failed loginsWait the Retry-After seconds (the rest of the block). Fix the token.
503The service is busy, not ready or shutting down, or holds back your organisationWait the Retry-After seconds (30), then follow the normal schedule.
401No token, or an unknown oneLong schedule. Check the token.
403Not authorised: device disabled, organisation suspended, outside the profile’s allowed networks, or a profile rule (tls_required, quarantined)Long schedule, or fix the configuration.
400, 413The message itself is wrong or too largeDo not send the same message again.

After the device is back

Follow the connect order in MQTT: subscribe, send hello, ask for the shared attributes, then send buffered readings before live ones. Two points matter after a long outage:

  • Buffered readings must fit the replay rate the device finds in the shared attribute session_limits. Do not send the whole buffer as fast as the link allows.
  • After an outage, many devices ask for their settings at the same time, and Hermetry paces them. Settings sections that are missing from the answer can arrive moments later on v1/attributes/set, in the same form. Apply them whenever they arrive.