English is currently available.
Industrial IoT DISCUSSION
Our edge gateway publishes about 200 values per second to a cloud broker over a cellular link. The link drops several times a week, sometimes for two or three hours, and the historian in the cloud then has holes. Setting MQTT QoS 1 did not help much: short drops are covered, but long outages still lose data, and after reconnecting we sometimes see duplicated rows.
How is store-and-forward normally implemented so that outages of a few hours are covered cleanly?
QoS 1 only protects a message between one client and the broker while the session exists. The client library keeps unacknowledged messages in a queue, usually in RAM and with a size limit. During a three-hour outage that queue overflows or is lost on a restart, which is the gap you see.
Reliable store-and-forward has to be a layer of its own in the gateway: write every sample to local persistent storage first, and have a separate sender that reads from that store and deletes or marks a record only after the broker has acknowledged it.
Size the buffer from the longest outage you want to survive. At 200 values per second, three hours is 200 × 10,800 = 2.16 million records, and 24 hours is 17.28 million. If a stored record takes about 50 bytes, a day is roughly 860 MB, so decide whether you really need every value or whether report-by-exception with a deadband would cut the rate first.
Use a disk-backed queue or a small embedded database with a ring policy that discards the oldest data when full, and think about flash wear: write in blocks every second or so instead of committing each sample individually.
Duplicates are inherent in at-least-once delivery: if the acknowledgement is lost when the link drops, the sender must send again. The clean fix is to make the write idempotent rather than to chase exactly-once delivery. Give every sample a key of tag identifier plus source timestamp, and have the cloud side insert with that key as unique (an upsert). A replayed record then overwrites itself and no duplicate row appears.
This only works if the timestamp is taken at the gateway when the value is sampled. If the cloud assigns the time on arrival, replayed data lands at the wrong time and duplicates cannot be recognised.
Time needs attention on the gateway too. Keep its clock synchronised by NTP and store timestamps in UTC, so that a buffer replayed across a daylight-saving change does not shift by an hour. If the gateway can boot without network access, a battery-backed RTC avoids samples stamped with a default date until the first sync.
When the link returns, replay in order but throttle it. Sending three hours of backlog as fast as possible on a cellular link can starve the live data and trip rate limits on the broker. A common scheme sends live values immediately on one topic and drains history in batches on a separate one.
Test it the blunt way before trusting it: pull the antenna or block the port for a few hours, then compare record counts per tag between the gateway store and the cloud database for that window. Repeat with a power cycle of the gateway in the middle of the outage, because that is the case that exposes queues held only in memory.
A per-message sequence number from the gateway makes this check easy, since any missing number on the cloud side is a lost record.
Back to Industrial IoT community
