Fixing a remote station overnight: why OTA is non-negotiable

One evening, one of our railway stations, several hundred kilometres from Ulaanbaatar, began reporting values that made no sense. The telemetry pointed not at a failing sensor but at a firmware bug triggered by a rare edge case. We prepared a fix that night, pushed it over the air, and watched from the dashboard as the station rebooted into the new version. By morning it was reporting normally. Nobody left the office.

Consider the alternative. A site visit means a vehicle, fuel, two days of an engineer's time, and winter-road risk, and it might not even be possible that week. With 9+ stations spread across 1,000 km, “drive out and reflash” does not reach them all. Remote deployment turns every small software decision into a logistics decision, and OTA is what turns it back.

OTA carries its own risk: a failed update hundreds of kilometres away is worse than the bug it was meant to fix. So we hold to a few rules. Update images are verified before they are applied. Firmware uses a dual-bank (A/B) layout, so if the new version fails to boot or fails to check in with the server, the device rolls back to the previous one on its own. Rollouts are staged: one station first, then the fleet. And downloads must survive slow, flaky cellular links, resuming where they stopped instead of starting over.

The other half of the answer is store-and-forward. At remote sites, network outages are not incidents — they are weather. Our devices buffer data locally for up to 30 days and replay it when connectivity returns, so the dashboard shows a gap closing rather than history lost. For alarm systems there is a further subtlety: the platform must distinguish “the site is offline” from “the site has nothing to report”. Silence is not the same as safety.

The two capabilities only make sense together. OTA lets us change a device's behavior remotely; store-and-forward guarantees that nothing is lost while we do. During an update window the station keeps logging; after the reboot, the buffered measurements are back-filled as if nothing had happened.

This shapes the whole architecture. We treat connectivity as unreliable by default and the device as the source of truth — the cloud is a mirror, not the master. That one decision cascades: timestamps are assigned at the edge, local storage is sized for weeks rather than minutes, and the protocol layer reconnects and resynchronizes without human help.

OTA is not a substitute for testing. It reduces the cost of the bugs that survive testing — and some always do. Every fleet we ship now leaves with OTA and local buffering enabled from day one, because we have learned that retrofitting either one after deployment is far more painful than building it in.