Protecting emergency fire intel with a robust OTA update infrastructure

Objectives

	Reduce resources spent on device management infrastructure
	Shorten device diagnosis and remediation processes
	Reduce security risk throughout operations and across the Echo fleet
	Minimize the risk of significant fleet downtime
	Enable more team members to manage the device fleet

Watch Duty’s Echo radios deliver remote listening, tone detection, and early warning to improve radio information in remote areas.

The Challenge

Robust and secure management of devices in remote areas

Watch Duty’s Echo radios are deployed far and wide to alleviate the radio dead zones that hinder reporting, first responders, agencies, and volunteer firefighters. With some existing in very rural areas, Echo radios are designed and deployed to remain evergreen.

However, managing, improving, and maintaining Echo radios – remotely, at scale, and in the field – created significant new challenges. Initially, the Watch Duty team built its own remote management infrastructure solution based on remote terminal access (SSH), custom scripts (download updates), and a VPN infrastructure (connectivity).

At dozens of devices, its homegrown remote management solution suffered from scalability and robustness issues. Dealing with individual device issues consumed significant time and effort. Only one employee (its creator) knew how to use the solution, resulting in a single point of failure (SPOF). The employee needed to be available 24/7 – creating internal risks, delays, and pressures – and he was also the only person who could manage the Echo devices. In scenarios where seconds matter, these scalability and SOPF factors could incur massive repercussions, such as radio frequencies changes during a wildfire requiring a device update.

The homegrown management solution also had inherent security risks. The sole employee using the solution, the only one with access, meant his credentials were present on all devices. If his credentials were lost, no devices could be managed. Or worse, if this single credential was compromised, the attacker could take control of all Echo radios. Lastly, the VPN infrastructure created another major attack vector. Any of the Echo radios could see the network traffic of the other radios; a single compromised Echo could impact the entire fleet.

While the Echo radios required a remote management solution, Watch Duty’s homegrown solution was fragile, high risk, and already consumed significant time and effort. The Watch Duty team knew it was just a matter of time until something disruptive would happen as its Echo fleet continued to scale. Watch Duty required a robust, secure, and scalable solution to avoid unplanned fleet downtime and manage future security issues.

OTA Software Update Solution Technical Requirements

	Integration with GitHub Actions for streamlined CI/CD
	Image-based updates for consistent and repeatable software builds
	Scalable and repeatable device provisioning
	Robustness, even in low and intermittent network connectivity

Leveraging a scalable and secure OTA update infrastructure

The Watch Duty team knew its Echo fleet required a new remote management solution. A team member experiencing similar challenges in his past company with its homegrown solution recommended Mender, stating, “Every good company in the world uses Mender.” With a solid reputation and track record of managing devices for some of the largest enterprises in the world, Mender offered a battle-tested, secure, robust solution for the Echo product.

With Mender, Watch Duty now has a standardized DevOps release process. Every code change triggers a GitHub Action to build a software package (Mender Artifact) that is automatically uploaded to hosted Mender for deployment. With a few clicks, this package can be deployed to test devices, tested, and rolled out to all production Echo radios.

Watch Duty replaced its entire homegrown infrastructure with Mender, immediately streamlining operations. Using Mender, Watch Duty seamlessly upgrades and maintains Echo radios without physically accessing them, including initiating a power cycle if the system is unresponsive. The Mender Troubleshoot Add-on enables Watch Duty to diagnose and troubleshoot devices in the field; its remote terminal and file transfer features allow Watch Duty to easily download log files or audio snippets from the radios and quickly diagnose any issues.

End-to-end security and encryption protect devices and the entire fleet from security threats, whether unintentional or malicious. The easy interface and role-based access control allow the whole Watch Duty team to utilize the solution in a secure manner. The Mender Configure Add-on securely and uniquely provisions credentials, such as for WiFi, without exposure to other devices, lowering the security attack surface.

“Our deployment process was initially limited, relying on a VPN mesh and custom scripts. Switching to Mender has streamlined our operations, enabling standardized setups with OS images, one-click GitHub PR deployments, and enhanced security through unique user accounts.”

Nick Russell, VP of Operations

The Mender Difference

	Deploy to test and production environments with a few clicks		Standardized and documented software deployment process
	Rapid diagnosis and remediation of device problems		Fast and repeatable new device onboarding

Technology Ecosystem

	Raspberry Pi 4- Headless Processor: Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.8GHz
	2GB LPDDR4-3200 SDRAM w/ 8GB SanDisk Industrial SD-card
	Diamond D130NJ Super Discone Antenna
	RX Frequency Range 25 - 1300 MHz
	Active vented IP65 NEMA 3X enclosure designed to withstand harsh weather conditions

“This shift has markedly improved our deployment’s robustness, speed, and scalability without the complexities of our previous system."

Nick Russell, VP of Operations

Benefits

Realizing robustness and speed at scale

With a streamlined and automated software release process using Mender, Watch Duty now enjoys short and low-effort software development iterations – submit code, deploy it, test it, and repeat – all with just a few clicks in the Mender UI, and completing the cycle in less than 5 minutes. Compared to the homegrown setup, this is a major time saver, enabling Watch Duty to move faster and focus on improving its user experience.

As an additional benefit, the whole team can make key changes to the Echo fleet, such as changing the radio frequency of devices with very little effort or know-how. The team can now collaborate due to the lower skill barrier, reducing the risk and stress on single employees.

Scalability is now built-in. New Echo devices can be easily built using a standard process. There is no longer a scalability risk from the infrastructure itself.

With Mender, Watch Duty now spends less time on infrastructure and more time on its products. Watch Duty now has the peace of mind of using a robust and proven OTA update solution.

Key Benefits

	Reduced time and spend with a fully automated CI/CD process
	Mitigated the risk of compromise with new security layers on device and fleet levels
	Decreased organizational risk and stress through team-wide fleet visibility and control
	Ensure scalability with a future-proof solution capable of meeting requirements for the next decade and beyond