Over-the-air (OTA) update best practices for industrial IoT and embedded devices

The foundation of the software stack is firmware, which is used by computer hardware for basic operations and applications. Firmware is programming written to the non-volatile memory of a device’s hardware. A developer can write the firmware into read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory. OEMs use embedded firmware to control the functions of various hardware devices and systems.

Firmware updates of a device's base software (such as the Linux kernel OS, RTOS, root file system, or drivers) are necessary to ensure performance. Firmware over-the-air (FOTA) updates are a more efficient means of deploying the firmware to target devices.

Firmware updates are most commonly associated with highly constrained or "bare metal" devices. These devices usually have only one function, such as environmental sensors measuring temperature or smart locks controlling a door lock remotely. Typically, there is only a small amount of memory storage available on these devices, so the OS must be lightweight and efficient. On a highly constrained device, the device client that enables the polling of an OTA software update must also be lightweight and efficient.

Developers can use image-based updates to create device consistency, lowering the risk of device downtime. In image-based updates, the pre-production environment is identical to the one deployed to the devices in the field. Image-based updates also allow the design to be more accessible for atomic installation, leveraging a failover partition if there is a problem during installation, such as power loss.

Package-based updates are more targeted, only changing a small part of the device software. A critical benefit of package-based updates is requiring less bandwidth. Utilizing less bandwidth lowers network data costs while increasing the speed of the update transfer and installation.

Software over the air (OTA) updates are a catch-all term for different types of OTA updates. There are three common types of over-the-air (OTA) software updates. Operating system updates cover the whole system, including system-level applications and the Linux kernel. Application updates deploy updates for software applications on a device. Container updates deploy new or updated containers, such as Docker, to devices.

The concept of remote updates to devices has been around for decades. In the late 90s and early 00s, OTA updates gained traction with OEMs in the telecommunications industry, with Nokia being a notable pioneer in mobile phones. At the same time, many automotive OEMs also started experimenting with deploying OTA firmware updates to fix software bugs and other issues in vehicles' electronic control units (ECUs). For instance, GM has been doing OTA updates to telematics control units (TCUs) in their vehicles since 1999.

In 2012, OTA updates gained traction and fame in the automotive industry when Tesla applied post-market OTA updates to the Model S. It was clear. Deploying updates over the air is a cost-saver and efficiency improvement compared to having customers make onsite visits to dealerships or service sites for manual updates.

As software becomes increasingly important, the future of OTA updates lies in their continuous innovation and release potential. The prominence of the continuous integration and continuous delivery (CI/CD) software development methodology requires an automated and efficient infrastructure for building, testing, and deploying software changes. OTA integration within a CI/CD process allows OEMs to deploy software quickly, improving the quality and relevancy of their products or services.

OTA updates enable OEMs to manage software configuration and hardware-software variations across complex, heterogeneous device fleets. As product complexity grows, the variations in software and hardware exponentially increase. Autonomous vehicles, industrial robots, AI-enabled products, and more are complex, multi-platform systems with hundreds of sub-components. Robust and granular OTA update mechanisms possess the capability to both update various components while managing the variations, dependencies, and intricacies of complex products.

With digitalization, mechanical and analog machines and devices are now connected and more intelligent. These software-defined devices form the backbone of today's modern world. Connected devices prevail across all industries, from power and energy to building management and infrastructure systems to smart home devices and digital assistants.

Historically, electrical devices' functionality was hardwired. The OEM formed the mechanical algorithm and connectivity using wires that connected relays for logic and potentiometers for setting values. These devices were mechanically elaborate, making them expensive to manufacture. They were also highly inflexible; changing the device was only possible by adding additional mechanical pieces and likely rewiring it. These applications are considered the ancestors of embedded technology.

Embedded systems solved these problems by being inexpensive and more flexible through software. For example, in a washing machine, the OEM gains immense freedom to change the model during a production run or use the same hardware for several models by changing the software alone. The early days of embedded systems focused on replacing their mechanical predecessors, typically in known and well-understood applications.

Then, it became apparent that embedded systems could provide functionality that was impossible to achieve in the classical mechanical way. A good example is the control of injection and ignition in combustion engines. The OEM can only tune the mechanical or electrical approach to a single point of operation. An electronic engine control unit can consider hundreds of conditions, such as temperature, load, and vibration, and react accordingly. Thus, embedded technology spurred a new generation of connected products.

Embedded systems enable and optimize connected products, dramatically expanding their range and application. At one end, embedded systems provide everyday functionality, such as radio controls for a garage door, a charger for an electronic bike, or the control panel of a microwave oven. Then, on the other side of the spectrum, embedded systems exist in the automotive, aerospace, and industrial processing industries. Modern airplanes use fly-by-wire technologies with a control system and a digitally controlled actuator that drives the rudder. Most connected devices rely on embedded systems to provide functionality.

Unlike classic electromechanical systems, where the functionality is in the physical hardware, software plays a crucial role in an embedded system.

The transition from individual embedded systems to products that integrate numerous embedded systems into a large, complex, and heterogeneous system is a natural evolution. Within these complex embedded system applications, a high-powered controller or primary device usually runs a program with up to hundreds of smaller microcontroller units (MCUs). An MCU is a small computer that controls the function of specific hardware. With typically one purpose or action, microcontrollers are low-powered devices with software customized to the specific hardware. MCUs can also utilize OTA updates to update their lightweight architecture.

With the growth in complexity comes concerns around robustness, security, response time requirements for specific functionality, physical space constraints, storage parameters, and power restraints. Plus, there are the foundational cost considerations and optimization underlying any commercial product or application.

The Internet of Things (IoT) is a catch-all term to describe internet-connected computing devices. Generally, IoT applied to commercial applications is organized into the following main categories:

• Industrial IoT (IIoT), where OEMs and end customers connect industrial production machines for data acquisition to generate insights for business and operational improvements.
• Healthcare/Medical IoT (IoMT), where critical healthcare and diagnostics devices are connected for software-defined optimizations and, in some cases, cloud services.
• Smart Cities/Smart Energy IoT, including integrating connected devices to optimize urban infrastructure and energy management, and smart cities focused on optimizing operational costs, increasing security, and providing valuable data for urban development. With smart energy, these segments facilitate the integration of renewable energy, optimize power distribution, reduce waste, and lower emissions.
• Automotive IoT covers the new categories of software-defined vehicles (SDVs) and autonomous vehicles (AVs), as well as the extended services in the connected vehicle ecosystem, such as subscription-based features, payments, and electric vehicle charging.

IoT impacts nearly all industries, with its segmentation extending into transportation, maritime, defense, oil and gas, robotics, and consumer appliances. The common denominator is the role of software and data connectivity in new generations of digital products alongside the need to manage, control, and secure the software, connectivity, and data flow.

In 2023, the enterprise IoT market size reached $269 billion, representing a 15% year-over-year growth. The overall outlook remains positive, with the number of connected IoT devices increasing by approximately 13% compared to 2023. The IoT market is projected to grow at a compound annual growth rate (CAGR) of 10.17% from 2025 to 2029, reaching a market volume of $1.56 trillion by 2029. 

Security is a primary concern for businesses. In fact, the majority of European IoT adopters cited security improvements as a key motivation for implementing IoT solutions. Reducing operational costs and improving efficiency are also significant drivers of IoT adoption. Additionally, adding IoT connectivity to products presents substantial growth opportunities for OEMs. Software is increasingly regarded as a strategic asset, enabling products to be developed and delivered more quickly, cost-effectively, and with ongoing enhancements once deployed in the field.

Planning early is essential. The OTA update process and supporting infrastructure should be considered at the start of the product R&D process. If an OTA mechanism is considered upfront in product design planning, it can be tested during the development phase. OTA updates can also aid in accelerating product development, facilitating faster iteration and delivery cycles.

Conversely, adding an OTA update mechanism later in the product development lifecycle is often complicated, might require rework, and risks go-to-market timelines. In the worst-case scenario, without an OTA update mechanism, an OEM might have to physically recall the first batches of production devices to resolve issues at a high cost. Or, if an OTA update is required two weeks before the product goes to market, lacking the necessary infrastructure and process could block the entire manufacturing stage. While changing software is relatively easy, rearranging manufacturing and device provisioning processes in the factory is not. 

Once the need for an OTA update process for a new connected product is established, the parameters that the OTA update infrastructure must support need to be identified. There are four key considerations for OTA update infrastructure: robustness, cybersecurity, operations, and ecosystem.

An estimated 8.5% of the devices in a large fleet can fail within three years when supported by a poorly designed OTA update solution (as researched by the Mender engineering team). The operational and resource impacts of sending engineers into the field to replace nearly 10% of a customer's device fleet every three years are significant.

The OTA update infrastructure must be robust: A software or system update should never "brick" or render a device inoperable. 

Automatic rollback is a critical feature for the OTA update infrastructure to ensure the device remains operational. An A/B partition design for the OS and version rollback ensures recovery even if the deployment is incomplete or corrupted during installation for any reason. Full image updates also ensure that updated devices remain operational.

Robustness in the OTA update infrastructure facilitates the regular updates of continuous improvements without the risk of breaking (or bricking) customer products in the process.

In addition to the robustness of the device update, the OTA update infrastructure should ensure robustness in the deployment of updates across a fleet of products. For example, in a 100,000-device fleet, the OTA update mechanism must spread out the timing of updates so that not all devices poll the management server for updates simultaneously, potentially causing the server to go down. A floating cycle, where devices ask for an update once per hour, avoids simultaneously polling for an update, overloading the server.

Deployment granularity ensures updates do not adversely affect a fleet. For example, scheduled deployments can avoid device downtime by performing the OTA update during a planned maintenance window. Automatic retry support and first boot updates (updating the device on first activation) also ensure the stability and operability of a device fleet.

A well-managed and maintained OTA update infrastructure is crucial, encompassing the device update and the server backend infrastructure that controls it. For large-scale fleets, using an integrated management server and device client significantly reduces costs and management and maintenance overhead. An OTA updates solution that integrates a management server for control and a device client in a single end-to-end architecture is robust and efficient.

The OTA update process relies on secure communication. As such, security within the infrastructure, the process around the OTA update, and management is critical to ensuring cybersecurity, safety, and regulatory compliance.

The OTA update infrastructure must adhere to secure-by-design principles. Encryption and cryptography are baseline requirements throughout the OTA update mechanism. To prevent “man in the middle” cyberattacks, the device running the OTA client should authenticate and secure the connection using industry-standard protocols like secure TLS to get the update, update the inventory data, and deliver status information. The client must verify the server’s identity by using a Certificate Authority from the device OS or by using pre-distributed keys, like self-signed certificates. There should be no open ports in the device, as this significantly increases the attack surface.

The processes around leveraging the OTA update infrastructure must also reinforce security and safety best practices. For example, multi-factor user authentication, role-based access control (RBAC), and pre-authorization of devices are standard capabilities to ensure security. It is essential that only authorized users be able to make authorized updates to authorized devices.

Similar to robustness, an end-to-end OTA update solution, integrating the management server and device client, best ensures security by design. An end-to-end solution can maintain the identification of the communication endpoints, ensure message authentication, and integrate with hardware security module (HSM) or trusted platform module (TPM) solutions so that the cryptographic operations can be passed to the hardware as a trusted source that provides an additional security layer. The client can use private keys stored in the HSM or TPM, eliminating the storage of private keys on the device and making it more difficult for attackers to impersonate devices. The device client should also ensure that the artifact (the transport mechanism containing the update) is from a trusted source, such as using the public part of the key pair used to sign the artifact.

An OTA update infrastructure is also pivotal in managing and mitigating software vulnerabilities, enabling remote patch management across thousands or even millions of devices. Leveraging a secure end-to-end infrastructure with robustness built in, OEMs can seamlessly deploy security patches and bug fixes across device fleets, with control and granularity. OTA updates significantly reduce the logistical complexities and expenses tied to manual or physical updates, which are neither feasible nor scalable in widespread IoT ecosystems. By automating patch deployment, OEMs can ensure that their entire fleet remains secure without the costly burden of on-site interventions.

Reliable deployments and real-time support to predictive maintenance and optimized infrastructure, the OTA update infrastructure is the cornerstone for managing fleets across any network, heterogeneous environment, scale, or complexity.

A well-defined and operational CI/CD software development process enables an OEM to move faster and be more agile in the market. Integrating an OTA update infrastructure allows OEMs to deliver critical features, fix bugs quickly, and release continuous improvements seamlessly.

The OTA update infrastructure should include management capabilities that enable fleet managers to control and deploy software updates across device fleets, at scale, and across diverse geographical locations. Automation should also be leveraged to reduce fleet management administrative burden and increase deployment accuracy. Capabilities such as dynamic grouping allow devices with certain key attributes to be automatically grouped and receive the right OTA updates.

A mTLS proxy container is also a very important capability for OEMs who, during mass production, want to sign client certificates during the manufacturing process, so that devices automatically get accepted into the OTA update infrastructure server when activated by the end customer, sometimes several months after manufacturing.

An OTA update mechanism must be aware of insufficient connectivity, failing connectivity, and quality of service issues for devices. For example, if no connectivity is available, the OTA update infrastructure should only then poll the server for the OTA update once it detects a qualified connection. The OTA update mechanism should also be able to resume an update in the case of a lost connection.

The OTA update infrastructure should also support incrementally updating the software image by allowing updates to just the delta, or the difference between the previous image and the new version. Incremental or delta updates also reduce the network costs of an update, requiring less data to transfer using cellular and satellite connectivity.  

Devices, software versions, software-hardware compatibility, scheduled, canary, phased, and full fleet deployments, and low-to-no connectivity environments, ensure secure fleet updates with flexible deployment controls.

The complexity of OTA updates increases with the advancement of IoT technology. Today, a single end product could contain hundreds of other ‘devices,’ each with their own update requirements, parameters, and dependencies. Multi-product updates are an essential consideration for an OTA update infrastructure.

Software configuration management

Multi-device products are in the automotive, aviation, agriculture, maritime, and transportation industries. A passenger vehicle would be an example of a "system" with multiple embedded computers, such as MCUs and ECUs, and a requirement to update the software on these different computers in an orchestrated and coordinated fashion.

The first critical use case in whole system updates is to support a name and a type for a system. In a passenger vehicle, the type could be an annual revision; it could contain a defined set of devices with clear identities, including the in-vehicle infotainment, the brake controller, and so on. Mapping several software artifacts to one system type for compatibility is essential in these use cases. Each system, such as a passenger vehicle, will have an identity, type, and software version. Ultimately, this translates device management into a higher-level system management concept.

The second critical use case is updating devices as a whole system, where it is essential to ensure that everything or nothing is updated. It may not be possible or desirable to have one device running an older software version and another running a newer one. Regulatory requirements may also require testing in a bundle, and compatibility checks are mandatory.

OTA software update mechanisms should have specific capabilities to enable whole-system updates. Synchronized updates are often required to ensure that version dependencies are adequately handled. For example, one component may require a different component to be on a specific software version before it can be updated. Or systems will have software versions that are dependent on the hardware components on which they are executed. Synchronized updates enable an OEM to manage the different variations and update each component within the whole system accurately.

Gateways also support the realization of updating multi-device products, facilitating updates through a central point. Gateways with one-time artifact download capabilities can also update devices or systems in segregated networks.

Remote device monitoring, configuration, and troubleshooting are essential elements in managing products once in the field. The OTA update infrastructure is tightly coupled with these functions and their effective execution. The OTA update infrastructure should ensure device reliability and performance with real-time edge monitoring and processing designed for connected devices. OEMs can identify and fix issues before customers even know they exist. OEMs can also leverage the OTA update infrastructure to easily pre-configure or change device configurations over-the-air while avoiding device-server authentication and management in multiple places. Running diagnostics, analyzing system log files, transferring files, accessing local services, and restarting the device should be capabilities of the OTA update infrastructure, allowing OEMs to resolve device issues live on the device, quickly diagnosing and applying fixes over the air.

The OTA update infrastructure must be flexible enough to handle different types of OTA updates, environments, and complexities. Alongside the OTA update, device management such as secure remote device access, troubleshooting, monitoring, logging, and configuration must also be considered.

By 2030, McKinsey estimates that the value of IoT products and services could contribute $5.5 trillion to $12.6 trillion in value globally. Factory IoT applications account for the most significant potential economic value at 26% in 2030. Human health applications are the second contributor, representing 10% to 14% of the estimated IoT economic value. Across industries, business-to-business (B2B) applications encompass the majority of IoT value creation, with about 65% of the estimated IoT value potential by 2030. Operation optimizations will deliver the most significant economic value from investments in IoT technology. Reduction in costs by remotely servicing equipment, anticipating equipment failure, predictive maintenance, and preventing failure through condition-based maintenance are considerable value generators.

When products are connected, a reliable means of updating the software, fixing bugs, and patching exploitable vulnerabilities is mandatory. When a fleet scales to hundreds and thousands of devices, efficient management, consistency, security, and quality assurance are paramount.

A secure and robust OTA updates mechanism provides considerable efficiency and value while enabling innovation and faster time to market for competitive advantage.