Top Trend #1: Opportunity Costs of Buying versus Building OTA Software Infrastructure

A 5 point framework for assessing whether to buy or build OTA software infrastructure.

The Mender team recently visited the TechAD automotive conference in Detroit. Albeit it was automotive industry-focussed, we learned about the motivations and drivers for choosing whether to build or buy the components of their software infrastructure. As in most industry segments, liability and risk management are the top business considerations for OEMs to build in-house technology teams. Many opt to retain control of their software, lowering their regulatory liability and managing their cyber risks. OS applications, commoditized products and infrastructure processes are most likely to be built in house.

On the flip side, the team also learned that organizations favored buying or partnering with third-party suppliers and vendors for primarily infrastructure (not core competence) and open-source components. Opportunity cost is the key driver and generally, software infrastructure is a top item to outsource. Companies outsource software building including OTA infrastructure. In determining whether to buy or build, organizations typically ask themselves three critical questions:

• Is it a core competence of the organization?
• Will it be commoditized?
• Does the company have in-house skills to develop the component?

If this logic and line of questioning is applied to OTA software as the lynchpin for the organization’s embedded systems, then selecting a best of breed professional OTA updates manager is the best strategic choice: Certainly to reduce the risk of cost and schedule overrun as Gallup famously reported this can be as high as 200% and 70% respectively for one in six IT projects.

This is what is at stake when choosing whether to build or buy in an OTA updates management solution.

• Faster time to market and high ROI to minimize financial risk
• Security and robustness to minimize operational risk

Quite simply, experienced experts will tell you that a managed solution for OTA updates delivers value in ways in which homegrown OTA builds won’t.

Start by asking yourself this simple question:

Would you develop your own email server from scratch (read Google Gmail)?

The answer for the majority of enterprise product managers would be a resounding “No”. Rather, you would leave it to Google as the market leading experts to do this.

So why do it yourself then when it comes to OTA software updating?

The Benefits of Partnering for OTA Updates

In this article we outline a five point framework for weighing up investing in a purpose built OTA updates manager for your new connected product (s) versus taking the DIY route internally within your organization.

Complexity, Cost, Control, Connectivity and Maintenance are 5 critical factors to consider in the assessment framework for buy versus build.

1. Complexity

What it takes to develop a best of breed OTA software updates manager can be overlooked when an engineer promises that they can start to hack some shell scripts or python scripts to do the job. Developing a best of breed OTA updates manager that is reusable across a portfolio of digital products is hard.

Matthias Luescher is principal firmware engineer with EDI . EDI is a specialist embedded systems consulting company out of Luzerne, Switzerland. Matthias has worked on many large device management projects at elevator provider Schindler. Matthias explains the problem with homegrown OTA updates managers based on his experience: “Many engineers tend to underestimate the real cost behind OTA. With a large fleet of connected devices with over 200,000 connected devices, it can cost “millions” to do homegrown OTA updating properly, and even after this investment, it may not be able to run it as a platform to be repeatable for new projects and products.

Matthias also explains that it is really difficult to make the homegrown OTA updates manager robust. For example, he says “A device could be connected via Ethernet and then could try to connect via wireless, at which point it could need credentials and then an A/B system update. Without the proper configuration managed in sequence, the device will never be able to get online and operate reliably and consistently.” The planning of such robust automated routines requires a sustained period of investment in development and simply trial and error over an extended period of time to find the optimal solution. Mender, for example, has worked on projects with over 10,000 companies over 7 years, where all manner of complex problems in the updating process and in device behavior have been encountered and overcome.

Matthias estimates that it will take at least 12 months to stabilize a new homegrown build. Now consider that it has taken the Mender product engineering team seven years to build a solution of the quality of Mender enterprise. When a product manager in an enterprise considers that they can have all this utility (from 7 years of hard work from ca. 40 high quality developers) instantly then the argument that “buying a managed solution is too expensive and aren’t our in-house engineers capable of doing it?” seems to ring hollow. The in-house engineer (s) may take on the challenge but it is highly unlikely they will be able to deliver a best of breed OTA updates manager within 12 months.

The list of requirements needed to have a secure and robust OTA solution is extensive. We cover the full requirements of a best of breed OTA update mechanism in this white paper. A complete end-to-end software updater including both the management server for deployment and the device client can range from 50,000-100,000 lines of code.

2. Cost

While the cost estimates can be broad, embedded systems software typically range from $15 to $40 per line of code. On the low end, an enterprise product manager can expect the cost of a complete OTA updates manager built from scratch to be a minimum of USD $750,000 with no sacrifices made to security and robustness.

Our product engineering and technical product marketing team ran a cost benefit analysis on the ROI from using a hosted service from Mender compared to building a homegrown OTA updates manager. Over 5 years, they calculated that the ROI from using the hosted version of Mender could be as high as 985% and the average payback period could be as low as 5 months. This assumed that an organization would scale from 50 devices in year 1 to 10,000 devices by Year 5.

3. Control

With a professional OTA updates manager such as Mender enterprise, you get immediate access to a range of capabilities to help you manage your device fleet as you scale. The cost of researching and developing each of these capabilities in-house would be high without the certainty of quality outcomes in each case as they could be outside of your organization’s core competencies.

Examples of these fleet control capabilities include:

• Dynamic grouping so that filters based on attributes matching one or more devices can be used rather than device IDs.
• Fleet automation and synchronization: Dynamic deployments and Synchronized updates¹.
• Phased rollouts to minimize deployment risk and ensure update success.

4. Connectivity

A homegrown OTA build also needs to be stable and to help the DevOps team by maintaining connectivity and being easy to update. See how the Gunnebo DevOps team used Mender to automate firmware from its CI CD pipeline to target devices in the field in this article. There is a long running cost involved to stabilize it, and homegrown OTA will break multiple times over the journey. You will also need to build a highly secure means to remote access and troubleshoot the device; and log all terminal sessions on the device. As the fleet scales, managing data transfer costs for the software image deployment over cellular and satellite connections will also become a consideration. The best practice is to use Auto Delta updates , and to build this kind of capability to perform incremental updates on the delta is non-trivial.

5. Maintenance

What will happen to your product's robustness if the talented developer who has been maintaining your homegrown OTA updates manager in their own unique way decides to leave your organization? Using a hosted service managed by experts gives you peace of mind that your projects can continue uninterrupted even when valued team members move on from your organization. Having a Service Level Agreement for critical support could also be very important for risk mitigation and disaster recovery. With a professional service such as Mender, you would also be able to influence the product roadmap with direct access to the Mender product management team. Your ideas for new features could be assessed and sanity checked by industry experts.

One of the biggest benefits of selecting a professional OTA updates manager for your project could be the professional development of your engineering team. Rather than having a talented developer in your team build and maintain a homegrown OTA update manager in the background, you could redeploy their skills to core product development that would give your connected product a competitive edge in the marketplace.

Read more

We have published a white paper which reveals the hidden costs of a homegrown OTA updates manager. It will help you to make a more informed decision based on all the available evidence.

Notes

1. The synchronized updates feature has since been deprecated – read more here.