Mender OTA updates and an automated CI/CD pipeline at Gunnebo Safe Storage

Mender is about to announce an updated integration with Azure IoT Hub. But our OTA updates solution has also been integrated with Microsoft's Azure DevOps toolsets by some software and embedded development teams. Progressive DevOps teams such as the team at Gunnebo Safe Storage are using end to end integration between development pipelines and OTA image updates so that they can use automation to increase development efficiency in the release process while maintaining quality assurance.

Gunnebo Safe Storage is using a combination of Microsoft Azure DevOps tools and Mender to enable an automated process for testing and releasing. Software releases for firmware are delivered OTA to its Everyday Safe product in the field. Bjorn Tore Nostdahl, DevOps Innovation Manager, and 2 members of his development team, Anna Biriukova and Vladyslav Oleynik describe the project in this article.

Low level implementation in C++

This is a low level implementation done in C++ on devices that are highly constrained. The target devices for the firmware updates run the ESP32. The ESP32 is a low-power system on a chip microcontroller with integrated Wi-Fi and dual-mode Bluetooth. It has 4 Mbytes of integrated FLASH memory and employs a Tensiica Xtensa LX7 dual-core microprocessor; and includes built-in antenna switches, RF balun, power amplifier, low-noise receive amplifier, filters, and power-management modules.

The artifact that needs to be transported is relatively small too at 1.5Mbytes in size so there is no requirement at this stage for a delta updating capability.

Pipelines in Azure DevOps and Mender

There are two main pipelines in operation:

1. A pipeline for unit testing
2. A pipeline for checking the artifacts for quality and security

The software deployment is done by embedding the bootloader and other components for factory provisioning into the artifact; and then having a separate artifact for the OTA update firmware for the release.

Mender is used in this workflow for 2 main purposes:

1. To perform the firmware image OTA update
2. To enable device provisioning in the factory

The Git workflow and the various pipelines the team have set up manage the release of the software to the target devices. This Git workflow is used for building the framework which features branches and pull requests to develop, and the develop branch is tested and prepared for early release candidates.

The Feature branches contain new features to be implemented in the product. Merge to develop tríggers the CI / CD pipeline and deploys the firmware to Mender.

This is all done in Azure DevOps with the coding done in VScode. From the VScode control console, the code is pushed to develop and this triggers the pipeline to compile the code. It will build the release nodes automatically and push the Mender artifact to the Mender server.

One pipeline builds the firmware, but it does not verify or analyze any code. In this pipeline the firmware is built after fetching the configurations, and at the end of this pipeline, binaries are produced and it is then validated if the code binaries and the publishable artifacts are in the block storage.

There is a separate pipeline that fetches the artifacts (binaries that were built before) and then the artifacts are copied to this separate machine. This is a Raspberry Pi to run the unit test. The binaries are uploaded to the Raspberry Pi which connects to the ESP32 and runs the unit test inside the ESP32. The unit test is run on the physical device in the Gunnebo laboratory. The development team did it this way as they could not find a suitable emulator for the ESP32.

The process is fully automated and a certain pipeline verifies the code by Sonarcube. We also have to assess if we have developers who have used code snippets that are copyrighted or under a license that we cannot accept, we must catch this in the build pipeline. It will also look for vulnerabilities in old code modules or code modules that have vulnerabilities, it will be caught in the white sources in the Sonarcube analysis.

When the binaries are uploaded to Azure blob storage where they are stored, it then releases the binaries to Mender. Every time the development team pushes to develop for a release candidate or master, they also push to Mender. The Mender APIs are used to upload the binaries to Mender and are encapsulating the firmware within Mender artifacts.

The development team has several builds for the provisioning purposes. There is the factory provisioning process and GDPT provisioning a special artifact for that. The code binaries for provisioning have to be gathered together and these include the bootloader, partitioning table and the binary itself, and the OTA binary. The artifact is then flashed from the factory.

The second release is a common update: Inside this release there is only one binary for updating purposes and this is for a firmware update to the Everyday Safe. A release is done whenever a change is committed.

For the Master branch an update is performed every 2 months For the Development branch an update is performed twice per day

Benefits of Azure DevOps and Mender workflow

The benefits of using Mender for automated OTA updating is that the developers do not have to copy paste files and share them on a file server. The QA test group can also update the devices immediately. It brings efficiency and removes bottlenecks in sharing files and testing. It also reduces the dependency on a single person, here you commit your code and someone has to approve it, but else the developers push it to the developer branch and the tester will get the new release automatically.

A summary case study of Gunnebo’s strategy for using Mender for robust and secure OTA software updates is available in this case study .