Mender pays close attention to Quality Assurance

QA leader Lluis Campos explains how the QA process is a social process embedded across the whole Mender engineering team.

Quality assurance is a process that like security runs to the very heart of Mender’s development culture.

Homegrown OTA software updates solutions can take at least 6 months if not 12 months to develop. They absorb development resources and still the software manager needs to be developed and maintained afterwards. See where Bjorn Nostdahl from Gunnebo talks about this in his recent blog post.

Another hidden cost, which enterprises should consider, is the requirement to apply the highest standards of quality assurance to the OTA software update development process.

QA is something the Mender engineering team invests in heavily.

It is not left to one dedicated QA engineer. Everyone in the Mender engineering team is developing QA in an iterative process. They design, plan, develop and maintain, and go through the same cycles for QA as they would with normal software development.

The Mender QA process follows an agile methodology when it comes to planning and preparing the tasks. The testing process is meticulous where the work is done at different levels in adherence with the V Model. The V-model is a type of Software Development Lifecycle model where process executes in a sequential manner in V-shape. It is also known as the Verification and Validation model. It is based on the association of a testing phase for each corresponding development stage. Development of each step directly associated with the testing phase.

The Mender team also places a huge focus on the code coverage in all server components, client code, and related tools like mender-artifact. Getting high code coverage is particularly challenging for the client code, as Mender is tightly integrated with the OS and needs to be tested together with different bootloaders and configurations that are difficult to simulate at the unit test level. The team tests how the client behaves with different “configurations”, combining processor architecture (x86, ARM), bootloader integration (u-boot, GRUB, both...), and a file-system (ext4, ubi).

The team uses Continuous Integration to check every single change that is made to the master code. This facilitates an “at any time” release schedule as the team can be confident that each change has passed all tests and quality checks. Typically for every code change, the team runs 368 Mender core acceptance tests, over 100 Mender client acceptance tests for 6 different hardware configurations and 209 system integration tests. These figures increase every time new features are introduced or bugs are fixed in Mender.

Furthermore, all our test code is open source, so that it can be inspected by anyone. The main test suites are client acceptance tests, backend integration tests, and system integration tests.

The following figure shows our main integration pipeline running in GitLab CI, where all the software stack is built and tested.

It’s a lot of hidden but mission critical work in the pursuit of excellence. Any organisation considering developing their own homegrown OTA software update manager, would do well to consider this.