This blog post will cover how acceptance and integration test coverage is collected from the Mender client tests.
Starting out, we thought this was going to be a straight forward problem, and there are plenty of blog posts on the web, covering the standard approach. For go test
can collect coverage from a do_main()
function, just as well as it collects coverage from any other unit tested function. For more details have a look at:
However, this did not do the trick for us, as our binary does some magic with the std file-descriptors at some point, and this did not go down well the go test
framework, which also apparently does some massaging of I/O.
Thus, instead of digging our way to the bottom of the go test
framework, to figure out what was going on (we gave up after duplicating the three std file-descriptors before calling main did not work).
At this point I was a bit bummed, and was cursing the world for handing me this ticket. It wasn't even estimated to take more than a day or two, and the boss is like
Still, in apathy, I started poking around the Go tool-chain (which is amazing, btw). Here I stumbled upon the go tool cover
, which you might be familiar with from looking at the coverage output from a Go test. However, it can also instrument Go source code for coverage collection. In fact, this is exactly what go test
does. This actually means that go test
is not needed at all!
This inspired the creation of the gobinarycoverage
tool, which this post will dissect in more detail. The gobinarycoverage
tool does basically the exact same thing as go test -cover
does, except that it does not add any external framework around you binary. This means that the program is guaranteed to function just like it would, if you were not collecting coverage!
Gobinarycoverage - A tool for collecting coverage from Go programs
The Gobinarycoverage tool consists of three main parts:
1) It instruments the source code of all the files which will be part of a main build through wrapping go tool cover
, so that coverage can be collected.
2) Then it generates a separate main file, which holds references to all the necessary structures in #1
3) It merges the main.go
file with the main file in #2 through using the ast
package provided by the Go standard library (thank you!), and parsing the two files into each their separate AST's, and then merges them accordingly.
Let's have a closer look at each step in more detail:
1 - Instrument Go source code with coverage capturing capabilities
The tool is taking advantage of existing go tools' functionality. Notably, it uses go list
, in order to figure out which packages main.go
imports. From this information, it runs go tool cover
on the returned packages. This will change the source code in the given packages to add in a counter at each block, and a GoCover
struct to each file, which is responsible for collecting the information.
An instrumented function will afterwards look like:
func (m *MenderError) Error() string {GoCover5.Count[2] = 1;
var err error
if m.fatal {GoCover5.Count[4] = 1;
err = errors.Wrapf(m.cause, "fatal error")
} else{ GoCover5.Count[5] = 1;{
err = errors.Wrapf(m.cause, "transient error")
}}
GoCover5.Count[3] = 1;return err.Error()
}
And the struct present in every file, collecting this information looks like:
var GoCover5 = struct {
Count [2]uint32
Pos [3 * 2]uint32
NumStmt [2]uint16
} {
Pos: [3 * 8]uint32{
35, 37, 0x20025, // [0]
39, 41, 0x20026, // [1]
},
NumStmt: [8]uint16{
1, // 0
1, // 1
},
}
2 - Generate a main.go file which can reach the generated coverage structs
This struct, then has to be imported into the new main.go
file. This means that the tools needs to generate source code on the fly, which imports these structs from every file it covers. Imports in the new main.go
file will then look something like after code generation has finished (this is simply done through a text template):
package main
import (
"fmt"
"io/ioutil"
"testing"
_cover0 "github.com/mendersoftware/mender/app"
_cover1 "github.com/mendersoftware/mender/cli"
...
...
However, the program still needs to access these structs, and extract the counters from within. This is where the program will generate a coverReport()
function, which needs to be called before exit, in order for coverage to actually be collected.
Thus the generated main file provides this function like so:
package main
import (
"fmt"
"io/ioutil"
"testing"
_cover0 "github.com/mendersoftware/mender/app"
_cover1 "github.com/mendersoftware/mender/cli"
...
...
)
...
...
func coverReport() {
<coverage-magic>
...
...
}
Now, this generated main file needs to be merged with the main.go
file of your project. This is done semi-automatically by the tool.
3 - Merge the generated main file with the packages main.go file
This is done through parsing both main files into each their own AST trees, and then merging the imports explicitly. The rest of the top level nodes are simply added to the new tree which is eventually written out to the main.go
file.
Thus if this is your main file before running the gobinarycoverage
tool on it:
package main
import (
"os"
)
func doMain() int {
<do-stuff>
...
...
}
func main() {
os.Exit(doMain())
}
This is how it will look afterwards:
package main
import (
"testing"
"fmt"
"os"
_cover0 "github.com/mendersoftware/mender/app"
_cover1 "github.com/mendersoftware/mender/cli"
...
...
)
...
...
func coverReport() {
<coverage-magic>
...
...
}
func doMain() int {
<do-stuff>
...
...
}
func main() {
os.Exit(doMain())
}
Still, there is no way for the tool to know, and hence cover all the exit paths of the program, for then to call coverReport()
before exiting. Therefore a small amount of human intervention is required at the end. For most programs, this would simply be making sure that coverReport()
is called before calling exit. In Mender this is simply done through a git patch
.
Example Usage
Using the tool is as simple as:
$ go get -u github.com/mendersoftware/gobinarycoverage
$ cd ~/go/src/github.com/mendersoftware/mender/
$ git apply patches//0001-Instrument-Mender-client-for-coverage-analysis.patch
$ gobinarycoverage github.com/mendersoftware/mender
$ go build .
$ ./mender
And the program will function just like it has done previously! However, note that this will for the moment dirty all the touched source files, so be careful if you have any unstaged changes that you care for (hopefully, this will be fixed some time in the future - when there is time).
That's all for now folks! Hope this is helpful for someone (:
Recent articles
The scope of EU Cyber Resilience Act (CRA) compliance
An overview of EU Cyber Resilience Act (CRA) compliance
Challenges in complying with the EU Cyber Resilience Act (CRA)
Learn why leading companies choose Mender
Discover how Mender empowers both you and your customers with secure and reliable over-the-air updates for IoT devices. Focus on your product, and benefit from specialized OTA expertise and best practices.