Collecting acceptance and integration test coverage with Golang

This blog post will cover how acceptance and integration test coverage is collected from the Mender client tests.

Starting out, we thought this was going to be a straight forward problem, and there are plenty of blog posts on the web, covering the standard approach. For go test can collect coverage from a do_main() function, just as well as it collects coverage from any other unit tested function. For more details have a look at:

• this
• and this

However, this did not do the trick for us, as our binary does some magic with the std file-descriptors at some point, and this did not go down well the go test framework, which also apparently does some massaging of I/O.

Thus, instead of digging our way to the bottom of the go test framework, to figure out what was going on (we gave up after duplicating the three std file-descriptors before calling main did not work).

At this point I was a bit bummed, and was cursing the world for handing me this ticket. It wasn't even estimated to take more than a day or two, and the boss is like

Still, in apathy, I started poking around the Go tool-chain (which is amazing, btw). Here I stumbled upon the go tool cover, which you might be familiar with from looking at the coverage output from a Go test. However, it can also instrument Go source code for coverage collection. In fact, this is exactly what go test does. This actually means that go test is not needed at all!

This inspired the creation of the gobinarycoverage tool, which this post will dissect in more detail. The gobinarycoverage tool does basically the exact same thing as go test -cover does, except that it does not add any external framework around you binary. This means that the program is guaranteed to function just like it would, if you were not collecting coverage!

Gobinarycoverage - A tool for collecting coverage from Go programs

The Gobinarycoverage tool consists of three main parts:

1) It instruments the source code of all the files which will be part of a main build through wrapping go tool cover, so that coverage can be collected.

2) Then it generates a separate main file, which holds references to all the necessary structures in #1

3) It merges the main.go file with the main file in #2 through using the ast package provided by the Go standard library (thank you!), and parsing the two files into each their separate AST's, and then merges them accordingly.

Let's have a closer look at each step in more detail:

1 - Instrument Go source code with coverage capturing capabilities

The tool is taking advantage of existing go tools' functionality. Notably, it uses go list, in order to figure out which packages main.go imports. From this information, it runs go tool cover on the returned packages. This will change the source code in the given packages to add in a counter at each block, and a GoCover struct to each file, which is responsible for collecting the information.

An instrumented function will afterwards look like:

func (m *MenderError) Error() string {GoCover5.Count[2] = 1;
        var err error
        if m.fatal {GoCover5.Count[4] = 1;
                err = errors.Wrapf(m.cause, "fatal error")
        } else{ GoCover5.Count[5] = 1;{
                err = errors.Wrapf(m.cause, "transient error")
        }}
        GoCover5.Count[3] = 1;return err.Error()
}

And the struct present in every file, collecting this information looks like:

var GoCover5 = struct {
        Count     [2]uint32
        Pos       [3 * 2]uint32
        NumStmt   [2]uint16
} {
        Pos: [3 * 8]uint32{
                35, 37, 0x20025, // [0]
                39, 41, 0x20026, // [1]
        },
        NumStmt: [8]uint16{
                1, // 0
                1, // 1
        },
}

2 - Generate a main.go file which can reach the generated coverage structs

This struct, then has to be imported into the new main.go file. This means that the tools needs to generate source code on the fly, which imports these structs from every file it covers. Imports in the new main.go file will then look something like after code generation has finished (this is simply done through a text template):

package main

import (
        "fmt"
        "io/ioutil"
        "testing"

        _cover0 "github.com/mendersoftware/mender/app"

        _cover1 "github.com/mendersoftware/mender/cli"

        ...
        ...

However, the program still needs to access these structs, and extract the counters from within. This is where the program will generate a coverReport() function, which needs to be called before exit, in order for coverage to actually be collected.

Thus the generated main file provides this function like so:

package main

import (
    "fmt"
    "io/ioutil"
    "testing"

    _cover0 "github.com/mendersoftware/mender/app"

    _cover1 "github.com/mendersoftware/mender/cli"

    ...
    ...
)

...
...

func coverReport() {
   <coverage-magic>
   ...
   ...
}

Now, this generated main file needs to be merged with the main.go file of your project. This is done semi-automatically by the tool.

3 - Merge the generated main file with the packages main.go file

This is done through parsing both main files into each their own AST trees, and then merging the imports explicitly. The rest of the top level nodes are simply added to the new tree which is eventually written out to the main.go file.

Thus if this is your main file before running the gobinarycoverage tool on it:

package main

import (
    "os"
)

func doMain() int {
   <do-stuff>
   ...
   ...
}

func main() {
    os.Exit(doMain())
}

This is how it will look afterwards:

package main

import (
    "testing"
    "fmt"
    "os"

    _cover0 "github.com/mendersoftware/mender/app"

    _cover1 "github.com/mendersoftware/mender/cli"

    ...
    ...
)

...
...

func coverReport() {
   <coverage-magic>
   ...
   ...
}

func doMain() int {
   <do-stuff>
   ...
   ...
}

func main() {
    os.Exit(doMain())
}

Still, there is no way for the tool to know, and hence cover all the exit paths of the program, for then to call coverReport() before exiting. Therefore a small amount of human intervention is required at the end. For most programs, this would simply be making sure that coverReport() is called before calling exit. In Mender this is simply done through a git patch.

Example Usage

Using the tool is as simple as:

$ go get -u github.com/mendersoftware/gobinarycoverage
$ cd ~/go/src/github.com/mendersoftware/mender/
$ git apply patches//0001-Instrument-Mender-client-for-coverage-analysis.patch
$ gobinarycoverage  github.com/mendersoftware/mender
$ go build .
$ ./mender

And the program will function just like it has done previously! However, note that this will for the moment dirty all the touched source files, so be careful if you have any unstaged changes that you care for (hopefully, this will be fixed some time in the future - when there is time).

That's all for now folks! Hope this is helpful for someone (: