Mender in its entirety consists of a number of microservices and tools besides the core client that runs on devices. Since we don't use a monorepo, each of these items has their own repository, dependencies, CI, etc. To keep a handle on the dependencies and our users away from outdated and potentially unsafe dependencies we decided to integrate Dependabot. While the integration of Dependabot on its own was straight forward, the required migration to Go modules still is a bit more involved.
That is until the news arrived, that the Github supported preview variant would be upgraded to an integrated tool combined with some changes to their previous config structure - just as we were about to finish the Go module migration & Dependabot integration.
The migration alone could be just a click in the Dependabot dashboard, but some additional changes to the config are required. Mostly this is to use some of the gained capabilities of Dependabot and to conform to our internal commit requirements that allow us to track the dependency updates in our changelog.
Whereas the requirement for an updated dependency management in our Golang repo required manual work & inspection per repository, this time the changes were easily scriptable. Coincidentally the 1.0 release of the Deno project happened just a few weeks earlier and so far I haven't had a reason to use the Github CLI for anything. All very good reasons to justify getting to know a few tools.
Prerequisites
As mentioned, I had previously not done anything with the tools involved so whoever wants to re-use any part of the script and hasn't installed the tools involved, check this for Deno and that for the Github CLI.
Deno was at version 1.0.5 when I worked on this and the project has made even more progress in the meantime. So while I don't expect any breakage to have happened, be warned: YMMV.
Apart from these two and the more prevalent git command being available, the script itself assumes all the relevant repositories are prefixed with mender and thus more easily identifiable. Furthermore, I work with my fork of each repository as origin, whereas the Mender source remote is known as mender.
Migrate ALL the repositories
The migration assumes that all relevant repositories have preexisting Dependabot configurations and - more often than not - at least some kind of changed files lingering around. That makes for a few steps that the script has to go through in each folder/ repository:
- Check if there is a configuration
- Track the branch the repo is currently on and stash away any changed files
- Set up a new branch to track the migration in
- Do the actual config migration
- Commit and push the changes
- Create a PR
- Restore the repo to its previous branch and changed files if necessary
Most of these steps are nothing more than the execution of their git commands and are best just read through in the full script. There is, however, the config migration part that gives a good example of the changes in the config structure and the ease with which missing capabilities (read: dependencies) can be added in a Deno script:
import init, {
parse,
stringify,
} from "https://deno.land/x/yaml_wasm@0.1.9/index.js";
await init();
...
const adjustDependabotFiles = async (repo: string) => {
...
const configText = await readTextFile(`${source}/config.yml`);
let [config] = parse(configText);
config.version = 2;
config.updates = config.update_configs.map((item: UpdateConfigs) => ({
"commit-message": {
prefix: "Changelog:All",
},
directory: item.directory,
"package-ecosystem": packageManagerMapper[item.package_manager],
schedule: {
interval: item.update_schedule,
},
}));
delete config.update_configs;
const prettyConfig = stringify(config).split(/\n/).slice(1).concat("").join(
"\n",
);
...
};
Using the import init, { parse, stringify, } from "https://deno.land/x/yaml_wasm@0.1.9/index.js"; line is all there is to help parse yaml files and write them, skipping any npm or pip installation. While the change in structure, as can be seen e.g. in the schedule or the package-ecosystem definition, could have been done through the Dependabot dashboard, the addition of the commit-message prefix should be a good enough reason for anyone to put this into a proper script!
Due to the convenient pull-request-creation-link that's returned on a push to a new branch nowadays, I haven't felt a need for the Github CLI in my day to day work and since I find that link really convenient I doubt this will change. But a script combined with the Github CLI is something else. The process of setting up a pull request for the specific repo is not more than:
gh pr create --title "dependabot update" --body "migrated to stable dependabot release" --base master --draft --repo mendersoftware/$repoName
To prevent excessive noise and tons of running CI pipelines all the PRs were created as draft pipelines - which made for a rewarding click-through adventure at the end. Having created the PR all that was left was restoring the repo to where it was, so checking out the previous branch and popping anything we stashed away.
Since I have around ~30 Mender related repositories in my folder (including a few experimental ones), working with the asynchronous functions of the Deno standard library turned out to be slightly more time consuming than what I initially planned to invest. To increase my efficiency, I took the liberty to opt out of the asynchronous nature for the file iteration part though, as it would lead to too many files opened at once (some of our repos have quite large change sets) and other unwanted effects.
for (const thing of readDirSync(folder)) {
const { name: dir } = thing;
if (dir.startsWith('mender')) {
await processRepo(`${folder}/${dir}`);
}
}
While executing the script the permission system Deno is working with is soon noticeable. Contrary to what is allowed in Node.js, anything executed by Deno needs explicit approval for everything that goes beyond downloading an import: executing a command? provide the --allow-run permission, accessing files? provide the --allow-read or --allow-write permissions, .... While it is possible to just allow everything or set more granular permissions, the mere existence of the permission system gives a prominent pointer to the access implications. In my opinion, this is especially relevant due to the convenient dependency retrieval and when compared to allowing the quiet execution of everything in a sudo sh run.sh call.
Due to the config file reading and writing, the occasional git invocation and the call to create a PR or update the pushed branch the final call to the script looks like this:
$ deno run --allow-read --allow-run --allow-write --allow-net dependabotUpdater.ts /my/repos/folder
and did what it was intended for. The script itself can be found in our GODS repo.
About Deno dependency management
Having worked with npm and the corresponding ecosystem for quite some time, one of the biggest differences while working with Deno was the dependency management. While there are several different approaches to keep track of dependencies in Deno projects, I am not sure there is a definitive solution yet. But since this is just a one off "good-old-damn-script" to run, I have no problems keeping the dependencies right in the script. That is, of course, just possible, because Deno takes care of the dependency retrieval and storage - all without the need for a node_modules folder per project. The dependencies will however be cached in your system's cache directory, so that there is no need to retrieve them on every execution. Due to this I could just take a look at the Deno libraries, include the location in the module import and run the script - and it just worked. For a single script that was very neat.
Wrap up
Getting Dependabot working for you is a helpful addition and even if it is just for the automatic reminder to take care of certain dependencies. Wherever it is possible I highly recommend setting it up - or any other automation for that matter - to help you take care of your projects.
Based on these few steps I've taken with Deno I can happily recommend it to anyone using Node for scripting purposes as the transition should be super easy. Especially being able to use Typescript and all the nice ES6 features out of the box without any babel installation is a relief. Or using await on the top level - no wrapping functions required. As to if I would use it for an entire project instead of Node... I don't see a terribly compelling reason to do so. But since the hurdle is so low and at least some benefits are to be gained, I also don't see a terribly compelling reason not to do so.
Regarding the Github CLI: I would at least recommend it in a script if you can or have to install other tools anyway.
As this post and the corresponding script might show: I'm not an expert on either Typescript or Deno. So if you see anything worrisome: please feel free to reach out and let me know what I could or should improve or create a merge request on Gitlab on the script itself.
Future work
As described in the introduction the Golang repos had to be migrated to Go modules for Dependabot to do its job. This was due to the dependency vendoring we used in almost all of the affected repositories or the absence of a vendoring mechanism during their creation. While the bulk of the things relevant to the vendoring process could be part of another blog post, the support for the go mod vendor effect in Dependabot is part of an open issue that prevents us to fully utilize the potential Dependabot brings to our repositories. Until this is integrated, we might likely get to see some additional work done on the integration.
