Caching

One of the design principles of Baserock is that it should always be safe and practical to share built artifacts. This guarantee is fundamental for packaging systems, but not so common in existing build tools.

We would like for all builds to be bit for bit reproducible, but right now it is enough for them to be "functionality reproducible". This means that we are confident that any given artifact will be functionally the same, if rebuilt from the same inputs in the same environment that it was before.

The Baserock 'artifact' concept is very similar to traditional packages, but totally inverted. A user of Debian, for example, can choose at runtime to install a prebuilt 'vim' package. By contrast, with a Baserock build tool, you include an instruction to build Vim from source in your definitions. When the build tool executes, it can check against local and/or remote artifact caches to see if a compatible build was already done, and if so, can reuse that build instead of rebuilding.

Thus, the Debian packages for 'vim' can be installed on any Debian OS of matching version and architecture, and they carry information about what OS version and architecture they expect, what other packages they need in order to run, and can run arbitrary code install and uninstall. By constrast, the Baserock artifacts for Vim need to encode the environment they were built in, so that the build tool can decide if the existing build is compatible or not.

Another way to look at this is that Debian packages depend on manually-constructed version numbers to know whether or not they are compatible with each other. A human has to specify in the Debian rules file that Vim now requires GLIB 2.20, where it used to require GLIB 2.19.

Cache identity

Baserock build tools don't require any manual versioning. Instead, they automatically work out a 'cache identity' for each build, based on all of the things that could cause functional changes. The exact set of factors isn't specified[1], but it includes host machine architecture, commands that will be used to build, Git SHA1 of the commit that will be built, and the cache identity of all dependencies that will be made available when the build runs.

Morph's code to calculate cache identity is currently here: http://git.baserock.org/cgit/baserock/baserock/morph.git/tree/morphlib/cachekeycomputer.py#n81

YBD's code to calculate cache identity is here: https://gitlab.com/baserock/ybd/blob/master/ybd/cache.py

Both tools will first create a Python dict containing all the factors, then will calculate the SHA256 hash of the dict, to produce a 64-character string that uniquely identifies the artifact. YBD features versioning of the cache-key algorithm itself, and supports sharing of cached artifacts via an artifact server, for example the one at http://artifacts1.baserock.org:8000

Caching in other build/integration/packaging tools

Buildroot: no built-in support

BitBake: sstate cache

CMake: possible with Artifactory.cmake (specifically for use with JFrog Artifactory)

Nix & GUIX: package substitutes

Related projects

ccache: this caches at the level of individual object files, not whole components. It can coexist with the caching in Baserock build tools.

JFrog Artifactory: commercial artifact cache server. No Baserock build tools integrate with Artifactory, currently.

Trove: Baserock server appliance that supports artifact caching (using 'morph-cache-server').

[1]. It probably should be specified!!!