Baserock/ projects/ deterministic-builds

Using Baserock

Community

Background

Bit-for-bit reproducibility in Baserock systems

Background

One of the key aims of Baserock is to be able to guarantee a build's result regardless of where and when it was built. To help achieve this, we looked towards work done by the Tor Project and Debian-Reproducible Project, to get a broad idea of the kind of issues we may face and possible approaches we could use in implementing bit-for-bit reproducibility.

Issues

There are a number of issues that may affect build reproducibility;

  • Time of build
  • Hostname of build machine
  • Locale/timezone of build machine
  • Timestamps in gzip headers
  • Internal ordering of .tar objects

Some of these have relatively simple fixes, e.g. setting the environment variable TZ=UTC sets a consistent timezone, removing odd time discrepancies, and setting LC_ALL=C gives a universal locale setting that helps with .tar file ordering; that is, the order isn't locale dependent or varying between alphabets/character sets.

Others, such as the issue of builds running at different times causing SHA1/MD5 discrepancies, require a more in-depth look. At first we tried to implement libfaketime in order to give each component in a build the same build time, but this caused issues in YBD when the build stage reached the make command; each component dependent on libfaketime would break.

Methodology

The first step we performed was to check exactly how a build would differ under different parameters. To do this we set up two near-identical virtual machines with different hostnames, and started building a minimal system on each. Once both builds had been completed, we used the following command to obtain a list of SHA1 and MD5 values for each built artifact:

find /src/cache/ybd-artifacts/ -type f -print0 | xargs -0 <sha/md5>sum > <sha/md5>sum.txt

These were then extracted onto a single machine and compared via diffchecker and vimdiff, which would highlight any cases where the obtained hash differed on the same artifact across the build. This would show which components obtained the build machine's hostname and hardcoded a timestamp internally, both of which were issues that would need to be solved.

Eventually, it may be worth having a list of components and the expected SHA1 hosted in an accessible location so that anyone attempting a build can check they get the expected output.

Work done so far

  • Set environment variable LC=ALL to stop shasum changes to file ordering/locale
  • Set chmod 755 to stop shasum changes due to RWX permissions
  • Set timezone to universal in the environment variables (TZ=UTC)
  • Modify KCONFIG_NOTIMESTAMP in busybox so that the binary doesn't include the build time: change
  • YBD: remove 'elapsed_time' from metadata inside an artifact: commit
  • YBD: Make the artifact tarballs deterministic: pull request

Work still to be done

  • Fix issue with glibc shasums varying between builds
    • Debian Reproducible have sent a patch including changes that would make glibc more reproducible
  • Fix Python .pyc and .pyo files somehow
    • Removing them is possible, but what happens if /usr in the deployed system is readonly? Will Python recompile every .py file every time it is loaded? That would probably be a performance regression
    • PEP 3147 goes into great detail about these files.
    • http://benno.id.au/blog/2013/01/15/python-determinism is also relevant
    • Debian solve this by creating the .pyc and .pyo files at package install time
  • Fix .gz files
    • gzip --no-name would be useful here, but Busybox gzip doesn't implement that option currently.
  • Fix all chunks which embed datetime in their build output. This should be possible by setting configuration flags in most cases, and if a project's build system does not make it possible, we should submit patches to them to make it possible.

Non-deterministic components

After running two builds of a 64 bit build system (x86 architecture), grepping the SHA1 of the build artifacts and running a comparison chec k on the shasum values, we obtained a list of components that were not reproducible:

For this, we ignored stage1 and stage2 components, which due to the nature of staging are not expected to be bit-for-bit reproducible, unlike their stage3 counterparts.

Ongoing maintenance plan

Bit-for-bit reproducibility is like other software functionality: anyone can break it at any time. To ensure this doesn't happen, we need ongoing testing and maintenance.

Continuous builder for Baserock

We have a prototype of a 'continuous YBD' system that can be used to check reproducibility of packages.

It consists of:

To set up your own, you'll need to:

  • set up a machine capable of running YBD. A Baserock 15.19 'build' system on x86_64 is known to work.
  • checkout this branch of YBD and run the deploy.yaml playbook. It has some documentation at the top of the file.

Automated build/check script to obtain non-deterministic components

The following script should, when ran, automate starting two consecutive builds, obtaining the SHA1 of each file in the unpacked artifact tarball, then diff both SHA1sum files to get a list of components that are not reproducible (i.e., the SHA1 of the file varies between builds)

To run:

  • Clone YBD
  • Clone definitions

  • Enter the 'definitions' directory, and run the following command (change the system morphology to that which you wish to check reproducibility for):

    .//path/to/ybd/scripts/build-and-check-reproducibility.sh systems/devel-system-x86_64-generic.morph x86_64

  • The diff should be outputted to diff.clean, which contains a list of all artifacts that are not reproducible.

With this script, it is possible to obtain components that are non-reproducible for any system. This means that we can check exactly which systems have non-deterministic components, which may vary between systems, and then take steps in order to fix them. This may either be a simple change to configuration we can do ourselves, or it may involve sending a patch to the upstream in question.

Bad ideas

faketime

The libfaketime library seems like a useful way of ensuring all timestamps that get embedded in binaries at build-time are set to a consistent value. However, we tried forcing builds to run under libfaketime and discovered that it causes issues with 'make'.

The particular problem we found was that Busybox's build system generates include/autoconf.h from .config based on mtimes, and if there are changes to .config but the mtimes of the two files are the same, it causes a linker error.

While the Tor build process uses 'faketime', the Debian project avoids it except for a small number of packages where it is used as a workaround, because they found that it breaks builds.