Baserock release process
These are the steps to go through to make a Baserock release. The release process is a work in progress, so expect it to require a lot of improvement over time. Also, further tool support is needed.
Releases tend to involve uploading lots of data. As a rule of thumb, you should aim to be uploading the release images and artifacts the night before the date that the release is due.
Prerequisites for making a release
Note: Typically, only Baserock maintainers have enough access to complete this process.
You will need:
- Push access to the
- Write access to the backing store for
- A fresh
x86_32at a push) VM to coordinate the release. This VM should have enough space to coordinate the release artifacts. In the past, 30G or more was required. Please refer to other guides for setting up such a VM.
You will also need access to:
- A local Trove synced with
x86_64distributed build infrastructure.
x86_32distributed build infrastructure.
armv7lhfdistributed build infrastructure.
- Configuration for the above suitable for use with your coordination VM.
morph.conf on your coordination VM is configured to use your
internal Trove by setting
trove-host as appropriate.
definitions.git for the systems we release.
In addition to the system images and other files produced by
release.morph, the release requires manually producing or updating
- Release notes, on
- Updated symlinks to "current" images and tarballs
For a GENIVI release, the following artifacts are produced:
- GENIVI baseline manifest and license manifest
- GENIVI release notes
Before making the release, make sure that the previous release of Baserock can build the new release. The 'master' branch of definitions is continuously built by http://mason-x86-64.baserock.org and http://mason-x86-32.baserock.org/ so, as long as these are running the previous release, you can be sure that there are no compatibility issues.
If there are planned incompatible changes to definitions, make sure that the version of Morph in the release is capable of understanding the new versions, so that the changes can be committed to 'master' of definitions.git after you tag the release.
You should do the release in a clean instance of the previous Baserock devel system release, preferably on x86 (for speed). This makes it possible for others to reproduce the release images you created.
- Create a new branch (
definitions.git. This is the "release branch", and any changes made during the release will happen there, avoiding the need to freeze master while the release is happening.
- Get the branch mirrored to your local Trove (push to
git.baserock.org, promote on your local Trove to get it lorried quickly).
- log into your local Trove, run this:
curl -X POST -d path=baserock/baserock/definitions http://localhost:12765/1.0/move-to-top
- then wait and check the Trove's status page until the
definitionsrepository has been lorried
- log into your local Trove, run this:
scripts/release-buildto produce release artifacts. This script builds all the artifacts needed for the release (using distbuild, so they end up on your local Trove), and creates the release artifacts (all the images and tar archives etc).
- See the
scripts/release-build.test.confsample configuration file and use that as a basis for your own.
- See the
- Compress the release artifacts with gzip.
release-uploadshould do this.)
scripts/release-upload release.morphto publish release artifacts. This copies build artifacts from your local Trove to
git.baserock.org, and other files to
download.baserock.org. After this script has finished successfully, Baserock users can start using the new release.
- See the
scripts/release-upload.test.confsample configuration file and use that as a basis for your own.
- See the
- Manually create the "current" symlinks pointing at the new release. (FIXME: The upload script should do this automatically, but that code hasn't been written yet.)
- Tag the HEAD of the release branch with the new release
baserock-X.Y). Push the release branch, including tags (
git push --tags).
- Prepare or update release notes, wikis, and websites.
- Drafting these can start early on, while building and uploading are running.
The upload script assumes that you have appropriate credentials to upload files to the servers using rsync. If a password is required, rsync will prompt for your password at that stage of the script. If you rely on SSH agent forwarding to provide access, and are leaving the release script running overnight, remember that the machine running your SSH agent must also remain running overnight!
GENIVI Baseline release steps
For the GENIVI Baseline systems there are several extra steps. Beware: these scripts are slow and can take several hours!
Generate a manifest for each baseline system, using the
morph generate-manifest /src/cache/artifacts/*.genivi-baseline-system-armv7lhf-jetson-rootfs > \ /src/release/genivi-baseline-system-armv7lhf-jetson-manifest.txt morph generate-manifest /src/cache/artifacts/*.genivi-baseline-system-x86_64-generic-rootfs > \ /src/release/genivi-baseline-system-x86_64-generic-manifest.txt
NOTE: You will need the rootfs artifacts because while generate-manifest claims to operate from disk images, the code is not in-sync with our current raw disk generator
Generate a license manifest for each baseline system, using the
license-checkscript in definitions.git:
scripts/licensecheck.py systems/genivi-baseline-system-armv7lhf-jetson.morph > \ /src/release/genivi-baseline-system-armv7lhf-jetson-license-manifest.txt scripts/licensecheck.py systems/genivi-baseline-system-x86_64-generic.morph > \ /src/release/genivi-baseline-system-x86_64-generic-license-manifest.txt
Extract the kernel zImage file from the ARMv7lhf system, by loopback mounting it and copying the file out. (FIXME: there is an unmerged patch in morph.git branch baserock/richardipsum/write_kernel to make
morph deployhandle this automatically).
Write GENIVI release notes. These should summarise the changes relevant to developers of systems based on the Baserock GENIVI Baseline system, and link to the Baserock release notes. See http://projects.genivi.org/GENIVI_Baselines/genivi-baserock/blog for examples.
You'll need to upload these extra files to the download.baserock.org server manually; the script doesn't take care of this yet (see 'future plans', below).
Announcing the release
Once you've successfully completed all the steps to produce and upload all the files for the release:
- Update all pages on
wiki.baserock.orgthat refer to the previous release number to the new release number.
- Send release notes to email@example.com and firstname.lastname@example.org.
Announcing the GENIVI release
- Send announcement email to email@example.com and firstname.lastname@example.org
- Post release notes into blog at http://projects.genivi.org/GENIVI_Baselines/genivi-baserock. Currently Jonathan Maw (jonathanmaw), Pedro Alvarez (pedroalvarez), Sam Thursfield (ssam2) and Rob Taylor (robtaylor) have admin access to do this. WARNING: the 'preview' renderer will show you something quite different to the actual rendering after you've pressed 'post'.
- Review wiki.baserock.org's GENIVI related how-tos.
Any incompatible changes to definitions that have been planned, but were waiting for a Baserock release with a version of Morph that could understand them, can now be merged to 'master' of definitions.git.
If you are able, upgrade http://mason-x86-64.baserock.org/ and http://mason-x86-32.baserock.org/, so that new features of Morph from the new release can be used. If you are not an op, remind a member of the operations team to do this.
Define and deploy a virtual machine configured specifically for doing releases, which can be used by anyone reasonably trusted that wants to make a release. This should replace the first couple of sections of this documentation, which quite lengthy and manual. Our cloud infrastructure has very fast upload bandwidth which makes releases much less time-consuming.
Checksum everything we release, to guard against corruption at all stages of the release process.
Consider the bit-for-bit reproducibility of the release images, and how we can make it easy for those who want to reproduce the Baserock release images that are produced as part of this process.
Allow greater flexibility in what we release. Some of us think that Baserock's release process should, in future, be as follows: the build-system image is released every week (this image is capable of building any other Baserock image), other systems can optionally be released at the same time according either to schedule or need. For example, we can continue to release the GENIVI Baseline demo system every 6 weeks, following the GENIVI Baseline release process. We can release other example systems when they have received significant changes to merit a new release.
Automate GENIVI releases. Ideally, this should be done in a generic way, treating the GENIVI Baseline system as if it was one of many example Baserock systems which each had their own release requirements.
Ensure that this script is useful for projects derived from Baserock, as well as for the Baserock project itself.