Baserock/ Feature Wishlist

Using Baserock

Community

Background

This aims to reflect wishes of some Baserock stakeholders, including contributors and commercial users.

Certain Wishes

Workflow

  • drop workspaces: https://storyboard.baserock.org/#!/story/40
  • drop system/stratum/chunk terminology, support recursive definition structures, minimise duplication
  • drop branch|checkout|edit|merge
  • core functionality
    • build: run a set of commands from definitions for a component in a working tree
    • assemble: construct a cache artifact from a git ref or equivalent
    • deploy: create a portable artifact in a specified location
    • curate: manage a collection of git repositories representing the sources used for the construction of artifacts
    • publish: share cached artifacts, either internally within an organisation, or in a public location
    • validate: run tests against a deployment: either against a static artifact, or a running system
    • integrate: track changes in repositories referenced in definitions, and create definitions patches including these changes.
    • update: atomically modify systems running from an artifact filesystem to use a modified artifact filesystem.
    • automate: cause assembly, publication, deployment, and validation from integration. Also, cause deployment to result in installation, instantiation, and/or boot of a bootable deployed artifact.
  • guarantee perfect traceability and reproducibility if user has pushed commit
  • no magic pushes or commits
  • don't duplicate git, or git commands
  • user should be able to use cached git repos easily and effectively
  • ability to build any level of component
    • an individual software component
    • a logically-coupled set of components
  • ability to deploy (from a definition file) any level of component
  • ability to replace/update components of a constructed system at runtime (Richard Maw has proposed some potential solutions for this).
  • fast loop development on a group of coupled components (e.g. weston, wayland, mesa)

Extended Features

  • Provide continous validation service: as a point of comparison, GNOME continuous can build a GNOME image on a gtk3 change in 15 minutes (https://build.gnome.org/#/build/20141126.4)
  • Support building apps for systemd sandboxing
  • Requires the ability to build non-bootable systems
  • Requires the ability to deploy appropriate format containers

Multi-platform

  • run on different platforms (in approximately this order of priority…)
  • on Baserock
  • on any linux
  • on any POSIX (inc MacOS)
  • ideally everywhere

Public services

  • Public cache server containing all chunks defined in master
  • Public git server containing mirror of all repos defined in master (must not require that downstream users download everything that appears anywhere upstream all the time)
  • Public patch tracker allowing anyone to submit stuff for inclusion
  • Public pre-merge CI on every candidate to patch tracker
  • Public automated candidate production for any upstream

Decomposition

  • Morph should become build tool, deploy tool, parsing library, sandbox wrapper, etc.
  • Mason should become candidate analysis service, test orchestrator, test runners, results reporting, etc.
  • Trebuchet should be separated into tools to generate delta, transmit delta, construct new system, etc.
  • Trove should be in separate parts: cache server, git server, lorry server, etc.
  • Any component of the Baserock tooling should work independently, without requiring specific implementations of other tools and/or services (e.g use of github as a git server)

Performance

  • get to a build faster (faster calculation of build-order, cache-keys e.g. < 5 secs)
  • ability to work effectively when offline or behind firewall (possibly with caches, possibly by building locally)
  • Reduced local caching: skip anything not actually needed, including skipping source where artifact is cacheable
  • Initial developer response to anything is required in single-digit seconds, and ideally less than a second.
  • Traceability and reproducibility guarantees only apply for committed sources referenced by committed definitions, and may not be reproducible on other hosts until all relevant commits are pushed.

Uncertain Wishes

The following sections are still under consideration

Cross-compilation and Canadian-compilation

  • This is not an immediate need: native build continues to be considered more reliable and widely supported
  • Any notation and cache-key calculation choices should assume these features are present

Non-git component sources

It may be interesting to allow component sources to be collected by means other than git. Some examples include:

  • pip mirror
  • npm mirror
  • gem mirror
  • tarball archive (via http or ftp)
  • network file system

It is unclear whether such means should be permitted for the collection of sources, or how to provide any reliable traceability or reproducibility when doing so. Note that many components packaged in these ways may contain large binary blobs, for which git is not an ideal storage solution. If they are not permitted, there must be a simple means by which owners of these classes of source may inject them into git for use by the Baserock tooling.

Non-Baserock artifact inclusion

It may be interesting to allow artifacts to be defined as specific accessible binary blobs, perhaps in formats other than the cache artifacts typically generated in Baserock. Such artifacts may include:

  • cache artifacts from another team using Baserock for which source/definitions are not shared
  • binary tarball drops from vendors not using Baserock
  • Other data formats where license restrictions prohibit modification

It is unclear whether such artifacts should be supported in a Baserock-managed build and deploy process. If they are not supported, there must be a simple means by which owners of these classes of artifact may inject them into a Baserock process to allow the contents to be present in a deployment.