"Build a system and debug a failing build" tutorial

Aim

  • Show how to

    • build a Baserock base system
    • avoid or fix common causes of build failure
  • Introduce the related help and reference documentation

Task(s) to be completed

  • Use morph build to build the base system successfully
  • Use morph build in an incorrect directory to see the associated error

Start state

  • a functioning development environment
  • a populated Trove
  • workspace not initialised

End state

  • a built base system ready to be deployed

Steps

  • Log in to development environment
  • Invoke morph build base-system-x86_64-generic from /root
  • See the build fail
  • Initialise the workspace
  • Check out the master branch of the Baserock system morphologies
  • Go to the workspace/master directory
  • Invoke morph build base-system-x86_64-generic
  • See the build succeeding

Mention building from outside the branch checkout directory when you have made a branch

Mention more advanced build debugging techniques that can be used

Public

Posted Fri Oct 17 13:21:33 2014

Tutorial 1.2 - "Building a System"

In this tutorial we are going to learn how to build a Baserock base system, and we'll see some of the common problems we might run into and how to get around them.

The starting point for this tutorial is that we have a functioning development environment, which we set up in the previous tutorial.

We build a baserock system using the morph build command. So let's go to our development environment and look at morph help build

# morph help build
Usage: morph [options] build SYSTEM

Build a system image in the current system branch

Command line arguments:

* `SYSTEM` is the name of the system to build.

This builds a system image, and any of its components that need building. The
system name is the basename of the system morphology, in the root repository
of the current system branch, without the `.morph` suffix in the filename.

The location of the resulting system image artifact is printed at the end of
the build output.

You do not need to commit your changes before building, Morph does that for
you, in a temporary branch for each build. However, note that Morph does not
untracked files to the temporary branch, only uncommitted changes to files git
already knows about. You need to `git add` and commit each new file yourself.

Example:

morph build devel-system-x86_64-generic

morph build acts like make: it works out which components need to be built, builds them, and creates our system image.

The parameter we give is the name of the system that we are going to build. We're going to build the 64-bit x86 base system so the command we'll use is

# morph build base-system-x86_64-generic

If we change the last parameter we can build other systems - 32 bit or ARM

If we run that command we'll get an error, because we haven't finished setting up our environment.

bash-4.2# morph build base-system-x86_64-generic
ERROR: Can't find the workspace directory.
Morph must be built and deployed within the system branch checkout within the workspace directory.

We need to run morph init to create our workspace for the tools to work in - again, we can get information by running morph help init

bash-4.2# morph help init
Usage: morph [options] init [DIR]

Initialize a workspace directory.

Command line argument:

* `DIR` is the directory to use as a workspace, and defaults to the current
  directory.

This creates a workspace, either in the current working directory, or if `DIR`
is given, in that directory. If the directory doesn't exist, it is created. If
it does exist, it must be empty.

You need to run `morph init` to initialise a workspace, or none of the other
system branching tools will work: they all assume an existing workspace. Note
that a workspace only exists on your machine, not on the git server.

Example:

morph init /src/workspace
cd /src/workspace

We'll put our workspace in a sub-directory of /src called workspace. Currently there's not much in /src

bash-4.2# cd /src
bash-4.2# ls
cache       morph       morph.conf  morph.log   tmp

Let's run morph init

bash-4.2# morph init workspace
bash-4.2# ls
cache       morph       morph.conf  morph.log   tmp         workspace

bash-4.2# ls workspace
bash-4.2# ls -a workspace
.       ..      .morph

and we can see that morph has created a workspace directory, which doesn't contain very much yet.

If we were to run that command again we'd get an error, because the workspace already exists:

bash-4.2# morph init workspace
ERROR: can only initialize empty directory as a workspace: workspace

Our next step is to use morph checkout to check out the master branch of the Baserock system morphologies, which tell morph how to build systems. Again, morph checkout is covered in the morph help checkout command.

bash-4.2# morph help checkout
Usage: morph [options] checkout REPO BRANCH

Check out an existing system branch.

Command line arguments:

* `REPO` is the URL to the repository to the root repository of a system
  branch.
* `BRANCH` is the name of the system branch.

This will check out an existing system branch to an existing workspace. You
must create the workspace first. This only checks out the root repository, not
the repositories for individual components. You need to use `morph edit` to
check out those.

Example:

cd /src/workspace
morph checkout baserock:baserock/morphs master

The parameters we give morph checkout are the repository containing the system morphologies, and the branch we want to work with.

We need to run morph build inside our workspace.

bash-4.2# cd workspace

We're going to build the master branch so our command is

bash-4.2# morph checkout baserock:baserock/morphs master
2013-06-19 11:30:45 Updating git repository baserock:baserock/morphs in cache
bash-4.2# ls -a
.       ..      .morph  master

morph has created a directory master which contains the branch we checked out, with all the system morphology files

> bash-4.2# ls -R master
> master:
> baserock:baserock
>
> master/baserock:baserock:
> morphs
>
> master/baserock:baserock/morphs:
> README                                        genivi-baseline-system-armv7-versatile.morph
> audio-bluetooth.morph                         genivi-baseline-system-x86_64-generic.morph
> base-system-armv7-highbank.morph              genivi-devel-system-armv7
> base-system-armv7-versatile.morph             genivi-devel-system-armv7-versatile.morph
> ....

Earlier, we encountered the error:

Morph must be built and deployed within the system branch checkout within the workspace directory.

so we need to move inside the master directory, then run morph build.

One of the useful properties of Morph is that it caches the artifacts it builds, and only rebuilds them if it is necessary because one of their dependencies has changed. So, as I happen to have built a base system in this VM before, the build won't take too long. If I hadn't done that the build would take significantly longer.

bash-4.2# pwd
/src/workspace
bash-4.2# cd master
bash-4.2# morph --verbose build base-system-x86_64-generic
2013-06-07 11:14:18 Starting build 6e166c30f67741039012e5b9c00575ff
....
2013-06-07 11:19:11 [Build 76/77] stratum bsp-x86_64-generic is cached at /src/cache/artifacts/06d7f46ac9bc7f44f655f731d6e7b0f81099971e6b3e34a90d0a2d2883df83ae.stratum.bsp-x86_64-generic
2013-06-07 11:19:11 [Build 77/77] [base-system-x86_64-generic-rootfs] Checking if system needs building 3478a8e
2013-06-07 11:19:11 [Build 77/77] [base-system-x86_64-generic-rootfs] The system is already built
2013-06-07 11:19:11 [Build 77/77] system base-system-x86_64-generic-rootfs is cached at /src/cache/artifacts/bef789bfe96c5492de7d87da8cd2bf550c677fddbba1a933f3d785ad6a07fcff.system.base-system-x86_64-generic-rootfs
2013-06-07 11:19:11 Build ends successfully
2013-06-07 11:19:11 Finished build 6e166c30f67741039012e5b9c00575ff

And there we are. Our system is built and up-to-date.

If we look in /src/cache/artifacts we can see everything that was created during this build and earlier ones. Baserock caches these artifacts and only rebuilds them if it is necessary because one of their dependencies has changed. Each artifact has the its cache key and artifact type (chunk, stratum, system) prepended to its name.

bash-4.2# ls /src/cache/artifacts | head
001b16071e0d99036497ff519b83e95df275d4ba72f1d1374ed2715dc93d7396.chunk.curl
05aeb6b96e7ad76523cfb033b158187af966075b78bd1425e0625b8fec775c9a.chunk.automake
06d7f46ac9bc7f44f655f731d6e7b0f81099971e6b3e34a90d0a2d2883df83ae.stratum.bsp-x86_64-generic
06d7f46ac9bc7f44f655f731d6e7b0f81099971e6b3e34a90d0a2d2883df83ae.stratum.bsp-x86_64-generic.meta
...
bef789bfe96c5492de7d87da8cd2bf550c677fddbba1a933f3d785ad6a07fcff.meta
bef789bfe96c5492de7d87da8cd2bf550c677fddbba1a933f3d785ad6a07fcff.system.base-system-x86_64-generic-kernel
bef789bfe96c5492de7d87da8cd2bf550c677fddbba1a933f3d785ad6a07fcff.system.base-system-x86_64-generic-rootfs
bef789bfe96c5492de7d87da8cd2bf550c677fddbba1a933f3d785ad6a07fcff.system.base-system-x86_64-generic-rootfs.overlaps
c351132412f8a0b2efd742ae3d630458bd8f52103c31be38b4c77ec59f827905.chunk.gobject-introspection
...

There are some other errors you may see when you do a build.

  • If you have more than one branch and you are in the workspace directory but not inside the directory corresponding to one of the branches you will see:

      bash-4.2# morph build base-system-x86_64-generic
      ERROR: Can't find the system branch directory.
      Morph must be built and deployed within the system branch checkout.
    
  • If you specify the name of the system's morph file instead of the name of the system that you want to build (easy to do using tab completion) you will see:

      bash-4.2# morph build base-system-x86_64-generic.morph
      2013-06-07 11:34:44 Starting build 683384b518ea4677ba26994893c1f794
      2013-06-07 11:34:44 Collecting morphologies involved in building base-system-x86_64-generic.morph from master
      ERROR: [Errno 2] No such file or directory: '/src/workspace/master/baserock:baserock/morphs/base-system-x86_64-generic.morph.morph'
    
  • If you try to build a stratum rather than whole system (which is easy to do because system and stratum morphology files all live together in baserock\:baserock/morphs/) at the moment you will see:

      bash-4.2# morph build core
    
      2013-06-07 11:37:25 Starting build 0794284fc65b4b729a9dcb111467b814
      2013-06-07 11:37:25 Collecting morphologies involved in building core from master
      Traceback (most recent call last):
        File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
          self.process_args(args)
        File "/src/morph/morphlib/app.py", line 193, in process_args
          cliapp.Application.process_args(self, args)
        File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 537, in process_args
          method(args[1:])
        File "/src/morph/morphlib/plugins/branch_and_merge_plugin.py", line 1555, in build
          branch, branch_dir, branch_root, system_name)
        File "/src/morph/morphlib/plugins/branch_and_merge_plugin.py", line 1621, in get_system_build_repos
          for info in system_morphology['strata']:
        File "/src/morph/morphlib/morph2.py", line 83, in __getitem__
          return self._dict[key]
      KeyError: 'strata'
    

In due course you will see a more informative error message.

There are more tips on debugging failed builds on the Tips and Tricks page at baserock.org

Posted Fri Oct 17 13:21:33 2014