"Add a component to a System" tutorial

Aim

  • Show how to add a new open source component to a Baserock base system
  • Introduce the related help and reference documentation

Task(s) to be completed

  • make the necessary changes in baserock:baserock/morphs
  • make the necessary changes in upstream:bonnie
  • build the system
  • deploy and run the system

Start state

The end state of the "Change an existing component" tutorial

  • a functioning development environment
  • a populated Trove
  • a populated artifact cache
  • a feature branch containing
    • a built system based on one of the X86 templates

End state

  • A deployed system running the new component

Steps

  • Outline the process
  • Get the source
  • Build outside of Baserock
  • Build in a Baserock system branch
    • Add a new stratum to the system .morph file
      • copy base system to new-system-system.morph
      • edit the system name
      • add the following lines
    • Create a stratum morphology
    • Create the morphology in a new feature branch in the 'bonnie' repo
    • Successfully Build the system
    • Lists some of the errors that we may encounter
      • Not committing the stratum .morph file
      • Not committing the .morph file in the bonnie repo
      • Not doing initial building without --no-git-update
Posted Fri Oct 17 13:21:40 2014

Tutorial 3.4 - "Adding a component to a System"

In this tutorial we are going to learn how to add a component to a system. The component we're going to add is the filesystem benchmarking tool bonnie

The starting point for this tutorial is at least after tutorial 2.1 - "Creating a new system from a template". We now have:

  • a functioning development environment
  • our project master branch at TROVE_ID/joebloggs/master

This time we will demonstrate by working on the project's master branch. However we will make the changes in our own system rather than modifying the base system.

According to the Adding stuff page on the baserock.org wiki the steps involved in adding a new component are:

  1. Get the component source
  2. Build the software in its normal way, without Baserock tools.
  3. Get the source into git, if it is not already.
  4. Add the program to a Baserock system using morphologies, and build that. 

The source for 'bonnie' lives in an svn repository at https://code.google.com/p/bonnie-64/ and it has been lorried on the Trove for this tutorial. To get the source, we go to our system branch checkout directory and clone the git repository from the Trove

cd /src/workspace/TROVE_ID/joebloggs/master
git clone ssh://git@TROVE_HOST/delta/bonnie.git upstream:bonnie
cd upstream:bonnie

Building bonnie is relatively straightforward, we could simply run make.

However, to add the program to our Baserock system we need to do the following

  • Create a chunk morphology to say how to build Bonnie and push that branch to the Trove
  • Create a new fstools stratum, since there is no obvious stratum for it to join.
  • Add the stratum to the system

Creating a Bonnie chunk

  • in the upstream:bonnie directory, create and checkout a new project master git branch

    bash-4.2# git checkout -b TROVE_ID/joebloggs/master
    
  • create the morphology file bonnie.morph with the following content:

    bash-4.2# cat >bonnie.morph <<\EOF
    kind: chunk
    name: bonnie
    build-commands:
    - make
    install-commands:
    - mkdir -p "$DESTDIR/$PREFIX"/bin
    - install -o 0 -g 0 -m 0755 Bonnie "$DESTDIR/$PREFIX"/bin/bonnie
    EOF
    
  • git add and git commit the new file

    bash-4.2# git add bonnie.morph
    bash-4.2# git commit -m "Morphology for building bonnie in Baserock"
    
  • git push the new branch, so that morph build can find it

    bash-4.2# git push -u origin HEAD
    

Creating the fstools stratum

  • In the morphs repository, create the file fstools.morph with the following content:

    bash-4.2# cd ../baserock:baserock/morphs
    bash-4.2# cat >fstools.morph <<\EOF
    name: fstools
    kind: stratum
    build-depends:
    - morph: build-essential
      repo: baserock:baserock/morphs
      ref: TROVE_ID/joebloggs/master
    chunks:
    - name: bonnie
      repo: upstream:bonnie
      ref: TROVE_ID/joebloggs/master
      build-depends: [ ]
    EOF
    
  • git add and git commit the new file

    bash-4.2# git add fstools.morph
    bash-4.2# git commit -m \
        "Create a stratum containing Bonnie"
    

Adding the fstools stratum to our System

  • In baserock:baserock/morphs, add the following lines to my-x86_64-generic.morph, before the configuration-extensions line.

    - morph: fstools
      repo: baserock:baserock/morphs
      ref: TROVE_ID/joebloggs/master
    

    If you've done this correctly the diff should look something like this:

    bash-4.2# git diff my-x86_64-system.morph
    diff --git a/my-x86_64-system.morph b/my-x86_64-system.morph
    index 3801c1b..0564f4c 100644
    --- a/my-x86_64-system.morph
    +++ b/my-x86_64-system.morph
    @@ -18,6 +18,9 @@ strata:
     - morph: bsp-x86_64-generic
       repo: baserock:baserock/morphs
       ref: TROVE_ID/joebloggs/master
    +- morph: fstools
    +  repo: baserock:baserock/morphs
    +  ref: TROVE_ID/joebloggs/master
     configuration-extensions:
     - set-hostname
     - ssh
    

    We don't need to commit the changes yet, as morph build will pick them up automatically.

Building, deploying and testing the system

So now we should be able to build our system, with bonnie added

bash-4.2# morph build my-x86_64-system

and deploy it

bash-4.2# morph deploy --no-git-update kvm my-x86_64-system \
    kvm+ssh://joebloggs@192.168.56.1/bonnie/home/joebloggs/VMs/bonnie.img \
    DISK_SIZE=2G HOSTNAME=bonnie AUTOSTART=yes

If we log into the deployed system we can run bonnie

# bonnie
File './Bonnie.176', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 3...Seeker 2...Seeker 1...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
            0  17.2 20.7  26.8  4.6  20.6  2.9 113.0 100.0 6626.9 99.4 130115 97.6

So there we are - we've added a component to our new system, so we should commit and push our changes.

# morph foreach -- git commit --amend -a -m 'Add Bonnie to my-x86_64-system'
baserock:baserock/morphs
[TROVE_ID/joebloggs/master 45f5495] Add Bonnie to my-x86_64-system
 2 files changed, 5 insertions(+), 2 deletions(-)
bash-4.2# morph foreach -- sh -c 'git push -u origin HEAD 2>&1'
To ssh://git@git.baserock.org/baserock/baserock/morphs
   1ede03f..f5b3192  HEAD -> TROVE_ID/joebloggs/master
Branch TROVE_ID/joebloggs/master set up to track remote branch TROVE_ID/joebloggs/master from origin.

Common pitfalls

There are some new errors we may see while doing this (as well as the ones we talked about in the earlier tutorials):

  • If we didn't git commit the new fstools.morph file in baserock:baserock/morphs

    2013-07-11 16:19:01 baserock:baserock/morphs: Update morphologies to use build branch instead of "TROVE_ID/joebloggs/master"
    ERROR: Command failed: git update-index --cacheinfo 100644 f9300450c9451fc86278f9323e9f518788c144b4
     fstools.morph
    error: fstools.morph: cannot add to the index - missing --add option?
    fatal: git update-index: --cacheinfo cannot add fstools.morph
    
  • If we didn't git commit the new bonnie.morph file in in the bonnie repository

    2013-07-11 16:23:19 Updating cached git repository upstream:bonnie
    ERROR: Failed to determine the build system of repo upstream:bonnie at ref 9007adb991231bd0f9242bb30c0b580f7c8f16b2: was looking for bonnie.morph
    
  • If you haven't pushed the branch of bonnie, you will get an error like the following when you try to build it.

    2013-07-05 13:10:50 Updating cached git repository upstream:nasm
    2013-07-05 13:10:51 Updating cached git repository upstream:syslinux
    2013-07-05 13:10:52 Caching git repository upstream:bonnie
    ERROR: Ref TROVE_ID/joebloggs/master is an invalid reference for repo git://git.baserock.org/delta/bonnie
    
  • If we do our first build with the --no-git-update option:

    ERROR: Repository upstream:bonnie is not cached yet
    

    (It's OK to use --no-git-update on subsequent builds as the source will already be cached)

  • If we didn't add build-depends to the bonnie stratum morphology

    2013-07-05 13:13:30 [Build 77/79] [bonnie] Running build-commands
    build failed
    # build
    # # make
    execv: No such file or directory
    ERROR: In staging area /src/tmp/staging/tmp5Dki6P: running command 'sh -c make' failed.
    
  • If the build-depends of fstools are strata that haven't been modified (e.g. it depends on build-essential and you haven't run morph edit my-x86_64-system build-essential)

    ERROR: Ref TROVE_ID/joebloggs/master is an invalid reference for repo git://git.baserock.org/baserock/baserock/morphs
    
Posted Fri Oct 17 13:21:40 2014