"Add a component to a System" tutorial

Aim

Show how to add a new open source component to a Baserock base system
Introduce the related help and reference documentation

Task(s) to be completed

make the necessary changes in baserock:baserock/morphs
make the necessary changes in upstream:bonnie
build the system
deploy and run the system

Start state

The end state of the "Change an existing component" tutorial

a functioning development environment
a populated Trove
a populated artifact cache
a feature branch containing
- a built system based on one of the X86 templates

End state

A deployed system running the new component

Steps

Outline the process
Get the source
Build outside of Baserock
Build in a Baserock system branch
- Add a new stratum to the system .morph file
  - copy base system to new-system-system.morph
  - edit the system name
  - add the following lines
- Create a stratum morphology
- Create the morphology in a new feature branch in the 'bonnie' repo
- Successfully Build the system
- Lists some of the errors that we may encounter
  - Not committing the stratum .morph file
  - Not committing the .morph file in the bonnie repo
  - Not doing initial building without --no-git-update

Posted Fri Oct 17 13:21:33 2014

Tutorial 3.4 - "Adding a component to a System"

In this tutorial we are going to learn how to add a component to a system. The component we're going to add is the filesystem benchmarking tool bonnie

The starting point for this tutorial is at least after tutorial 2.1 - "Creating a new system from a template". We now have:

a functioning development environment
our project master branch at TROVE_ID/joebloggs/master

This time we will demonstrate by working on the project's master branch. However we will make the changes in our own system rather than modifying the base system.

According to the Adding stuff page on the baserock.org wiki the steps involved in adding a new component are:

Get the component source
Build the software in its normal way, without Baserock tools.
Get the source into git, if it is not already.
Add the program to a Baserock system using morphologies, and build that.

The source for 'bonnie' lives in an svn repository at https://code.google.com/p/bonnie-64/ and it has been lorried on the Trove for this tutorial. To get the source, we go to our system branch checkout directory and clone the git repository from the Trove

cd /src/workspace/TROVE_ID/joebloggs/master
git clone ssh://git@TROVE_HOST/delta/bonnie.git upstream:bonnie
cd upstream:bonnie

Building bonnie is relatively straightforward, we could simply run make.

However, to add the program to our Baserock system we need to do the following

Create a chunk morphology to say how to build Bonnie and push that branch to the Trove
Create a new fstools stratum, since there is no obvious stratum for it to join.
Add the stratum to the system

Creating a Bonnie chunk

in the upstream:bonnie directory, create and checkout a new project master git branch
```
bash-4.2# git checkout -b TROVE_ID/joebloggs/master
```

create the morphology file bonnie.morph with the following content:

bash-4.2# cat >bonnie.morph <<\EOF
kind: chunk
name: bonnie
build-commands:
- make
install-commands:
- mkdir -p "$DESTDIR/$PREFIX"/bin
- install -o 0 -g 0 -m 0755 Bonnie "$DESTDIR/$PREFIX"/bin/bonnie
EOF

git add and git commit the new file

bash-4.2# git add bonnie.morph
bash-4.2# git commit -m "Morphology for building bonnie in Baserock"

git push the new branch, so that morph build can find it
```
bash-4.2# git push -u origin HEAD
```

Creating the `fstools` stratum

In the morphs repository, create the file fstools.morph with the following content:

bash-4.2# cd ../baserock:baserock/morphs
bash-4.2# cat >fstools.morph <<\EOF
name: fstools
kind: stratum
build-depends:
- morph: build-essential
  repo: baserock:baserock/morphs
  ref: TROVE_ID/joebloggs/master
chunks:
- name: bonnie
  repo: upstream:bonnie
  ref: TROVE_ID/joebloggs/master
  build-depends: [ ]
EOF

git add and git commit the new file

bash-4.2# git add fstools.morph
bash-4.2# git commit -m \
    "Create a stratum containing Bonnie"

Adding the `fstools` stratum to our System

In baserock:baserock/morphs, add the following lines to my-x86_64-generic.morph, before the configuration-extensions line.

- morph: fstools
  repo: baserock:baserock/morphs
  ref: TROVE_ID/joebloggs/master

If you've done this correctly the diff should look something like this:

bash-4.2# git diff my-x86_64-system.morph
diff --git a/my-x86_64-system.morph b/my-x86_64-system.morph
index 3801c1b..0564f4c 100644
--- a/my-x86_64-system.morph
+++ b/my-x86_64-system.morph
@@ -18,6 +18,9 @@ strata:
 - morph: bsp-x86_64-generic
   repo: baserock:baserock/morphs
   ref: TROVE_ID/joebloggs/master
+- morph: fstools
+  repo: baserock:baserock/morphs
+  ref: TROVE_ID/joebloggs/master
 configuration-extensions:
 - set-hostname
 - ssh

We don't need to commit the changes yet, as morph build will pick them up automatically.

Building, deploying and testing the system

So now we should be able to build our system, with bonnie added

bash-4.2# morph build my-x86_64-system

and deploy it

bash-4.2# morph deploy --no-git-update kvm my-x86_64-system \
    kvm+ssh://joebloggs@192.168.56.1/bonnie/home/joebloggs/VMs/bonnie.img \
    DISK_SIZE=2G HOSTNAME=bonnie AUTOSTART=yes

If we log into the deployed system we can run bonnie

# bonnie
File './Bonnie.176', size: 104857600
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 3...Seeker 2...Seeker 1...start 'em...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    GB M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU M/sec %CPU  /sec %CPU
            0  17.2 20.7  26.8  4.6  20.6  2.9 113.0 100.0 6626.9 99.4 130115 97.6

So there we are - we've added a component to our new system, so we should commit and push our changes.

# morph foreach -- git commit --amend -a -m 'Add Bonnie to my-x86_64-system'
baserock:baserock/morphs
[TROVE_ID/joebloggs/master 45f5495] Add Bonnie to my-x86_64-system
 2 files changed, 5 insertions(+), 2 deletions(-)
bash-4.2# morph foreach -- sh -c 'git push -u origin HEAD 2>&1'
To ssh://git@git.baserock.org/baserock/baserock/morphs
   1ede03f..f5b3192  HEAD -> TROVE_ID/joebloggs/master
Branch TROVE_ID/joebloggs/master set up to track remote branch TROVE_ID/joebloggs/master from origin.

Common pitfalls

There are some new errors we may see while doing this (as well as the ones we talked about in the earlier tutorials):

If we didn't git commit the new fstools.morph file in baserock:baserock/morphs

2013-07-11 16:19:01 baserock:baserock/morphs: Update morphologies to use build branch instead of "TROVE_ID/joebloggs/master"
ERROR: Command failed: git update-index --cacheinfo 100644 f9300450c9451fc86278f9323e9f518788c144b4
 fstools.morph
error: fstools.morph: cannot add to the index - missing --add option?
fatal: git update-index: --cacheinfo cannot add fstools.morph

If we didn't git commit the new bonnie.morph file in in the bonnie repository

2013-07-11 16:23:19 Updating cached git repository upstream:bonnie
ERROR: Failed to determine the build system of repo upstream:bonnie at ref 9007adb991231bd0f9242bb30c0b580f7c8f16b2: was looking for bonnie.morph

If you haven't pushed the branch of bonnie, you will get an error like the following when you try to build it.

2013-07-05 13:10:50 Updating cached git repository upstream:nasm
2013-07-05 13:10:51 Updating cached git repository upstream:syslinux
2013-07-05 13:10:52 Caching git repository upstream:bonnie
ERROR: Ref TROVE_ID/joebloggs/master is an invalid reference for repo git://git.baserock.org/delta/bonnie

If we do our first build with the --no-git-update option:
```
ERROR: Repository upstream:bonnie is not cached yet
```
(It's OK to use --no-git-update on subsequent builds as the source will already be cached)

If we didn't add build-depends to the bonnie stratum morphology

2013-07-05 13:13:30 [Build 77/79] [bonnie] Running build-commands
build failed
# build
# # make
execv: No such file or directory
ERROR: In staging area /src/tmp/staging/tmp5Dki6P: running command 'sh -c make' failed.

If the build-depends of fstools are strata that haven't been modified (e.g. it depends on build-essential and you haven't run morph edit my-x86_64-system build-essential)
```
ERROR: Ref TROVE_ID/joebloggs/master is an invalid reference for repo git://git.baserock.org/baserock/baserock/morphs
```