Baserock/ projects/ Importing expressions from NixOS

Using Baserock

Community

Background

Importing expressions from NixOS

Here we are going to describe our experiences trying to import NixOS expressions. Our main goal is to parse NixOS expressions to create something that could be used to build via Baserock build tools.

Expressions

Nix expressions is a language. That's why when you browse the NixPkgs repo you will hardly understand what's going on when installing a package. Every package has a default.nix file with information about how to build the package (name, version, sources, ...). In some of them there are more useful information than in others, but they mostly inherit the 'build instruction from other places.

Lets see an example with the Nox package:

Confusing, right? At least in the last file we can see the generic phases for building and installing Python packages, but it's really confusing how to get there to extract the information. That's why I decided to try Nix tools to extract it.

Using Nix to extract information from expressions

We found in Nix wiki a useful Cheatsheet that compares some useful commands in Ubuntu with their equivalent in Nix.

In the following examples we are going to use the package firefox, but bear in mind that you can use any package.

Derivations

A derivation is a representation of the build instructions used for building a package. They are files with the extension .drv which are listed in the store directory, usually /nix/store.

With the nix-instantiate command we can evaluate Nix expressions to generate a derivation:

$ nix-instantiate '<nixpkgs>' -A firefox
/nix/store/8l1vqkvdqcc62wl0yavwmkm9mnm09g3m-firefox-40.0.2.drv

Now we can use the derivation file, and even use it to get more information.

List package depenencies

To list the immediate dependencies you can use the --references query of the nix-store command:

$ nix-store --query --references $(nix-instantiate '<nixpkgs>' -A firefox)
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh
/nix/store/1gb9gjl6kp84a3sy0kkfb5685dhhsl2j-bash-4.3-p39.drv
. . .
/nix/store/xlxpjnrz69zlhjf8hcqrmqgjaagb7y7p-jemalloc-3.6.0.drv
/nix/store/xsh3sxf2ilivmnzkffz7l0rhqcwnl6q0-unzip-6.0.drv
/nix/store/zayax2nj2rjkj3cbd6x40c1n1wf6gykz-hunspell-1.3.3.drv

And as you can see, this command points us to the derivation files of the dependencies, and should be able to use them to find the dependencies of the package recursively. It sometimes points to other things that are not derivations, but shell scripts, or even folders with the dependency itself.

Build instructions

All the information is already (I believe) in the derivation file, and there is an option in nix-store to show the shell environment needed to run the build. This option is --print-env.

$ nix-store --print-env $(nix-instantiate '<nixpkgs>' -A firefox)
export buildInputs; buildInputs=''
export builder; builder='/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash'
export configureFlags; configureFlags='--enable-application=browser --disable-javaxpcom --with-system-jpeg ...
...
export system; system='x86_64-linux'
export _args; _args='-e /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh'

If you run these commands and then run:

$builder $_args

It will run all the phases of creating a package, assuming that the dependencies needed are already in the system. An easy trick to fetch them is to run:

nix-shell '<nixpkgs>' -A firefox

This command will put you in an environment ready to build firefox, and part of that process involves downloading the dependencies. If you exit from that environment you will be able to run $builder $_args, and now with the dependencies.

At this point you probably think that could be easy to create a script that given a package, calculates recursively its dependencies, and runs the $builder script for every of them. I still think that it should be possible, although I failed in my first attempt because my script found some dependenciy loops.

Trying to convert the expressions to definitions

After having having all the information and evaluated it, we can say that it is going to be difficult to convert the expressions to something similar to the Baserock definitions. The scripts that are being called to build a single package are really big, and looks a bit difficult to extract the essential build steps from them, given that the language is really flexible.

Another difficulty to achieve this might be the fact that there are paths hardcoded in all the build instructions (not in the expressions themselves, but in the information that we could extract using Nix tools)

Worth noting that there is an expressions parser tool called simple-nix, but I failed to get it working easily, and gave up after an hour trying. It might be useful to explore to see if it's possible to do anything useful with it.