Baserock/ misc-docs/ deploy-ceph-with-baserock

Using Baserock

Community

Background

Deploying Ceph clusters with baserock

These are intermediate notes on how to set up a ceph cluster using baserock as a deployment tool.

First, we'll need a modified ceph.configure. This is held in the jmac/test-deploy-ceph-cluster branch on https://github.com/jmacarthur/definitions.

The changes to ceph.configure are:

  • Modification of the systemd one-shot scripts to execute a setup script from /root instead of running ceph-diskk prepar. On storage nodes, ceph-disk prepare is run from that script along with other things.
  • Additional scripts written into /root to set up monitor and storage nodes.
  • Fake 'partprobe' tool (/sbin/partprobe) which just returns zero.
  • Simplification of the names and format of the systemd files (e.g. systemd_monitor_fname_template) to remove some parameters (for example, disk is hardcoded as /dev/sdb)

The other changes are:

  • A new configuration extension, 'ssh' which copies an SSH identity from the definitions directory to all deployments. This enables storage nodes to copy the ceph key from the monitor node without a password. There may be a better way to do this.
  • Addition of the 'tools' strata to ceph-service-x86_64-generic.morph.
  • Support for CREATE_DISKS. This has been posted to the baserock-dev mailing list and is currently being reworked.

The current problems are:

  • systemd doesn't start the /root/setup-ceph-head script properly. I don't know why. It may be run too early in the boot process. Running the last line from setup-ceph-head ("/etc/init.d/ceph start mon.0") causes the ceph monitor to start correctly (i.e. 'ceph status' shows HEALTH_ERR along with other information). Running the whole script manually also sets up ceph correctly, but you must stop systemd from attempting to start it if you do this; running the setup-ceph-head script twice will break everything.
  • It's not in a state suitable for upstreaming - the fake partprobe tool is a hack, and the ssh configuration extension is over the top to simple copy a file from one deployment to another. There is probably an existing way to do this in baserock.