Baserock/ guides/ Deploying a Swift cluster

Using Baserock

Community

Background

Deploying a Swift cluster

This guide shows how to deploy OpenStack with Swift. In this guide we deploy a three node Swift cluster, one controller node with two storage nodes. We assume each storage node has three storage devices attached to it, /dev/sdc, /dev/sdd and /dev/sde, these should all be formatted with xfs.

The exact means of deployment: kvm, rawdisk etc are not discussed here, if you are not familiar with building and deploying Baserock systems you should read Using and Developing Baserock Systems before continuing. If you're planning to deploy directly to hardware see Deploying to hardware.

General OpenStack deployment options are not described here, but are described in detail in the OpenStack on Baserock guide

There are several cluster options we need to define in order to deploy our Swift cluster:

  • SWIFT_ADMIN_PASSWORD - this is the Swift user's password, this user has complete control over the Swift account that will be created for the service tenant, the user can create and destroy containers, upload, download and destroy objects.

  • SWIFT_PART_POWER - Swift divides all its physical storage into blocks called partitions, these are not to be confused with traditional hard disk partitions which bear no relation to Swift partitions. The number of partitions in your cluster will be SWIFT_PART_POWER raised to the power of two, the number of partitions has to be a power of two because Swift computes an md5 hash of the object's full name (the name of the object plus the name of the container and account the object belongs to) and uses the first n bits (where n = SWIFT_PART_POWER) as an index into the table of partitions.

  • SWIFT_REPLICAS - every object in Swift is replicated across different devices and nodes, this variable defines how many times an object should be replicated.

  • SWIFT_MIN_PART_HOURS: this defines the minimum amount of time that must pass between subsequent relocations. Defined as 1, this means that if an object is relocated, 1 hour must elapse before it can be moved again.

  • SWIFT_STORAGE_DEVICES: this defines the storage devices that belong to this Swift cluster, each device must have associated with it, the ip of the node it is attached to, a device name without the leading /dev path (so for a device /dev/sda, this name will be sda) and a weight, the weight can be any number, but typically each device in the cluster should be assigned the same weight.

  • SWIFT_CONTROLLER: this sets whether the node is a controller node or a storage node.

Our cluster setup

Nodes

For this guide we assume three nodes, and a single management network, which amongst other things will be used by Swift for object replication. The controller node will have hostname swift-controller and ip 10.0.0.1, storage node 0 will have hostname swift-storage-0 and ip 10.0.0.2, and storage node 1 will have hostname swift-storage-1 and ip 10.0.0.3

Devices

We assume each storage node has three devices attached, /dev/sdc, /dev/sdd and /dev/sde

Deploy a Swift controller

Our Swift controller will be based on the openstack-system-x86_64.morph system. Configuration of Swift is performed by two configuration extensions, swift-storage and openstack-swift-controller, neither of these are enabled in the openstack-system-x86_64.morph system by default, so we will need to add them ourselves.

Add the extensions

cat >> systems/openstack-system-x86_64.morph << EOF
- swift-storage
- openstack-swift-controller
EOF

Build the system

morph build systems/openstack-system-x86_64.morph

Deploy the system

Use the cluster template below to deploy OpenStack on a single node, replacing the secret values SWIFT_ADMIN_PASSWORD, SWIFT_HASH_PATH_PREFIX, and SWIFT_HASH_PATH_SUFFIX with your own secret values. Do not lose these values, and keep them secret.

Also make sure to set the passwords used for the various OpenStack services.

name: openstack-one-node-swift
kind: cluster
systems:
- morph: systems/openstack-system-x86_64.morph
  deploy:
    release:
      type: extensions/rawdisk
      location: swift-controller.img
      DISK_SIZE: 10G
      INSTALL_FILES: openstack/manifest swift/manifest

      HOSTNAME: swift-controller

      #########################################################################
      ## Swift config options
      #########################################################################

      SWIFT_CONTROLLER: True

      SWIFT_ADMIN_PASSWORD: changeme

      SWIFT_PART_POWER: 10
      SWIFT_REPLICAS: 3
      SWIFT_MIN_PART_HOURS: 1

      SWIFT_STORAGE_DEVICES: [{ ip: 10.0.0.2, device: sdc, weight: 100 },
                              { ip: 10.0.0.2, device: sdd, weight: 100 },
                              { ip: 10.0.0.2, device: sde, weight: 100 },

                              { ip: 10.0.0.3, device: sdc, weight: 100 },
                              { ip: 10.0.0.3, device: sdd, weight: 100 },
                              { ip: 10.0.0.3, device: sde, weight: 100 }]

      # This value can be any random string or number
      # but each node in your Swift cluster must have the same values
      SWIFT_REBALANCE_SEED: 3828

      # NOTE: Replace SWIFT_HASH_PATH_PREFIX and SWIFT_HASH_PATH_SUFFIX
      # with your own unique values,
      #
      # `openssl rand -hex 10' can be used to generate unique values
      #
      # These values should be kept secret, do not lose them.
      #
      SWIFT_HASH_PATH_PREFIX: 041fc210e4e1d333ce1d
      SWIFT_HASH_PATH_SUFFIX: 4d6f5362a356dda7fb7d

      #########################################################################

      RABBITMQ_HOST: swift-controller
      RABBITMQ_PORT: 5672
      RABBITMQ_USER: rabbitmq
      RABBITMQ_PASSWORD: veryinsecure

      CONTROLLER_HOST_ADDRESS: swift-controller
      MANAGEMENT_INTERFACE_IP_ADDRESS: 10.0.0.1

      KEYSTONE_TEMPORARY_ADMIN_TOKEN: 22f3aa1cf538e3f6d5e8
      KEYSTONE_ADMIN_PASSWORD: veryinsecure
      KEYSTONE_DB_USER: keystoneDB
      KEYSTONE_DB_PASSWORD: veryinsecure

      GLANCE_SERVICE_USER: glance
      GLANCE_SERVICE_PASSWORD: veryinsecure
      GLANCE_DB_USER: glanceDB
      GLANCE_DB_PASSWORD: veryinsecure

      NOVA_SERVICE_USER: nova
      NOVA_SERVICE_PASSWORD: veryinsecure
      NOVA_DB_USER: novaDB
      NOVA_DB_PASSWORD: veryinsecure
      NOVA_VIRT_TYPE: qemu

      CINDER_SERVICE_USER: cinder
      CINDER_SERVICE_PASSWORD: veryinsecure
      CINDER_DB_USER: cinderDB
      CINDER_DB_PASSWORD: veryinsecure
      # Storage device to be used by Cinder
      CINDER_DEVICE: /dev/sdb

      NEUTRON_SERVICE_USER: neutron
      NEUTRON_SERVICE_PASSWORD: veryinsecure
      NEUTRON_DB_USER: neutronDB
      NEUTRON_DB_PASSWORD: veryinsecure
      METADATA_PROXY_SHARED_SECRET: novaneutronmetasecret

      HOSTS_CONTROLLER: 10.0.0.1 swift-controller

      # Network interface to be used, only needed if there are more
      # than one available.
      # EXTERNAL_INTERFACE: eno1

Deploy the storage nodes

Our controller nodes will be based on the swift-system-x86_64.morph, two nodes can be deployed with the following cluster.

The values SWIFT_PART_POWER, SWIFT_REPLICAS, SWIFT_MIN_PART_HOURS, SWIFT_STORAGE_DEVICES, SWIFT_REBALANCE_SEED, SWIFT_HASH_PATH_PREFIX, and SWIFT_HASH_PATH_SUFFIX must be defined with the same values that were used to deploy the controller, the result of any mismatch between these variables is undefined.

name: example-swift-storage-cluster
kind: cluster
systems:
- morph: systems/swift-system-x86_64.morph
  deploy-defaults:
    INSTALL_FILES: swift/manifest

    CONTROLLER_HOST_ADDRESS: 10.0.0.1

    SWIFT_PART_POWER: 10
    SWIFT_REPLICAS: 3
    SWIFT_MIN_PART_HOURS: 1

    SWIFT_STORAGE_DEVICES: [{ ip: 10.0.0.2, device: sdc, weight: 100 },
                            { ip: 10.0.0.2, device: sdd, weight: 100 },
                            { ip: 10.0.0.2, device: sde, weight: 100 },

                            { ip: 10.0.0.3, device: sdc, weight: 100 },
                            { ip: 10.0.0.3, device: sdd, weight: 100 },
                            { ip: 10.0.0.3, device: sde, weight: 100 }]

    # This value can be any random string or number
    # but each node in your Swift cluster must have the same value
    SWIFT_REBALANCE_SEED: 3828

    # NOTE: Replace SWIFT_HASH_PATH_PREFIX and SWIFT_HASH_PATH_SUFFIX
    # with your own unique values,
    #
    # `openssl rand -hex 10' can be used to generate unique values
    #
    # These values should be kept secret, do not lose them.
    #
    SWIFT_HASH_PATH_PREFIX: 041fc210e4e1d333ce1d
    SWIFT_HASH_PATH_SUFFIX: 4d6f5362a356dda7fb7d

    FSTAB_SDC: /dev/sdc /srv/node/sdc xfs noatime,nodiratime,rw 0 0
    FSTAB_SDD: /dev/sdd /srv/node/sdd xfs noatime,nodiratime,rw 0 0
    FSTAB_SDE: /dev/sde /srv/node/sde xfs noatime,nodiratime,rw 0 0

  deploy:
    node0:
      type: extensions/rawdisk
      location: swift-storage-0.img
      DISK_SIZE: 10G
      HOSTNAME: swift-storage-0
      MANAGEMENT_INTERFACE_IP_ADDRESS: 10.0.0.2
    node1:
      type: extensions/rawdisk
      location: swift-storage-1.img
      DISK_SIZE: 10G
      HOSTNAME: swift-storage-1
      MANAGEMENT_INTERFACE_IP_ADDRESS: 10.0.0.3

Note, since all data in Swift is replicated you may want to mount the xfs with the 'nobarrier' option, this will provide increased performance by relaxing the ordering constraint on journal writes. xfs enables write barriers by default for safety, beware, disabling this slightly increases the risk of journal corruption.

Testing Swift

Define the following environment file and source it,

$ cat > swiftrc << EOF
export OS_TENANT_NAME=service
export OS_USERNAME=swift
export OS_PASSWORD=changeme
export OS_AUTH_URL=http://swift-controller:35357/v2.0
EOF

$ source swiftrc

Verify authentication with Swift

Run swift stat to verify Swift can authenticate with Keystone.

$ swift stat
     Account: AUTH_56ce9cb09c29461c86c4d489f75e23ad
     Containers: 0
        Objects: 0
          Bytes: 0
X-Put-Timestamp: 1429292739.08127
     Connection: keep-alive
    X-Timestamp: 1429292739.08127
     X-Trans-Id: tx16a4703f180443b3ae0ed-00553146c2
   Content-Type: text/plain; charset=utf-8

Verify uploading/downloading objects

Upload object

$ echo "Hello world" > hello
$ swift upload test_container hello
hello

List containers for this account

$ swift list
test_container

Download object

$ rm hello
$ swift download test_container hello
hello [auth 0.456s, headers 0.496s, total 0.496s, 0.000 MB/s]

$ cat hello
Hello world