Clusterducks

vDisks: Persistent datastores for (nonpersistent) devices

vDisks are clusterducks method for storage of device-specific data. They can be automatically snapshot and replicated without significant performance loss at intervals as frequently as 1-minute.

For most clusters, configuration management (CM) engines are the tool du jour for ensuring all devices are in the desired state.

In clusterducks, the recommended mode of operation for OS images is non-persistent - the entire OS image is reprovisioned from a golden snapshot at boot time. While it is possible to run a network of systems with persistent OS storage so each device effectively have their own image, it will lead to image sprawl; most of the benefits of using shared storage will disappear.

Ephemeral OS storage is a problem for most configuration engines that require agent software to be independently configured for each device, though there are workarounds.

In the case of Puppet, an agent may run on each managed device that contains (at the bare minimum):

  • A hostname
  • An authentication key that is registered to the central server

If we managed a hypothetical fleet of 10 virtual machines without clusterducks vDisk:

  • One (golden) image is created with the puppet agent included
  • 9 other VMs boot into individual clones of a snapshot of this golden image
  • Each VM is individualized and given a hostname and a puppet key
    • These changes must not be lost, or puppet will not recognize the device. Reprovisioning would be a disaster!

vDisk Overlay

Linux provides the overlayfs filesystem and clusterducks supplies a working initrd module for dracut. FreeBSD has a similar unionfs but clusterducks does not provide native integration with it as of v0.9.2.

GNU\Linux

For vDisk overlay support in Linux, clusterducks provides a dracut module that uses the storage server API during boot time to discover and connect to its assigned iSCSI targets.

If a vDisk is configured and available in a VM or bare metal device:

  • If you want to partition the vDisk, go ahead and do so. You can also format the whole disk.
  • For whole disk,
    • $ mkfs.xfs /dev/sdb
    • For partitions,
    • $ mkfs.xfs /dev/sdb1
    • XFS is recommended, though it does have the ability to expand, it cannot shrink. Start small and work your way up to avoid dataset bulge.
  • In clusterducks, browse to your device's vDisk assignment page and bring up the Assignment Properties dialogue for your newly created vDisk
  • Enable "OverlayFS"
  • Click the "+" button (Add Partition/Mountpoint)
  • In the "key" field, enter your vDisk partition number (the X in sdbX) or 0 for whole-disk filesystems (sdb)
  • In the "value" field, enter the path for that partition to be mounted to without the leading / (i.e. "data" will mount on /data)
  • Save the settings and reopen the dialogue
  • In overlayfs settings, use:
    • Lower: /
    • Upper: /data/overlayfs
    • Work: /data/workdir
    • These settings will create an overlay filesystem on your vDisk mounted on /data

Once these settings are changed, reboot the VM. If the initial rootdisk contains the clusterducks module, your device will attach to this vDisk during boot and enable the overlay. Any device filesystem changes will now be applied to the overlay on the vDisk, and the OS image will be left alone.

This has a few effects:

  • Application/device data is stored separately from OS data, simplifying management and classification of what's important; OS data is less important than instance details
  • One OS image may be shared among many vastly-different VM or bare metal devices
    • Simplified management in the case of source-compiled distributions like Gentoo, where keeping multiple independent images up-to-date is taxing on shared resources
  • OS reprovision can occur regularly because local image changes remain in overlayfs
    • Leave all overlayfs devices in a locked/frozen state, they reprovision on reboot
    • Deleting a file from the overlay will remove it from the root filesystem unless it is mounted read-only. However, a reboot of a thawed device will reprovision the OS image layer and it will "reappear" in the unioned filesystem.