you are anything like me, you are always messing up fiddly configurations with multiple servers within one installation. Isolating them in different machines is more resistant to this particular fault as well as providing security against the machines servers creating ways in to interfere with other servers, or even getting stuck in a dead loop and causing other processes to be slowed down.

The chroot is not as secure as VM or physical isolation, this is not a reason to do it. If your issue is just to keep the individual server configurations distinct, you can do it and not use up a lot of extra storage space to boot.

BTRFS filesystem

First thing to note is that you should put everything (maybe except the /boot partition) into a single btrfs volume. With the default installation of 16.04 from the server install ISO, this automatically creates a root volume with two subvolumes, one for home and one for the root, labeled

@home

and

@

These can have snapshots taken very quickly, they only copy the file map and not the file, and the BTRFS filesystem allows multiple files to allocate the same storage block, and always writes new blocks instead of changing old ones, and clearing old ones that no file tree database has a reference to, when it reuses them.

This magic is called Copy on Write (CoW) and optimises the amount of writing to disk that is done, while enabling snapshots and subvolumes. What it allows you to do is take one filesystem, and copy it in about 5 seconds, and then you can change the second one separately to the original, it's a whole new filesystem, but only the database of the file map is copied.

With btrfs, we can then take our simple base installation of our operating system (in my case, Ubuntu 16.04.1, the recommended platform for running witnesses), snapshot it, and on the snapshot filesystems instead try to get your individual server set up. If you mess up the image, you can just delete the subvolume, create it again and go and try again. The core system will never be changed, only the subvolumes.

So how to do it

First, you need to set up your server installation. I am going to make the assumption you are familiar with how to set up an installation. I recommend a configuration like this:

• /boot make this ext2 or ext4
• swap I have set this to double my 16Gb system memory, because the witness uses a memory mapped file which can get very big. Unlike most program data, however, the great majority of it is not needed at any one time, depending on the enabled functions.
• / The root volume should be formatted as btrfs

As usual you should set up a non-root user as directed in the server text mode configuration program. This user is what you use when logging into the system remotely. It is not advisable to have a password on root because there is bots on the internet that find open SSH servers and tries to log in as root. The default password security configuration on Ubuntu is no root password, and generates one default user who has the right to become root.

Once you have got this up and running and have a console on it (with a VPS, over the internet via virtual terminal). You will need to enable the SSH server, use the following commands:

sudo update-rc.d ssh enable
sudo service ssh start

After doing this, you can log into the server using its' IP address and with your default username:

ssh [email protected]

Creating your chroots

Log into your system and type the following command:

mount

and it will output information like this:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8196240k,nr_inodes=2049060,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643228k,mode=755)
/dev/sda3 on / type btrfs (rw,relatime,space_cache,subvolid=257,subvol=/@)

The one you see that says /dev/sda3 on / is your root partition on btrfs. You can see it says subvol=/@, which indicates that on device /dev/sda3 there is a thing that looks like a folder but it is a special BTRFS node called a 'subvolume'. Subvolumes are isolated from each other but they can share data blocks.

So, to make new subvolumes, you need to mount the device that / is mounted on (/dev/sda3 in this case)

Firstly, we need to have that btrfs root mounted somewhere, edit the /etc/fstab and copy the entry for the root ( / ) and change the mount point to something like /btrfsroot. Here's mine:

UUID=0efc70e9-081d-4e5f-b8c8-74b8ef180d75 /               btrfs   defaults,subvol=@ 0       1
...
UUID=0efc70e9-081d-4e5f-b8c8-74b8ef180d75 /btrfsroot      btrfs   defaults 0       1

Then create that mountpoint:

sudo mkdir /btrfsroot

and you can then mount it like so:

sudo mount /btrfsroot

It will be automatically mounted in future if it is added to the /etc/fstab.

I have written a couple of scripts that you will want to use because they are very simple and create and mount your snapshots of your core system automatically.

#!/bin/bash
btrfs subvolume snapshot $1 $2
echo "PS1='`echo $2`: \[\e]0;\u@\h: \w\a\]\u@\h:\w$ '">> $2/root/.bashrc
rm $2/etc/resolv.conf
cp /etc/resolv.conf $2/etc/

This script creates the snapshot of your chosen system subvolume, adds a distinctive prompt to the root user's profile to indicate which one it is in, and correctly copies a working resolve.conf so the chroot knows where to ask for DNS name resolution.

To invoke this, you run the script with the following parameters:

createchroot.sh <source system subvolume> <new subvolume>

This could look like this:

createchroot.sh /btrfsroot/@ /btrfsroot/@apache

I save this script and the next on into files in /usr/local/bin/ and use chroot a+x /usr/local/bin/createroot.sh to make them runnable like the example just above.

Next, to actually use the chroot. For that, I have created a second script:

#!/bin/bash
mount -o bind /proc $1/proc
mount -o bind /sys $1/sys
mount -o bind /dev $1/dev
mount -o bind /dev/pts $1/dev/pts
rm $1/etc/resolv.conf
cp /etc/resolv.conf $1/etc/resolv.conf
export OLDPROMPT="$PS1"
chroot $1 /bin/bash
export PS1=$OLDPROMPT
umount $1/proc
umount $1/sys
umount $1/dev/pts
umount $1/dev

This script cleanly sets up and shuts down a session inside the chroot, and ensures the old prompt reappears when we exit the chroot.

Conclusion

From here on, once you are inside your chroots, you can administer the system as normal, as though it was inside the main installation, but none of the changes you make will affect the main installation. You can then easily just wipe it out and start again, or you can, with a little btrfs magick, make progressive backups to keep track of your changes and enable you to roll it back.

Note that once you have the chroots correctly set up, after you install a server on it, you will need to create other scripts to automatically launch the servers from your root system's configuration. Whether they are installed as services in the virtual system or if you just build source and write your own startup scripts. The objective of this post was to get you to the position where you can set up and log in to administer space-optimised chroot containers to avoid breaking your core server when you want to run more than one thing.

For example with running Django, you basically must also run a reverse proxy, and you might have to also run other things together, like mysql or other things. So being able to box them up like this protects you from newbie mistakes and lets you get up and running faster by keeping the files separated. This arrangement is also simpler in terms of managing network ports and the like, the system core is common to all three, the network security should be added to the core system, however.

---

We can't code here! This is Whale country!

Written with StackEdit.