Summary

The TL;DR is that, as long as ZFS is being used as the underlying file system, mount commands on top of that will not work. It's simply not supported. I was also able to confirm this over email with Ubuntu/LXD developer, Stéphane Graber, with him saying, in part, that:

overlay doesn't work on top of ZFS. This isn't a permission or a container issue, it's a filesystem issue.

I've outlined two possible solutions below in more detail. Which one works for you may depend on your setup, as was my case, so read carefully.

Jenkins Workaround (My Case)

This "solution" is really more of a workaround than anything else. TL;DR: Don't let jobs that need overlays get scheduled in nodes that live within a ZFS pool.

My server setup boils down to a RAIDed NVMe OS installation drive and a set of larger data drives. The OS drive lacks the capacity for the work that needs to be done and all of the data drives in this setup are VDEVs in the ZFS Pool. This means that there's no other place where the more ideal case of using LXD to add a new non-ZFS pool (see later) could be implemented. (In fact, LXD itself lives within the ZFS pool I had set up.)

Therefore, the workaround here was to re-arrange the Jenkins labels for both the jobs and the nodes. (If you're not familiar with Jenkins, it relies on these labels to determine which nodes can service which jobs/tasks.) The labels are arranged in such a way that Jenkins will never schedule a job that requires mounting an overlay to a LXC-based node that cannot support it.

As the system administrator, you should know how your nodes are set up. The process of re-labeling jobs and nodes is done manually in the node/job configuration pages from Jenkins itself. In this case, you simply make sure that nodes hosted within a ZFS pool never have labels that match those of jobs that need overlays.

Note that your Docker containers must still be launched with the --privileged option for the mount commands to work. This is independent from the ZFS-specific issue described.

Adding Non-ZFS Pools (Ideal)

Note that this solution assumes you have extra drives and/or locations that are outside of the existing ZFS pool. Also note that, as I had said before, I was not able to confirm this myself due to my particular setup. Make sure you understand these steps before trying to apply them, as you do run the risk of destroying your own data, or at least accidentally "hiding" it, if you fail to understand what you're doing.

The TL;DR: is from Stéphane Graber:

Your only way out of this is to have you /var/lib/docker not be on ZFS. You can do that either by completely changing your storage pool to something else or by creating a second pool, allocate a dedicated volume from that and attach it to /var/lib/docker inside your container.

That's effectively the setup we made for Travis-CI where the instance itself is on a ZFS pool but /var/lib/docker comes from a directory (ext4) pool so overlayfs is happy.

This would've been a more ideal solution, but I don't have the ability to implement this in my setup. I'm including it here for the sake of completeness, but this is untested by me and you may need to do more work to properly adapt this solution to your setup.

Note that, in my case, it is Jenkins that's being used to run the jobs and Docker bind-mounts some directories from the host into itself. Therefore, paths in later steps will focus on /var/lib/jenkins instead of /var/lib/docker.

First, understand that you cannot trust the /etc/fstab info from inside the LXC. For example, my LXCs says:

$ cat /etc/fstab
LABEL=cloudimg-rootfs   /        ext4   defaults        0 0

While it clearly claims to be ext4, this ext4 filesystem may still be (and in my case, actually was) on top of ZFS. Therefore, fstab data should be ignored. Stéphane mentions you should rely on /proc/mounts:

Your container is on ZFS, not on ext4, ignore /etc/fstab and look at /proc/mounts instead

In this case, you need to use LXD/LXC to set up a pool that is completely independent/separate from the pre-existing ZFS pool. Then create volumes there and attach them to your LXCs. (This likely means you'll need extra drives, b/c ZFS likes to consume whole drives when adding them to a pool. As an example, if you're using ZFS on Linux, and your installation drive is using an ext4 file system, then you will not be able to include this drive as part of the zfs pool.)

The steps below are what I used prior to remembering that the extra non-ZFS pool was still on top of the existing ZFS pool, so, had it not been for that detail, this should've worked. Note that these steps assume a pre-existing Jenkins installation that you want to preserve. Otherwise, you can remove steps as needed.

After marking your Jenkins node offline and SSH'ing into it, move the Jenkins home directory to a backup location and create a new empty directory for it:

mv /var/lib/jenkins /var/lib/jenkins.old
mkdir /var/lib/jenkins

From a separate shell, SSH into the system hosting the LXCs. I chose to stop my LXC, but this may not be required:

$ lxc stop jenkins-node-01

Then create a non-ZFS storage pool and a storage volume inside of it. Here, the pool's name is jenkins and its driver is dir:

$ lxc storage create jenkins dir
$ lxc storage list              
+----------+-------------+--------+------------------------------------------------+---------+
|   NAME   | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+----------+-------------+--------+------------------------------------------------+---------+
| jenkins  |             | dir    | /var/snap/lxd/common/lxd/storage-pools/jenkins | 0       |
+----------+-------------+--------+------------------------------------------------+---------+
| lxd-pool |             | zfs    | tank/lxc                                       | 7       |
+----------+-------------+--------+------------------------------------------------+---------+

It's up to you to make sure that your LXD installation is not hosted within the ZFS pool in question, like in my setup. Otherwise, this is where you'd be going back to square one.

Then create the node-specific volumes inside of it. This is what I ran for my #1 node:

$ lxc storage volume create jenkins jenkins-node-01
$ lxc storage list                                 
+----------+-------------+--------+------------------------------------------------+---------+
|   NAME   | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+----------+-------------+--------+------------------------------------------------+---------+
| jenkins  |             | dir    | /var/snap/lxd/common/lxd/storage-pools/jenkins | 1       |
+----------+-------------+--------+------------------------------------------------+---------+
| lxd-pool |             | zfs    | tank/lxc                                       | 7       |
+----------+-------------+--------+------------------------------------------------+---------+

Note that, after creating the new jenkins-node-01 volume inside the jenkins pool, it now shows the pool is hosting 1 volume. Attach the volume to your node in the correct path:

$ lxc storage volume attach jenkins jenkins-node-01 jenkins-node-01 jenkins /var/lib/jenkins

Note that both the volume and the node are named jenkins-node-01. This is not an error. To confirm the volume is attached, you can use the lxc config show <node-name> and the volume should show up under the devices section. (If you decided to skip the "backup" step, you will be mounting the volume on top of the pre-existing directory and when you go back to your LXC, the directory will be empty because the volume on top of it is empty. Your data has not been destroyed; only hidden "under" the volume you just mounted. Just detach the volume and don't skip the "backup" step.)

If you had stopped your LXC, you may now run lxc start <your-node> and SSH back into it. From within the LXC node, change the file ownership back to the jenkins account (or whatever account you used) and then copy over the data from the prior backup location into it:

$ chown -R jenkins:jenkins /var/lib/jenkins
$ cp -r /var/lib/jenkins.old/ /var/lib/jenkins/

Your node should now be ready for use and can be brought back online from the Jenkins admin GUI. After you've verified that everything is working as expected, you should be able to remove the /var/lib/jenkins.old/ backup directory.

Be aware that, from now on, if this volume gets destroyed, your data goes with it. If you have backup processes, such as those that use lxc export ..., you may need to modify your process as the export command only includes the containers, not their volumes as you might assume.

posted about 4 years ago

CC BY-SA 4.0

ghost-in-the-zsh‭

703 reputation 8 12 88 45