Firstly, ZFS is installed with a root mirror, using Fedora on ZFS.

DNS was broken out of the box for the installer, so disabled systemd-resolved and set a nameserver in resolv.conf.

systemctl disable systemd-resolved.service
systemctl stop systemd-resolved.service

nano /etc/resolv.conf
# set nameserver to 1.1.1.1

Follow the Fedora on ZFS guide. Clone the repo and check the prereqsites.

git clone https://github.com/gregory-lee-bartholomew/fedora-on-zfs.git
cd fedora-on-zfs
./prereqs

Install Fedora to the two disks we want. This is a HP mini PC, so has one NVME disk and one 2.5" SSD.

./install fedora-disk-server.ks /dev/sda /dev/nvme0n1

After the install was complete, did the usual first-time server setup

  • Configured network
  • Copied SSH keys
  • Disabled password based login over SSH

Discourse requires Docker, so it is installed alongside podman, and our current user was added to the docker user group.

sudo dnf -y install dnf-plugins-core
sudo dnf-3 config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin git

sudo usermod -aG docker $USER
# log out and back in.

We need to configure Docker daemon to use ZFS as the file system driver, as there are issues with OverlayFS and ZFS. Edit /etc/docker/daemon.json and set the storage-driver. I also enable some log settings to manage disk space.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  },
  "storage-driver": "zfs"
}

Start Docker and check the storage status

sudo systemctl enable --now docker

docker info

In the Docker Info output, look for:

Storage Driver: zfs
 Zpool: root
 Zpool Health: ONLINE

Now we need to build Discourse. On the old server Discourse was stopped and the data directory was copied to the new server.

root@oldserver /m/d/forums# ./launcher stop app
root@oldserver /m/d/forums# rsync -av --progress /flash/discourse_data/ user@newserver:/home/forums/discourse_data

On the new server, Discourse was cloned and the app.yml file contents was manually copied.

git clone https://github.com/discourse/discourse_docker.git

nano containers/app.yml

Paste in the contents of the app.yml from the old server. Edit the volumes section to ensure they point to the correct directories on the new server.

There are known issues with PNPM on ZFS, so we need to ensure that PNPM is not using hardlinks. The best way seems to be configuring the package-import-method=copy option in npmrc.

If we do not make this change, each build of Discourse will fail during the pnpm install run with a ERR_PNPM_EAGAIN error similar to:

 ERR_PNPM_EAGAIN  EAGAIN: resource temporarily unavailable, copyfile '/home/discourse/.local/share/pnpm/store/v3/files/5d/1d47fa39f7a7c3a6c03f060be94465085bee995b832a5530d42a8eb77c98d20a1bf8686207f2f1d56dd03148a13360b59a3dbfa6df8211490b365074ee18e4' -> '/var/www/discourse/node_modules/.pnpm/schema-utils@4.3.3/node_modules/schema-utils_tmp_359/dist/keywords/absolutePath.js'
 
[...]

FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && if [ -f yarn.lock ]; then
  if [ -d node_modules/.pnpm ]; then
    echo "This version of Discourse uses yarn, but pnpm node_modules are preset. Cleaning up..."
    find ./node_modules ./app/assets/javascripts/*/node_modules -mindepth 1 -maxdepth 1 -exec rm -rf {} +
  fi
  su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'
else
  su discourse -c 'CI=1 pnpm install --no-frozen-lockfile && pnpm prune'
fi failed with return #<Process::Status: pid 356 exit 1>
Location of failure: /usr/local/lib/ruby/gems/3.3.0/gems/pups-1.3.0/lib/pups/exec_command.rb:131:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"yarn", "cmd"=>["if [ -f yarn.lock ]; then\n  if [ -d node_modules/.pnpm ]; then\n    echo \"This version of Discourse uses yarn, but pnpm node_modules are preset. Cleaning up...\"\n    find ./node_modules ./app/assets/javascripts/*/node_modules -mindepth 1 -maxdepth 1 -exec rm -rf {} +\n  fi\n  su discourse -c 'yarn install --frozen-lockfile && yarn cache clean'\nelse\n  su discourse -c 'CI=1 pnpm install --frozen-lockfile && pnpm prune'\nfi"]}
bootstrap failed with exit code 1

In the app.yml file, scroll down to the hooks section. Add a new after_code hook that creates a .npmrc file that forces the use of copy instead of hardlinks:

hooks:
  after_code:
    - exec:
        cd: $home
        cmd:
          - echo "package-import-method=copy" >> /var/www/discourse/.npmrc
# remaining existing hooks, such as plugin installs:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
# [...]

Build the new container!

./launcher rebuild app

The build and start should complete without issues (At least regarding PNPM).