Jenn TechnologyNixOS & ZFS Case Study

Bare-Metal Deployment & Disaster Recovery

This setup is an example of a deployment for an on-site business bare metal server, simplified here to use 2 x 16TB HDDs with offsite remote backup to another 16TB HDD. The client had previously been a victim of a ransomware attack and was concerned about a repeat. As a result, various security and backup/nuke/restore protocols have been deployed to reassure the client that if an intrusion is detected, they can recover swiftly and safely.

1. Boot & Partitioning

Rationale: We employ a LUKS encrypted block device as the foundation to protect against physical theft of the server. ZFS is layered on top to provide bitrot protection and atomic snapshot capabilities across multiple terabytes of data.

NixOS is best installed remotely. Do not type commands directly on the server monitor. Boot the physical machine into the NixOS installer, set a temporary password, and SSH in from another local device:

bash
# On the physical console, set temp password:
passwd nixos

# Get local IP address:
ip a

# On your local laptop/desktop, SSH in to copy-paste commands:
ssh nixos@YOUR_INSTALLER_IP_ADDRESS

Partitioning the SSD (LUKS Root)

Format your system SSD (usually /dev/nvme0n1 or /dev/sda) with LUKS encryption:

bash
# Wipe and partition
sudo sgdisk -Z /dev/nvme0n1
sudo sgdisk -n 1:0:+1G -t 1:ef00 -c 1:BOOT /dev/nvme0n1
sudo sgdisk -n 2:0:0 -t 2:8309 -c 2:cryptroot /dev/nvme0n1
udevadm settle

# Format EFI
sudo mkfs.vfat -F 32 -n BOOT /dev/nvme0n1p1

# Setup LUKS Encryption (Write down this passphrase in your external vault!)
sudo cryptsetup luksFormat /dev/nvme0n1p2
sudo cryptsetup luksOpen /dev/nvme0n1p2 root

# Format Ext4 Root
sudo mkfs.ext4 -L root /dev/mapper/root

# Mount filesystems
sudo mount /dev/mapper/root /mnt
sudo mkdir -p /mnt/boot
sudo mount /dev/nvme0n1p1 /mnt/boot

Partitioning the ZFS Storage Pool (HDDs)

Configure datasets and encryption keys for your hard drives (e.g. /dev/sda and /dev/sdb):

bash
# Generate secure, random passphrases (SAVE these key strings to Bitwarden!)
sudo mkdir -p /mnt/etc/zfs/keys
sudo tr -dc 'A-Za-z0-9!?%#' < /dev/urandom | head -c 32 > /mnt/etc/zfs/keys/datapool.key
sudo tr -dc 'A-Za-z0-9!?%#' < /dev/urandom | head -c 32 > /mnt/etc/zfs/keys/backuppool.key
sudo chmod 600 /mnt/etc/zfs/keys/*.key

# Copy keyfiles to the host temporarily so zpool create commands can read them
sudo mkdir -p /etc/zfs/keys
sudo cp /mnt/etc/zfs/keys/*.key /etc/zfs/keys/
sudo chmod 600 /etc/zfs/keys/*.key

# Create Main Storage Pool
sudo zpool create -f \
  -o ashift=12 \
  -O encryption=aes-256-gcm \
  -O keyformat=passphrase \
  -O keylocation=file:///etc/zfs/keys/datapool.key \
  -O compression=on \
  -O mountpoint=legacy \
  datapool /dev/sda

# Create Backup Pool
sudo zpool create -f \
  -o ashift=12 \
  -O encryption=aes-256-gcm \
  -O keyformat=passphrase \
  -O keylocation=file:///etc/zfs/keys/backuppool.key \
  -O compression=on \
  -O mountpoint=legacy \
  backuppool /dev/sdb

# Create legacy datasets for services
sudo zfs create -o mountpoint=legacy datapool/nextcloud
sudo zfs create -o mountpoint=legacy datapool/odoo
sudo zfs create -o mountpoint=legacy datapool/paperless
sudo zfs create -o mountpoint=legacy datapool/hr-system
sudo zfs create -o mountpoint=legacy datapool/uptime-kuma
sudo zfs create -o mountpoint=legacy datapool/invoiceninja

# Optimize postgres database block write layout (16k matching write records)
sudo zfs create -o mountpoint=legacy datapool/postgres
sudo zfs set recordsize=16k datapool/postgres

# Clean up temp keys
sudo rm -rf /etc/zfs/keys

Mounting ZFS Datasets and Installing

bash
# Mount datasets inside /mnt target
sudo mkdir -p /mnt/mnt/datapool /mnt/mnt/backuppool
sudo mount -t zfs datapool /mnt/mnt/datapool
sudo mount -t zfs backuppool /mnt/mnt/backuppool

sudo mkdir -p /mnt/mnt/datapool/nextcloud /mnt/mnt/datapool/odoo /mnt/mnt/datapool/paperless /mnt/mnt/datapool/hr-system /mnt/var/lib/postgresql /mnt/var/lib/private/uptime-kuma /mnt/var/lib/invoiceninja

sudo mount -t zfs datapool/nextcloud /mnt/mnt/datapool/nextcloud
sudo mount -t zfs datapool/odoo /mnt/mnt/datapool/odoo
sudo mount -t zfs datapool/paperless /mnt/mnt/datapool/paperless
sudo mount -t zfs datapool/hr-system /mnt/mnt/datapool/hr-system
sudo mount -t zfs datapool/postgres /mnt/var/lib/postgresql
sudo mount -t zfs datapool/uptime-kuma /mnt/var/lib/private/uptime-kuma
sudo mount -t zfs datapool/invoiceninja /mnt/var/lib/invoiceninja

# Copy config and trigger build
sudo cp -r /path/to/your/nixos-config/* /mnt/etc/nixos/
sudo cp /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hardware-metal.nix

cd /mnt/etc/nixos
sudo git init
sudo git add -A
sudo git commit -m "baremetal build"
sudo nixos-install --flake .#office-server-metal --no-root-passwd

2. Remote Self-Destruct Trigger

Rationale: A dedicated, hardened management node (such as a low-power, business-class micro PC like a Dell OptiPlex Micro, HP ProDesk Mini, or Lenovo ThinkCentre M-series tiny) is used exclusively for out-of-band management. This node holds the SSH keys and automation scripts required to wipe or restore the main office-server without exposing these dangerous capabilities directly to the primary network.

This allows you to securely delete LUKS headers and reinstall the clean configuration from the management node, triggered with one-click on an Android device (LineageOS).

1. Nuke Shell Script (on Hardened Management Node)

bash
#!/usr/bin/env bash
# /opt/nuke-rebuild.sh
set -euo pipefail

TARGET_IP="100.64.1.20" # Target office-server Tailscale IP
FLAKE_PATH="/home/operator/nixos-config"

# Shred root encryption key (destroys active data immediately)
ssh -o ConnectTimeout=5 root@$TARGET_IP "cryptsetup luksErase /dev/disk/by-partlabel/cryptroot" || echo "LUKS already wiped or unreachable."

# Force clean bootstrap over network
nixos-anywhere --flake "$FLAKE_PATH#office-server-metal" root@$TARGET_IP

# Report completion
apprise -t "Wipe & Restore Finished" -b "The office-server has been successfully wiped and reinstalled." "tgb://YOUR_TELEGRAM_BOT_TOKEN/YOUR_CHAT_ID"

2. Termux Widget Trigger (on Android Phone)

Using Termux from F-Droid, you can place a widget button on your home screen. When tapped, it connects to the Pi and runs the script:

bash
# In Termux on your phone, generate and trust SSH key to the Pi:
ssh-keygen -t ed25519
ssh-copy-id nuke-operator@YOUR_MANAGEMENT_NODE_TAILSCALE_IP

# Create shortcut script:
mkdir -p ~/.shortcuts
cat << 'EOF' > ~/.shortcuts/Nuke_OfficeServer.sh
#!/usr/bin/env bash
ssh -t nuke-operator@YOUR_MANAGEMENT_NODE_TAILSCALE_IP "sudo /opt/nuke-rebuild.sh"
EOF

chmod +x ~/.shortcuts/Nuke_OfficeServer.sh

Now, add the **Termux:Widget** 1x1 shortcut directly to your LineageOS phone launcher screen.

3. Offsite Replications (Syncoid)

Rationale: Using syncoid over a Tailscale mesh VPN provides an encrypted transport layer with zero open firewall ports. Because ZFS uses incremental, block-level transfers, bandwidth usage is minimized, and sending raw encrypted datasets ensures the remote target cannot read the contents.

We automate ZFS pool backups over Tailscale to a remote server. The backups are sent encrypted so the remote server cannot read the raw data:

nix
  # Add to configuration / zfs-storage.nix
  services.syncoid = {
    enable = true;
    interval = "daily";
    commands = {
      "offsite-backup" = {
        source = "datapool";
        target = "root@your-offsite-backup-tailscale-ip:offsitepool/datapool-backup";
        # '--sendoptions=w' keeps ZFS data encrypted in transit and on target
        extraArgs = [ "--recursive" "--sendoptions=w" ]; 
      };
    };
  };

4. Disaster Recovery & Rollback

Rationale: Leveraging ZFS rollbacks instead of traditional file-based restoration allows for the instantaneous recovery of multi-terabyte datasets while perfectly preserving relational database integrity.

If you suspect the system was compromised 48 hours ago, for example, you can revert all datasets (Nextcloud, Paperless, Postgres) back to a safe snapshot taken before that window (e.g. 3 days ago).

bash
# 1. List available snapshots on backup pool
sudo zfs list -t snapshot -r backuppool/datapool-backup/nextcloud

# 2. Stop host application services
sudo systemctl stop phpfpm-nextcloud paperless-web odoo postgresql hr-system invoiceninja

# 3. Pull clean snapshots back to the main pool (if wiped)
sudo syncoid --sendoptions=w root@offsite-backup:offsitepool/datapool-backup/nextcloud datapool/nextcloud
sudo syncoid --sendoptions=w root@offsite-backup:offsitepool/datapool-backup/postgres datapool/postgres

# 4. Rollback to safe snapshot (e.g. June 1st) - This instantly discards post-compromise edits!
sudo zfs rollback -r datapool/nextcloud@autosnap_2226-06-01_00:00:00_daily
sudo zfs rollback -r datapool/postgres@autosnap_2226-06-01_00:00:00_daily
sudo zfs rollback -r datapool/paperless@autosnap_2226-06-01_00:00:00_daily
sudo zfs rollback -r datapool/odoo@autosnap_2226-06-01_00:00:00_daily

# 5. Start databases and services
sudo systemctl start postgresql
# Allow database to boot, then start applications:
sudo systemctl start phpfpm-nextcloud paperless-web odoo hr-system invoiceninja

5. DNS & Subdomain Routing

Rationale: Reverse proxying traffic exclusively over Tailscale rather than port forwarding on a public IP keeps the office-server invisible to the public internet and automated Shodan scanners.

Configure access via subdomains of your domain (e.g., yourdomain.com) routed through Nginx over Tailscale.

1. Domain Registrar Setup

Create a Wildcard A Record pointing to your office-server's private Tailscale IP (e.g. 100.64.1.20):

TypeHostValueTTL
A Record*100.64.1.20Automatic

2. Nginx Reverse Proxy (services.nix)

nix
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;

    virtualHosts = {
      "yourdomain.com" = {
        locations."/" = {
          proxyPass = "http://127.0.0.1:8082"; # Homepage Dashboard
        };
      };
      "nextcloud.yourdomain.com" = {
        locations."/" = {
          proxyPass = "http://127.0.0.1:80";
        };
      };
      "invoicing.yourdomain.com" = {
        locations."/" = {
          proxyPass = "http://127.0.0.1:8000";
          proxyWebsockets = true;
        };
      };
      "kuma.yourdomain.com" = {
        locations."/" = {
          proxyPass = "http://127.0.0.1:3001";
          proxyWebsockets = true;
        };
      };
    };
  };

6. Maintenance & Automated Security

Routine system operations are declared declaratively to run automatically:

  • Intrusion Detection System (IDS): Both suricata (Network IDS) and wazuh-agent (Host IDS) are deployed natively via Nixpkgs on the office-server. They monitor local file integrity and network traffic, reporting alerts back to a Wazuh Manager hosted securely on an isolated cloud instance, ensuring attackers cannot tamper with the logs.
  • Automatic Weekly Scrubs: ZFS performs automated read checking and bitrot scrubbing.
  • Monthly Garbage Collection: Nix store packages older than 14 days are deleted to clean up drive space.
  • Weekly Store Optimization: Finds and hard-links duplicate packages weekly to maximize storage space.
  • Daily Audit Logs: Scans ZFS status, package vulnerabilities (CVEs), and NixOS Discourse RSS feed, pushing warnings via Apprise.

7. Declarative System Configuration

Rationale: NixOS enables complete infrastructure-as-code. Rather than manually configuring services, the entire operating system, including the IDS agents, reverse proxies, and networking rules, is defined declaratively. Below is a simplified example of how this is structured using Flakes.

1. flake.nix

nix
{
  description = "Enterprise Office Server Configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs, ... }: {
    nixosConfigurations.office-server-metal = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./hardware-metal.nix
        ./configuration.nix
      ];
    };
  };
}

2. configuration.nix (Snippet)

nix
{ config, pkgs, ... }:

{
  # Bootloader & ZFS
  boot.loader.systemd-boot.enable = true;
  boot.supportedFilesystems = [ "zfs" ];
  networking.hostId = "a1b2c3d4";

  # Security & IDS
  services.suricata = {
    enable = true;
    interface = "eth0";
  };

  services.wazuh.agent = {
    enable = true;
    serverAddress = "siem.yourdomain.com";
  };

  # Essential Packages
  environment.systemPackages = with pkgs; [
    zfs cryptsetup htop git
  ];

  system.stateVersion = "24.05";
}
💬