FreeBSD and ZFS SysAdmin Lab: A Hands-On Skills Guide¶

This guide assumes a strong Linux background and works from what you already know. FreeBSD is a different operating system, not a Linux distro, so the goal is to build muscle memory in the areas where it differs, and to produce evidence (repos, ports, patches, write-ups) that proves it.

Existing container, Kubernetes, CI/CD, and scripting skills carry over directly. Spend your time on FreeBSD internals, ZFS depth, ports and packaging, and upstream contribution.

0. Set Up Your Lab¶

You need somewhere you can break things.

• Primary lab: a spare box or a Proxmox/KVM host with 16 GB+ RAM. Two physical disks are enough. Use them as a mirror (so you can practice real disk replacement, resilvers, and SMART on actual hardware) and do all multi-disk topology work with virtual or file-backed disks (see below). SSDs are fine.
• What two physical disks cover well: ZFS-on-root mirror, real zpool replace/detach/attach drills, resilver timing, SMART monitoring, gpart labelling, and boot environments.
• What needs virtual or file-backed disks instead: raidz1/2/3, dRAID, stripes of many vdevs, special vdevs, and SLOG/L2ARC experiments. The logic and failure behavior are identical, so the practice still counts. Only the performance numbers differ, so treat fio results from these as comparisons between layouts, not real throughput.
• Need a third or fourth real disk? A USB-to-SATA dock or enclosure works for lab purposes. Avoid relying on USB bridges for SMART data or for anything you would run in production, since some hide SMART or reset under load.
• Nested virtualization: if running FreeBSD as a guest and you want to practice bhyve inside it, enable nested virt on the host (kvm_intel nested=1 or the AMD equivalent) and use a CPU type of host.
• Disks for ZFS: attach at least 6 small virtual disks (2–10 GB each) to one VM so you can build mirrors, raidz, and dRAID pools and rip disks out.
• No spare disks at all? Use file-backed vdevs on a FreeBSD host (sparse files, so they cost almost nothing):

sh for i in 1 2 3 4 5 6; do truncate -s 2G /var/tmp/vdisk$i; done zpool create lab raidz2 /var/tmp/vdisk1 /var/tmp/vdisk2 /var/tmp/vdisk3 \ /var/tmp/vdisk4 /var/tmp/vdisk5 /var/tmp/vdisk6 zpool offline lab /var/tmp/vdisk3 # simulate a failed disk zpool destroy lab # clean up when done

Put the files on a filesystem that is not the pool they back, and don't use file-backed pools for anything you care about. - Snapshot the VM itself before risky experiments. That gives you two layers of undo: the hypervisor and ZFS boot environments. - Keep a lab journal (a Git repo of Markdown notes). Every problem you solve becomes documentation, which doubles as a reference you can reuse later.

Run FreeBSD RELEASE (currently the 14.x or 15.x line; check freebsd.org) for stability, and a STABLE or CURRENT VM later for ports and contribution work.

Phase 1: FreeBSD Fundamentals (Weeks 1–3)¶

1.1 Install and Orient¶

1. Install using bsdinstall with ZFS on root (auto or manual). Do it at least three times: once with the defaults, once with a mirrored root, and once with GELI-encrypted root.
2. Read the layout. Understand the base system vs. /usr/local split. Base lives in /, /usr, and /etc; third-party software and its config live under /usr/local (/usr/local/etc).
3. Learn the config model: - /etc/rc.conf (and /etc/rc.conf.d/) instead of systemd units - sysrc to edit it safely - service <name> start|stop|status|onestart - /etc/sysctl.conf and /boot/loader.conf (tunables set at boot) - /etc/periodic.conf and periodic(8) daily/weekly/monthly scripts
4. Read rc.d scripts in /etc/rc.d and /usr/local/etc/rc.d, then write your own rc script for a simple daemon using rc.subr and daemon(8).

1.2 Package, Update, and Upgrade Workflows¶

• pkg install / upgrade / audit -F / lock / autoremove / info / which
• freebsd-update fetch install and freebsd-update upgrade -r <release> for base updates
• Boot environments with bectl: create a BE before every upgrade, roll back on failure, and practice booting a broken BE from the loader menu
• Practice a full major-version upgrade (e.g., 14.x → 15.x) on a VM, including reinstalling packages against the new ABI

1.3 Linux-to-FreeBSD Translation Cheat Sheet¶

1.4 Jails (Core Skill)¶

Build up in this order:

1. Manual jail using jail.conf and a bsdinstall jail or base.txz extraction into a ZFS dataset. Understand jail.conf syntax, exec.start, mount.devfs, and allow.* parameters.
2. Thin jails and templates using ZFS clones (zfs clone from a template snapshot).
3. VNET jails with epair interfaces and a bridge, each jail with its own network stack and its own pf rules.
4. Resource limits via rctl (memory, CPU, process counts).
5. Jail managers: try Bastille and iocage, then decide why you'd pick a manager vs. plain jail.conf. Understanding the primitives pays off when something breaks.
6. Advanced: jail -m live modification, nullfs mounts, per-jail ZFS datasets with zfs jail, and upgrading a jail's userland.
7. OCI/Podman on FreeBSD (ocijail) as a bridge to your container background.

Exercise: Build a three-tier stack (nginx jail, app jail, PostgreSQL jail) on a private bridge with pf NAT and rdr, each jail on its own ZFS dataset with quotas, snapshotting and replicating them independently.

1.5 Networking Fundamentals on FreeBSD¶

• Configure interfaces, aliases, VLANs, LAGG (LACP/failover), bridges, and tap/epair
• Firewalls: write a real pf.conf with tables, anchors, macros, NAT, rdr, scrub, and rate limiting (max-src-conn). Then implement the same policy in ipfw to understand both.
• Routing: static routes, FIBs (setfib), and a dynamic routing daemon (FRR or BIRD from ports) for OSPF/BGP basics
• VPNs: WireGuard (in-kernel on recent releases), OpenVPN, and IPsec (strongSwan / racoon-era vs. modern). Build a site-to-site tunnel between two VMs.
• Debugging tools: tcpdump, sockstat -4l, netstat -s, pfctl -sr -vv, pflog, dtrace, and ngctl (netgraph) for exposure
• Services: unbound, nsd/knot, isc-dhcpd or kea, NTP (ntpd/chrony)

Phase 2: ZFS Mastery (Weeks 3–6)¶

This is the heart of the lab, so go past "I've made a pool."

2.1 Core Concepts to Truly Understand¶

• Pools → vdevs → datasets/zvols; how redundancy lives at the vdev level
• Copy-on-write, transaction groups (TXGs), the ZIL and SLOG, ARC/L2ARC
• ashift, recordsize, compression (lz4/zstd), atime, xattr, dnodesize, sync, logbias, primarycache, special_small_blocks
• Checksums, self-healing, scrub vs. resilver, and why RAID-Z resilver differs from mirror resilver

2.2 Structured Exercises (Use File-Backed or Virtual Disks)¶

1. Layouts: build stripe, mirror, raidz1/2/3, and dRAID pools. Compare capacity, IOPS, and fault tolerance with fio.
2. Failure drills (do these repeatedly until they are routine): - Offline and replace a disk (zpool replace, zpool detach/attach) - Corrupt a disk with dd and watch a scrub repair it - Fill a pool to 90%+ and observe the performance cliff - Import a pool from a "dead" system (zpool import -f, -d, -o readonly=on, -N) - Recover with zpool import -F and -T (rewind TXG) after simulated unclean shutdown - Handle a full pool that won't delete files (reserved space, refreservation)
3. Snapshots and clones: holds (zfs hold), rollbacks, clone promotion, zfs diff, and snapshot naming schemes
4. Replication: - Hand-roll zfs send | ssh ... zfs recv full and incremental (-i, -I) - Use -R, -w (raw encrypted), -c (compressed), -L, and resumable sends (recv -s, send -t) - Automate with sanoid/syncoid and zrepl, and understand their trade-offs - Intentionally break a replication chain (delete a common snapshot) and recover
5. Encryption: native ZFS encryption, key loading, raw sends, and keylocation strategies
6. Delegation and quotas: zfs allow, user/group quotas, refquota vs. quota, and per-jail datasets

2.3 Tuning and Troubleshooting¶

• Workload-specific tuning:
• Databases (PostgreSQL 8K recordsize, MySQL/InnoDB 16K, logbias=throughput)
• VM zvols (volblocksize, sync)
• Media/backup (recordsize=1M)
• NFS/SMB serving
• ARC: vfs.zfs.arc.max (sysctl names vary by release, so check sysctl -a | grep zfs), reading arc_summary / sysctl kstat.zfs.misc.arcstats
• Observability:
• zpool iostat -v -l -w 1
• zpool status -v (learn every field)
• gstat -p, zfs get all, and zdb for deeper inspection
• dtrace and procstat -kk for hung I/O
• Reading zpool events -v
• Read this class of problems until you can diagnose them from symptoms: slow scrubs, resilver storms, fragmentation, dedup table memory blowups (and why to avoid dedup), and misaligned ashift.

2.4 Large-Scale Storage¶

• Special vdevs (metadata/small blocks on NVMe), dRAID with distributed spares, SLOG sizing and PLP considerations
• fio methodology: sequential vs. random, queue depth, sync writes
• Export ZFS via NFS, iSCSI (ctld), and SMB, and know where each hurts performance
• SMART monitoring (smartmontools), drive naming with gpart labels, and GPT/glabel conventions for maintainable pools

Phase 3: Ports, Packages, and Compiler Errors (Weeks 5–8)¶

This is the most distinctive skill area and the one many Linux admins have never touched. Treat it as a craft.

3.1 Learn the Ports Framework¶

1. Clone the ports tree with Git (git clone https://git.FreeBSD.org/ports.git /usr/ports).
2. Read the Porter's Handbook end to end; it is the definitive reference. Focus on USES, OPTIONS, LIB_DEPENDS/BUILD_DEPENDS/RUN_DEPENDS, PLIST, pkg-descr, staging (DESTDIR/STAGEDIR), MASTER_SITES, USE_GITHUB, and distinfo.
3. Study Mk/bsd.port.mk and Mk/Uses/*.mk to see how the framework works.
4. Look at the ports of software you know well (nginx, postgresql, node_exporter) to see idioms.

3.2 Build a Real Build Pipeline with Poudriere¶

• Install poudriere; create a jail (poudriere jail -c), a ports tree, and a bulk build list
• Build packages for multiple FreeBSD versions and architectures (including cross-builds via QEMU user-mode if you want to push it)
• Use poudriere testport to validate ports the way committers do
• Serve the resulting package repository over HTTP and point a client at it; sign it with a repo key
• Learn ccache, tmpfs, and ZFS-backed builds for speed

3.3 Port-Making Progression¶

1. Update an existing port to a newer upstream release (make makesum, fix PLIST, test, portlint -AC).
2. Create a new port for a small C utility, then a Go, Rust, or Python-based tool (use USES=cargo, USE_GITHUB, GH_TUPLE, and USES=python framework helpers).
3. Patch a port: use make patch, edit, and make makepatch to generate clean files/patch-*.
4. Add options and conditional dependencies, and test all option combinations.
5. Run the quality tools: portlint, portfmt, stage-qa, check-plist, and poudriere testport on each of your ports.

3.4 Fixing Compiler and Build Errors (Practice Deliberately)¶

Grab failing ports from the FreeBSD package build logs (pkg-status.freebsd.org) and from Bugzilla, and try to fix them. Common failure classes to learn:

• Newer Clang strictness: implicit function declarations, incompatible pointer types, -Werror promotions
• -fno-common / duplicate symbol issues
• Linux-isms: <malloc.h>, /proc, epoll, sysinfo, GNU extensions, #include <endian.h> vs. <sys/endian.h>
• Linker problems: missing -lm, -lpthread, -lexecinfo, RPATH/LDFLAGS, USES=localbase
• Build-system quirks: autotools (USES=autoreconf libtool), CMake, Meson, and hardcoded /usr paths
• Python/Ruby/Node ecosystem wheels that need USES= helpers
• Toolchain choices: USES=compiler:c++17-lang, USE_GCC, CFLAGS sanitization

Method: reproduce in poudriere, read the first error, minimize, patch, test on both amd64 and aarch64 if possible, and send the patch upstream and to the ports tree.

3.5 Submit Work¶

• File ports PRs via FreeBSD Bugzilla (check the current contribution docs on freebsd.org for the exact workflow, as processes evolve)
• Respond to committer review promptly and gracefully
• Aim for a track record: 5–10 accepted updates or fixes are a strong, verifiable record of the skill

Phase 4: Virtualization (Weeks 7–9)¶

• bhyve: run FreeBSD and Linux guests, first raw (bhyve + bhyvectl), then via vm-bhyve. Use ZFS zvols for disks, tap/bridge networking, UEFI (bhyve-firmware), and VNC console. Learn PCI passthrough (ppt) and cloud-init-style provisioning.
• Snapshot/replicate VM zvols and restore them on another host, which ties directly into your ZFS replication skills.
• Proxmox: since you have KVM familiarity, set up Proxmox with ZFS storage, replicate VM disks between nodes, and run FreeBSD guests (install qemu-guest-agent, VirtIO drivers).
• KVM/libvirt on Linux: be able to compare features and explain when you'd choose bhyve over KVM (integrated ZFS, jails alongside VMs, simplicity) and when you wouldn't.

Phase 5: Monitoring, Availability, and Security (Weeks 8–10)¶

Monitoring¶

• Prometheus: install node_exporter from ports and enable ZFS collectors (check FreeBSD collector support), plus smartctl_exporter and blackbox_exporter. Build Grafana dashboards for pool health, ARC hit rate, scrub status, and latency.
• Alerting on what matters for ZFS: DEGRADED/FAULTED pools, checksum errors, scrub age, capacity >80%, replication lag, and SMART pre-fail.
• Nagios/Icinga and Zabbix: stand up each once. Use check_zpool-style plugins in Nagios and Zabbix agent items/triggers for ZFS. Being able to work with all three is useful, since environments differ.
• FreeBSD-native signals: periodic daily output emails, zfsd (for automatic hot-spare handling), devd hooks, and syslog forwarding.

High Availability¶

• CARP + pfsync for failover firewalls/load balancers
• ZFS replication + scripted failover (a poor man's HA storage pair), including split-brain avoidance and fencing thinking
• HAST (know that it exists and its trade-offs) and shared-nothing designs
• NFS/iSCSI over a failover VIP; discuss failure modes in detail

Security¶

• pkg audit, vuxml, and a patch cadence (freebsd-update cron)
• SSH hardening, securelevel, sysctl hardening, mac_* frameworks, auditd
• Jail isolation and pf egress control
• Understanding procstat, ktrace/kdump, and truss for incident forensics

Phase 6: Automation and Documentation (Continuous)¶

• Shell: write portable /bin/sh (not bash-only) scripts. Run shellcheck, and know BSD vs. GNU tool differences.
• Python: system tooling on FreeBSD (py311-* packages, subprocess, psutil platform gaps).
• Ansible on FreeBSD: set ansible_python_interpreter=/usr/local/bin/python3, use community.general.pkgng, sysrc, ansible.posix, jail and ZFS modules (community.general.zfs), and write roles that are idempotent on FreeBSD. Build one role set that provisions a complete jail host with pf, monitoring, and ZFS replication.
• Custom tooling: a small Rust, Go, or shell utility that audits pools, checks replication lag, or reports outdated packages is a good project. Keep an Ansible/shell baseline too, since those are the common denominators.
• Document as you go: runbooks for "replace a failed disk," "recover a pool," "upgrade a jail host," and "restore from replication." Treat documentation as a first-class deliverable.

Phase 7: Community and Upstream Contribution (Weeks 9–12 and Ongoing)¶

• Contribute upstream:
• Start with documentation fixes, man page corrections, and ports updates
• Triage bugs on Bugzilla and reproduce them
• Review the OpenZFS repo's "good first issue" items, read ZFS-related commit history, and test patches
• Read the Committer's Guide and Developer's Handbook on freebsd.org
• Community presence: subscribe to freebsd-ports, freebsd-stable, and freebsd-fs; join the forums and Discord/IRC; watch BSDCan and EuroBSDCon talks.
• Containers/orchestration: you likely have Docker/K8s covered. Add the FreeBSD angle (Podman + ocijail, Linux jails via linuxulator, and where Kubernetes fits, or doesn't).
• CI/CD: run FreeBSD builders in GitLab CI (with a FreeBSD runner) or Cirrus CI (which supports FreeBSD natively), and automate poudriere builds or port linting on commits.

Suggested 12-Week Schedule¶

Capstone Projects (Pick Two or Three)¶

1. "Lab in a box": a FreeBSD host that runs jails and bhyve VMs on ZFS, replicates to a second host every 15 minutes, is monitored by Prometheus with ZFS-specific alerts, and is fully provisioned by Ansible.
2. Private package repository: poudriere builds for two FreeBSD releases with custom-patched ports, signed and served over HTTPS, with CI triggering rebuilds.
3. Disaster recovery demo: documented, timed recovery of a "destroyed" primary from replicated snapshots, including a VM and a jail, with an RTO/RPO write-up.
4. Fix-a-failing-port sprint: clear 10 failures from the package build logs and get them merged.

Resources¶

• FreeBSD Handbook and man pages (read man 8 jail, man 5 pf.conf, man 8 zfs, man 8 zpool, man 7 ports)
• FreeBSD Porter's Handbook, Developer's Handbook, and Committer's Guide
• Michael W. Lucas: FreeBSD Mastery: ZFS, FreeBSD Mastery: Advanced ZFS, FreeBSD Mastery: Jails, and Absolute FreeBSD
• OpenZFS documentation (openzfs.github.io) and the OpenZFS developer talks
• The FreeBSD Journal and vendor engineering blogs (for example from FreeBSD/OpenZFS consultancies), which show real-world problems and fixes
• BSDCan / EuroBSDCon / OpenZFS Developer Summit video archives
• FreeBSD Bugzilla and pkg-status.freebsd.org for real, live practice targets

Self-Assessment Checklist¶

You are ready to move on when you can answer these with a real example from your lab:

• "A pool shows DEGRADED with checksum errors on one disk. Walk me through triage and repair."
• "Replication has stalled; how do you find out why and resume it?"
• "How would you tune ZFS for a PostgreSQL workload versus a VM datastore?"
• "A port fails with an implicit function declaration under Clang. What do you do?"
• "Explain how you'd upgrade a fleet of jail hosts with minimal downtime and a rollback path."
• "Compare bhyve, KVM, and jails, and when you'd use each."
• "Point me to a patch or PR you've submitted."

The best evidence of skill is public, verifiable work: accepted ports changes, a lab write-up, or a well-documented repo. Even two or three of those show you can do the work, not just read about it.