Compiling a Custom FreeBSD Kernel¶

A complete walkthrough for building and installing a custom kernel on FreeBSD, using the native buildworld/buildkernel toolchain and, where available, ZFS boot environments for safe rollback.

1. Overview and prerequisites¶

FreeBSD builds the kernel and the base userland ("world") from one unified source tree rather than treating the kernel as a standalone project. The standard flow is:

buildworld -> buildkernel -> installkernel -> installworld -> mergemaster (or etcupdate)

buildworld compiles the entire base system (libc, compiler toolchain, core utilities) using itself, which is also what makes cross-version upgrades safe - you always build with a toolchain compatible with the source you're building. buildkernel then compiles the kernel using the freshly built tools.

Budget: 30 GB+ of free space for /usr/src and /usr/obj, and 30 minutes to a few hours of CPU time depending on hardware and whether you run a full buildworld or just rebuild the kernel.

You need:

• The base system's own toolchain (clang/lld, included by default) - no extra compiler packages needed for a native build.
• Source tree checked out under /usr/src (Section 2).
• Root access for installkernel/installworld; the build steps themselves (buildworld, buildkernel) should be run as an unprivileged user.
• If you're only rebuilding the kernel (not the whole world) after a small config tweak, you can often skip buildworld - see Section 8 for when that's safe.

2. Obtaining source¶

Modern FreeBSD (12+) tracks source in Git rather than Subversion.

Clone the source tree:

git clone https://git.FreeBSD.org/src.git /usr/src
cd /usr/src
git checkout releng/14.1     # a release branch, or:
git checkout main            # HEAD/current

Or update an existing checkout:

cd /usr/src
git pull

Legacy Subversion checkouts (older systems still configured this way):

svnlite checkout https://svn.freebsd.org/base/releng/14.1 /usr/src

Which branch to track:

• releng/X.Y - a specific release branch, receiving only security and critical fixes. Matches what freebsd-update tracks for that release; the conservative choice for a production machine.
• stable/X - the stable branch for a major version, gets new features backported but stays ABI-stable within that branch.
• main - current development (what will become the next major release); expect occasional breakage, useful for testing new features or hardware support.

Check uname -r against the branch you intend to track so buildworld/buildkernel starts from a consistent base rather than jumping major versions in one step.

3. Kernel configuration file¶

Unlike Linux's .config, FreeBSD kernel options live in a plain-text config file under /usr/src/sys/<arch>/conf/, starting from the GENERIC config every install ships with:

cd /usr/src/sys/amd64/conf
cp GENERIC MYKERNEL

Name it in caps by convention; this name becomes KERNCONF in later build steps. Edit MYKERNEL directly:

# Pull in everything GENERIC has, then layer changes on top
include GENERIC
ident       MYKERNEL

# Remove something you don't need
nooptions   COMPAT_FREEBSD32

# Add a driver or option not in GENERIC
device      if_wg          # WireGuard
options     KDTRACE_HOOKS  # DTrace support

# Disable debugging aids for a leaner/faster production kernel
nooptions   INVARIANTS
nooptions   INVARIANT_SUPPORT
nooptions   WITNESS
nooptions   WITNESS_SKIPSPIN

The include GENERIC pattern (available since FreeBSD 9) is strongly preferred over copying and editing the whole file - your custom config then only has to state deltas, which survive source updates far more cleanly than a full copy that silently drifts from GENERIC over time.

Common things people add/remove:

• Add: device if_wg, options KDTRACE_HOOKS (DTrace), device vmm (bhyve virtualization host), ZFS-related options if not already present.
• Remove for a smaller/faster kernel: INVARIANTS/WITNESS/DDB debugging aids (leave these in if you're troubleshooting a driver or planning to file a bug - they make crash dumps far more useful).
• makeoptions DEBUG=-g plus options KDB and options DDB together give you an interactive kernel debugger on panic - worth keeping on a build you're actively developing against.

4. Building¶

cd /usr/src
make -j$(sysctl -n hw.ncpu) buildworld
make -j$(sysctl -n hw.ncpu) buildkernel KERNCONF=MYKERNEL

buildworld must complete before buildkernel when you're changing branches or doing a significant version jump, since buildkernel relies on the toolchain buildworld just produced. For a same-version kernel-only tweak (just editing MYKERNEL, not moving to new source), you can often run buildkernel alone - see Section 8 for the exact conditions where that's safe.

Tuning the build with /etc/src.conf and /etc/make.conf:

# /etc/src.conf - controls what gets built
WITHOUT_TESTS=yes
WITHOUT_DEBUG_FILES=yes
WITH_CCACHE_BUILD=yes

# /etc/make.conf - compiler/linker flags, global build options
CFLAGS+= -O2 -pipe

src.conf toggles which subsystems/components are built at all (tests, docs, debug symbols); make.conf carries compiler flags and other make-level tuning that applies more broadly, including to ports if you build those too.

• -j$(sysctl -n hw.ncpu) parallelizes the same way nproc does on Linux.
• A full buildworld typically takes 30 minutes to a couple of hours depending on hardware; buildkernel alone is much faster, usually a few minutes.
• Logs go to your terminal by default; redirect to a file (... 2>&1 | tee build.log) for a long unattended build.

5. Installing¶

sudo make installkernel KERNCONF=MYKERNEL
sudo make installworld
sudo mergemaster -Ui          # or: sudo etcupdate

Order matters: install the kernel first, reboot into it, then install world - this is FreeBSD's documented upgrade order because a newer kernel is generally compatible with an older userland (for one boot), but not reliably the reverse. For a same-version kernel-only change, you can skip installworld/mergemaster entirely and just install and boot the new kernel.

mergemaster -Ui compares your /etc against the just-built template /etc, showing a diff for every changed file and letting you merge, keep yours, or take the new one interactively. etcupdate does the same job with a more automated three-way merge and less interactive prompting - many admins prefer it for routine upgrades and fall back to mergemaster only when a merge needs manual judgment.

Skipping this step after a version bump is a common source of a system that boots but has broken /etc/rc.d scripts or mismatched config file formats - don't skip it on anything but a pure kernel-config tweak with no world rebuild.

6. Rebooting and boot environments¶

sudo reboot

If your root filesystem is ZFS (the default on most modern installers), take a boot environment snapshot before installing anything new - this is FreeBSD's equivalent of a Btrfs/LVM snapshot rollback, and it's the single best safety net for kernel upgrades:

sudo bectl create pre-custom-kernel     # snapshot the current, working BE
sudo make installkernel KERNCONF=MYKERNEL
sudo make installworld
sudo reboot

If the new kernel or world is broken, select the old boot environment from the loader menu at boot (it lists available BEs automatically on ZFS-root systems), or roll back explicitly:

sudo bectl list                          # see available boot environments
sudo bectl activate pre-custom-kernel    # make it the default for next boot
sudo reboot

On a UFS root (no ZFS), there's no equivalent snapshot mechanism built in - your safety net is keeping the old /boot/kernel around as /boot/kernel.old (FreeBSD does this automatically on installkernel unless you pass -DINSTALLKERNEL overrides) and manually restoring it if needed.

7. Verifying and troubleshooting¶

uname -a                    # confirm KERNCONF ident and build timestamp
dmesg | grep -i error       # scan for driver/hardware issues
kldstat                     # loaded kernel modules
zpool status                # if using ZFS, confirm pools are healthy
service -e                  # services enabled to start (sanity check after mergemaster)

If you built with options KDB and options DDB, a panic drops you into an interactive debugger (db> prompt) instead of just rebooting - bt for a backtrace is the first thing to run there before you reboot and lose the state.

8. Tips for repeat builds¶

When you can skip buildworld: if you're only editing MYKERNEL (adding/removing a driver or option) and haven't updated the source tree or changed branches, make buildkernel KERNCONF=MYKERNEL alone against your existing /usr/obj toolchain is generally safe and much faster. Any time you git pull/git checkout a different point in source history, do a full buildworld first.

Speeding up iteration:

# Enable ccache once, in /etc/src.conf
WITH_CCACHE_BUILD=yes

# Rebuild and install just the kernel after a config-only change
sudo make buildkernel KERNCONF=MYKERNEL
sudo make installkernel KERNCONF=MYKERNEL

Keeping source current:

cd /usr/src
git pull
# then a full buildworld/buildkernel/installkernel/installworld/mergemaster cycle

Diffing your config against GENERIC to see your full set of deltas at a glance:

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/MYKERNEL

(Trivial when you used include GENERIC - the diff is exactly your intentional changes, nothing more.)

Cross-building for another architecture (e.g. building an arm64 kernel from an amd64 host) uses the same buildworld/buildkernel targets with TARGET/TARGET_ARCH set:

make TARGET=arm64 TARGET_ARCH=aarch64 buildworld
make TARGET=arm64 TARGET_ARCH=aarch64 buildkernel KERNCONF=GENERIC

Useful for building images for Raspberry Pi or other arm64 boards from a faster amd64 build machine, then transferring the result rather than compiling natively on the slower target.