Architecture overview
Management server runs Proxy-DHCP + TFTP (dnsmasq) and NFS root exports so nodes boot completely without MicroSD cards.
┌─ Management server ─────────────────────────────────────────┐
│ dnsmasq → Proxy-DHCP (no IP ownership) + TFTP │
│ tftp-root = /srv/netboot/tftpboot/<SERIAL>/ │
│ nfs-kernel-server │
│ /srv/netboot/nfs/base-image (golden RO template) │
│ /srv/netboot/nfs/nodes/<SERIAL> (RW root per node) │
└───────────────────────┬─────────────────────────────────────┘
│ LAN
┌────────────┼────────────┐
▼ ▼ ▼
Pi 4/5 Pi 4/5 Pi 4/5
BOOT_ORDER 0xf21 / 0xf241 EEPROM
root=/dev/nfs nfsroot=SERVER:/…/nodes/SERIAL,vers=3,rw
ip=dhcp rootwait
| Component | Role | Path / setting |
|---|
dnsmasq | Proxy-DHCP + TFTP | /srv/netboot/tftpboot, dhcp-range=SERVER,proxy |
nfs-kernel-server | Per-node root RW | /srv/netboot/nfs/nodes/<SERIAL> |
| Golden image | Clone source | /srv/netboot/nfs/base-image (64-bit Lite) |
| Node EEPROM | Network boot first | BOOT_ORDER=0xf21 (Pi4) / 0xf241 (Pi5) |
| cmdline | NFS root | root=/dev/nfs nfsroot=…,vers=3,rw ip=dhcp rootwait |
| fstab | No local disks | /tmp, /var/log → tmpfs |
Proxy-DHCP means the office router still owns IP/gateway; dnsmasq only answers PXE/TFTP — no dual-DHCP fight.
Automation — pifleet-netbootctl
Idempotent Bash CLI: server setup, golden image import, per-serial lifecycle. Downloads are linked above.
Idempotent
Inventory TSV
MAC → serial
exports.d
Quick start on the netboot host
curl -fsSL -o /usr/local/sbin/pifleet-netbootctl \
https://r2d2.fmedia.co.za/dashboard/assets/netboot/scripts/pifleet-netbootctl
chmod 755 /usr/local/sbin/pifleet-netbootctl
export SERVER_IP=192.168.10.10
sudo -E pifleet-netbootctl setup-server
sudo pifleet-netbootctl import-base /path/to/raspios_lite_arm64.img.xz
sudo install -m 600 ~/.ssh/id_ed25519.pub /srv/netboot/keys/authorized_keys
sudo pifleet-netbootctl add-node 10000000a1b2c3d4 kiosk-01 aa:bb:cc:dd:ee:01 192.168.10.101 pi4
sudo pifleet-netbootctl list
sudo pifleet-netbootctl status
| Command | Effect |
|---|
setup-server | Packages, dirs, sysctl, dnsmasq proxy, NFS threads |
import-base IMAGE | Unpack img/xz → base-image + diskless fstab/identity unit |
add-node SERIAL HOST MAC [IP] [MODEL] | Clone rootfs, TFTP, export, dnsmasq, inventory |
sync-node SERIAL | Resync from golden; keep /etc/pifleet/identity |
remove-node SERIAL | Drop boot services; FORCE=1 deletes rootfs |
rebuild-exports / rebuild-dnsmasq | Regenerate from inventory |
Generated cmdline.txt
console=serial0,115200 console=tty1 root=/dev/nfs \
nfsroot=SERVER:/srv/netboot/nfs/nodes/SERIAL,vers=3,tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noatime \
rw ip=dhcp rootwait elevator=deadline cgroup_enable=memory
One-time EEPROM (from temporary SD)
curl -fsSL https://r2d2.fmedia.co.za/dashboard/assets/netboot/scripts/set-pi-network-boot.sh | sudo bash
sudo reboot
Edge-case hardening
Mass cold boots stress DHCP latency, TFTP, NFS, ARP, and 1GbE. These mitigations ship with setup-server.
| Failure | Symptom | Fix |
|---|
| NFS hang | server not responding | NFSv3 hard mounts; nfsd threads=32; UDP off; large r/wsize |
| ARP overflow | Flaky reachability | neigh gc_thresh 4096/8192/16384 |
| ProxyDHCP race | PXE timeout | EEPROM DHCP_TIMEOUT=60; proxy-only dnsmasq |
| TFTP storm | Slow boot | Lean per-serial trees; same L2 as server |
| 1GbE saturation | Long first boots | Fully staged golden image; no mass apt |
| Writeback storms | Server load spikes | Node /var/log tmpfs; dirty_ratio tuning |
| Clone split-brain | Duplicate host keys | Per-node identity seeder; preserve on sync |
| ESTALE | Stale file handles | Stable fsid per serial; reboot after sync |
Server sysctl highlights
net.core.rmem_max / wmem_max = 16777216
net.ipv4.neigh.default.gc_thresh1/2/3 = 4096 / 8192 / 16384
sunrpc.tcp_slot_table_entries = 128
vm.swappiness = 10
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
nfsroot flags
vers=3,tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noatime
Never put NFS root across Wi‑Fi. Keep netboot on a quiet management VLAN.
Node identity & security
Bound to SoC serial (TFTP + NFS path) and Ethernet MAC (inventory). Secrets are injected into the NFS root before first boot — not via open TFTP.
- EEPROM network boot → DHCP (router) + ProxyDHCP → TFTP
/srv/netboot/tftpboot/<SERIAL>/
- Kernel mounts
nfsroot=…/nodes/<SERIAL>
pifleet-identity.service applies hostname, SSH keys, optional static IP- Password SSH disabled in golden image; enroll PiFleet agent after first boot
Identity files per node
/etc/pifleet/identity/hostname
/etc/pifleet/identity/mac
/etc/pifleet/identity/ip
/etc/pifleet/identity/ssh/authorized_keys
/etc/pifleet/identity/eth0.network # if static IP given
MAC ↔ serial
# inventory/nodes.tsv
SERIAL HOSTNAME MAC IP MODEL
# dnsmasq
dhcp-host=aa:bb:cc:dd:ee:01,set:pi_10000000a1b2c3d4
# TFTP convenience
/srv/netboot/tftpboot/aabbccddeeee01 → SERIAL
- TFTP is unauthenticated — management VLAN only.
- NFS no_root_squash required for rootfs; export only trusted subnets.
- No private keys in TFTP — public authorized_keys only inside NFS roots.
- Prefer router DHCP reservations for stable IPs with ProxyDHCP.
Checklist: setup-server → import-base → plant SSH keys → add-node → set EEPROM → boot without SD → dashboard online.
