umbrelOS on systemd-nspawn: the setup story

Why nspawn, not a VM

The goal was to run umbrelOS — Umbrel's full x86 OS image, normally installed on bare metal or a VM — without dedicating a whole virtual machine to it. systemd-nspawn boots the actual OS (real systemd as PID 1, real Docker, real everything) directly on the host kernel, for a fraction of the overhead of a VM.

The only things this setup actually needed: the dashboard reachable from the host itself, and later an outbound-only Cloudflare Tunnel from inside it. Neither requires VM-grade isolation, which made nspawn the right tool rather than a compromise.

Pulling umbrelOS off the .img

umbrelOS ships as a 20 GB GPT disk image. Loop-mounting it (udisksctl loop-setup, no root needed) showed the layout:

Rather than run the container off the loop device (fragile — tied to the .img staying mounted forever), root slot A and the data partition were rsync'd straight onto the host disk at /var/lib/machines/umbrel. Once that copy finishes, the container has zero runtime dependency on the original .img — it can be deleted, archived, or kept only as a clean "factory reset" source.

The EFI partition was skipped entirely: nspawn execs systemd straight out of the root directory, it never touches a bootloader.

First boot: straight into emergency mode

The first systemd-nspawn -b dropped straight into an emergency shell. The boot log pointed at one real failure: mnt-root.mount, a leftover entry in umbrelOS's own /etc/fstab that bind-mounts / onto /mnt/root — useful on real hardware for Mender's A/B update tooling, meaningless (and fatal) inside a container where / is already the whole world nspawn was given. Since local-fs.target depended on it, one failed mount took the whole boot down with it.

The fix was one line: comment out that fstab entry. mender-client.service was masked too, since a container has no business managing the two root slots.

Everything else — the /home, /kopia, /var/lib/docker, /var/log binds into the data partition — mounted cleanly on the first try. A few cosmetic failures remained (sys-kernel-debug.mount, sys-kernel-tracing.mount, sys-kernel-config.mount, systemd-remount-fs.service) — all expected in a container, since debugfs/tracefs/configfs aren't delegable and there's no real block device to remount.

Getting the network right

This was the long part. Four separate problems stacked on top of each other.

macvlan doesn't work over Wi-Fi. The obvious choice for giving the container its own LAN presence is --network-macvlan, but the only connected interface on this host is wlan0 — and macvlan needs a second MAC address on the wire, which almost no consumer Wi-Fi AP will forward. --network-ipvlan was the fix: it shares the host's MAC and just adds another IP, which Wi-Fi tolerates fine.

NetworkManager can't drive an nspawn-created ipvlan device. Inside the container, iv-wlan0 showed up unmanaged — nspawn creates the interface directly inside the container's network namespace, bypassing the normal udev "device added" event NetworkManager relies on. Forcing it managed via a NetworkManager.conf override got it to disconnected, but activation kept failing: NM's ipvlan device class requires a parent interface to construct the connection, and that parent (wlan0) doesn't exist inside the container's namespace at all — it's not visible across the boundary. NetworkManager was never going to be able to manage this interface.

No DHCP client, so: static IP. With NetworkManager out of the picture and no dhclient/udhcpc installed, the pragmatic fix was a static IP (192.168.1.50/24, gateway 192.168.1.1) applied by a small oneshot systemd unit (iv-wlan0-static.service) inside the container, enabled at boot.

The host couldn't reach its own container. Even with the static IP up and the container reaching the internet fine, the host itself got Destination Host Unreachable pinging it. This is a known ipvlan/macvlan limitation: a host can't talk to its own child interface over the same physical NIC, because that traffic would have to bounce off the Wi-Fi AP and back, and consumer APs don't reflect frames like that. The fix was to give the host a sibling ipvlan interface on wlan0 too (ipvl-shim), route the container's /32 through it, and add a static ARP entry (the shared MAC made ARP resolution over the shim itself fail outright). That let the kernel shortcut host→container traffic internally instead of putting it on the wire. Made persistent via a umbrel-host-access.service unit on the host.

Dashboard's up — what's next

http://192.168.1.50 now loads the umbrelOS first-run wizard, and the account created there works. The container persists on real disk, boots with sudo machinectl start umbrel (passwordless, via a narrow NOPASSWD: /usr/bin/machinectl sudoers rule), and the host can reach it reliably across reboots.

Next up: Umbrel's own Cloudflare Tunnel app, installed and configured from inside the Umbrel dashboard itself, to expose select apps to the internet without opening a single router port. That part is self-serve from here — it only needs outbound access, which the container already has.

Chasing updates: 1.5 → 1.7.4 → 2.0

The container was originally set up from a 1.5.0-era image that predates umbrelOS's hardware-detection features entirely — it just ran Docker and apps against whatever filesystem it was given, no questions asked about the underlying disk. That's why the initial setup went as smoothly as it did: there was no code in it trying to inspect real hardware, so nothing to trip over.

The built-in "Update" button never worked, and never will, in this setup. umbrelOS's system updates run through Mender, which swaps between two disk partitions (A/B slots) and tracks the active one via a boot-record file on the EFI partition. Neither exists in a directory-backed nspawn container — confirmed by watching a real update attempt download a 1.7GB artifact and then fail at the install step with "No match between boot and root partitions." Every future OS version bump means manually repeating what's described below, not clicking a button.

umbrelOS quietly reorganizes its own disk layout between releases, with no announcement. The 1.5.0 image had root directly on partition 2. By 1.7.4, a separate /boot partition had been split off partition 2, and the real root moved to partition 4 — the same shift held for 2.0. The first 1.7.4 update attempt assumed the old layout and silently copied the wrong (now much smaller, boot-only) partition over the container. No harm came of it: the copy step was deliberately built to only add/overwrite files, never delete, so the mistake just left a few unused files (a stray vmlinuz, initrd.img) sitting around — nspawn doesn't read either, since it shares the host kernel directly. The fix going forward: verify the actual root partition by checking for /opt/umbreld inside it, rather than trusting a partition number across releases.

With that fixed, 1.7.4 went in cleanly. The jump straight to 2.0 — umbrelOS's biggest release yet, adding virtual machines ("Machines"), GPU passthrough, and a new Storage Manager with RAID ("FailSafe") — used the exact same re-copy process and landed just as cleanly, confirmed by umbreld itself reporting "version": "2.0.0".

One thing from 2.0 doesn't have a clean fix: the "couldn't check your storage" banner. Storage Manager needs to know which physical disk umbrelOS is installed on, so it can stop you from accidentally adding your system disk to a RAID pool. It finds this by asking the kernel what device backs / — and since this container's root is really just a directory living on the host's actual filesystem, the true answer is the host's real 3.6TB system drive. umbrelOS then tries to inspect that drive directly and can't, because the container was deliberately never given real access to it. Masking /sys/block and /proc/partitions (the sources for general disk enumeration, e.g. what shows up as a candidate for RAID) fixed the endless retry loop for that, and is worth keeping as a safety measure on its own. But this specific banner traces back to the kernel's mount table, not to anything enumerable — there's no masking trick for it, and the only real fix (creating a device node for the host's actual system disk inside the container) is exactly the kind of access this setup exists to avoid granting. It's being left as a known, permanent, cosmetic limitation: it doesn't touch the dashboard, Docker, or any app.

The phantom account lock

After the 2.0 reprovision, the dashboard greeted every visit with "Create account" instead of a login prompt — the account created hours earlier hadn't survived the fresh-install process (see above: /data gets discarded on each reprovision, and that's where accounts live, not just app data). Typing a name and password into the form and clicking Create produced the same "umbrelOS couldn't check your storage" text seen on the dashboard banner — except now it looked like it was blocking account creation entirely, not just decorating a settings screen.

It wasn't. Calling the registration endpoint directly (curl -X POST http://192.168.1.50/trpc/user.register -d '{"name":"...","password":"..."}') returned a clean success. The storage error shown on the account-creation screen turned out to be a UI red herring — the same continuously-failing background poll from the Storage Manager banner, rendered on a screen it has nothing to do with, making a working action look broken.

That diagnostic call had a side effect worth noting: it created a real (throwaway) account, which then blocked the actual intended registration (user.exists() returns true once any account exists). Undoing it took two tries. The first instinct — delete umbrel.db, the SQLite database — was wrong: umbreld doesn't track account existence there at all. Reading user.ts directly showed exists() checks a user key in a separate key-value store, which turned out to be /home/umbrel/umbrel/umbrel.yaml — a plain YAML file with the account's name, language, and bcrypt password hash sitting in it directly. Deleting that specific block (and restarting umbrel.service to pick up the change) reset things cleanly.

One boundary held throughout: account names and passwords, even for this fully local, self-owned server, were always typed by hand — into the browser form directly, or into a terminal command run locally — never passed through chat for automatic entry. A minor inconvenience against a rule that doesn't bend for context.

The real blocker: apps couldn't install at all

With an account finally in place, installing Cloudflare Tunnel failed outright: docker compose ... up --build --detach --remove-orphans exiting 1. The actual error, buried in the log, was about tor_proxy — a small infrastructure container every app's environment depends on, not something specific to Cloudflare Tunnel: error mounting "sysfs" to rootfs at "/sys" ... operation not permitted. Since that shared container is a prerequisite for installing anything, this meant every app in the App Store was broken, not just this one.

The cause traced back to something present in every machinectl status umbrel output all day without being questioned: ID Shift: 599064576. systemd-nspawn defaults to placing containers in a private (UID-shifted) user namespace when launched via machinectl/systemd-nspawn@.service — confirmed in man systemd.nspawn under PrivateUsers=. Docker containers that mount their own private sysfs (which is most of them, not some special case) hit a kernel-level namespace-ownership check that this extra shifted layer breaks.

The fix was one line, on the host side this time, not inside the container: PrivateUsers=no under [Exec] in /etc/systemd/nspawn/umbrel.nspawn, plus a full container restart (the setting only applies at boot). After that: no more ID Shift in the status output, the three sys-kernel-* boot failures that had been written off as "expected in a container" disappeared too (they were the same root cause, not a fundamental limitation — worth remembering next time something looks unfixable), and Cloudflare Tunnel installed cleanly through to a running docker ps with tor_proxy, cloudflared_web_1, and cloudflared_connector_1 all up.

Would I do this again?

A KVM VM with virtio drivers would have sidestepped nearly every real problem above. Mender's A/B update mechanism needs real disk partitions — a VM naturally has them. Storage Manager's "what physical disk am I on" question has a real, ownable answer in a VM. The PrivateUsers nested-sysfs bug that quietly blocked every single app install, not just Cloudflare Tunnel, doesn't exist when Docker gets a genuine kernel instead of one nested inside a shifted user namespace. Those weren't edge cases — they were most of today's actual work.

The one place nspawn stays ahead: storage flexibility. /data is just files on the host's real filesystem, so it grows and shrinks with whatever's actually free on disk — no ceiling to plan around, nothing to resize. A VM can get close with a sparse, generously-sized virtual disk, but that's a deliberate setup choice rather than the default, and it's a promise rather than a guarantee once more than one VM shares the same physical drive.

Knowing what's known now: probably a VM, for anything meant to use Umbrel's own update mechanism, its App Store without caveats, or features like Machines/GPU passthrough down the line. nspawn was the right call only because the actual ask stayed narrow — local dashboard access and one outbound Cloudflare Tunnel — and even then, it took a full day of reverse-engineering umbrelOS's assumptions about owning real hardware to get there cleanly.

Machines, three blockers deep: Windows XP actually boots

The earlier reflection above treated "Machines" (2.0's VM feature) as another wall hit on purpose, alongside GPU passthrough and the storage-check banner. That turned out to be only half right — unlike the mknod device-node restriction (a genuinely hardcoded, undocumented-as-overridable part of nspawn's sandbox contract), none of what blocked Windows XP from starting was a real dead end. Each layer, once found, had a real fix.

Layer one: AppArmor. virsh create failed immediately with "Security driver model 'apparmor' is not available" — unsurprising, since /dev/kvm doesn't exist in the container either and neither does a working AppArmor kernel interface. Setting security_driver = "none" globally in /etc/libvirt/qemu.conf didn't help: the domain XML umbreld generates hardcodes <seclabel model='apparmor'> directly, overriding the global default. That line lives in umbreld's own source (domain.ts:252), not a config file — but since umbreld runs its TypeScript straight through tsx with no build step, deleting that one line and restarting the service was enough to make the global security_driver = "none" setting actually take effect.

Layer two, briefly: a missing storage bind-mount. Manually testing a stripped domain XML hit "Cannot access storage file" next — turned out to be an artifact of testing outside umbreld's own flow, not a real gap. Reading libvirt.ts directly showed umbreld already bind-mounts the persistent disk into the runtime directory itself (mount --bind machineDirectory storageDirectory) as a normal part of starting a machine; it just never got that far while AppArmor was still failing first.

Layer three: read-only sysfs. With both fixed, the VM got further and hit "Unable to set bridge port vnet0 isolated to 1: Read-only file system" — libvirt needs to write a bridge-port attribute under /sys/class/net/<bridge>/brif/<port>/, which resolves through /sys/class into the separately-mounted, read-only /sys/devices. Unlike device-node creation, this one is just a mount flag: mount -o remount,rw on both mountpoints worked immediately, no nspawn-level restriction blocking it. Made permanent as a new boot-time unit, libvirt-sysfs-rw.service.

With all three fixed, Windows XP started through the normal dashboard — not a manual virsh test — and its console showed a real SeaBIOS boot, iPXE, and a disk boot attempt (NTLDR is missing, expected: the actual OS install had been interrupted mid-way by earlier testing, nothing to do with the fixes themselves). The domain runs type='qemu' with -accel tcg, pure software emulation — no /dev/kvm passthrough needed at all for a guest as old and light as XP.

Worth carrying forward: not every wall in this setup is the same kind of wall. Some (device-node creation) are load-bearing parts of what nspawn is. Others (AppArmor, sysfs permissions) just look that way until you actually read the error and go looking for where it's coming from.