Building a Container by Hand

Goal

Wanted to build a container from scratch, namespaces + overlayfs + cgroups + veth, zero docker/podman involved, then get hermes gateway run running inside it.

Problem is my host's macOS (Darwin), and it doesn't have namespaces/cgroups/overlayfs natively. So a Linux VM was unavoidable.

Why a VM at all? Containers aren't really a separate technology, they're just a bundle of Linux kernel features (namespaces, cgroups, overlay filesystems). macOS runs Darwin, which simply doesn't have these. Docker Desktop on Mac secretly spins up a tiny Linux VM under the hood for this exact reason, I'm just doing that step myself, visibly, with lima.

Setup

1. VM tool

Went with lima (brew install lima), picked it over colima/UTM/vagrant since it's a lightweight, CLI-driven Linux VM.

What/why: lima boots a small real Linux VM on your Mac and hands you an SSH-like shell into it (limactl shell). Everything from here on happens inside that Linux VM, not on the Mac directly, so your Mac's own ps/ip/mount commands are completely irrelevant here, only the VM's copies matter.

2. Start Alpine VM (not Ubuntu — smaller: ~100MB vs ~600MB download)

limactl start --name=raw-container template:alpine

Note: template://alpine is deprecated syntax as of Lima v2.0, use template:alpine.
Chose "Proceed with the current configuration" at the prompt.
arch=aarch64 (Apple Silicon), image: nocloud_alpine-3.23.4-aarch64-uefi-cloudinit-r0.qcow2

What/why: This downloads and boots a minimal Alpine Linux VM. "aarch64" = ARM64, matching Apple Silicon, the VM's CPU architecture always matches the host chip, not something I chose, lima just picks it automatically. Went with Alpine only because it's small/fast to download, any Linux distro would've worked fine.

3. Install tools inside VM

Alpine minirootfs doesn't ship util-linux/iproute2/iptables by default.

limactl shell raw-container sudo apk add util-linux iproute2 iptables curl

Verified present: unshare, nsenter, chroot, ip, iptables, curl.

What/why: These are the actual tools that do the container-building, and they're just normal Linux command-line programs, nothing docker-specific about any of them:

unshare — creates new namespaces (the isolation)
nsenter — lets you "enter" a process's namespace from outside (used later to configure its network)
chroot — changes what a process considers to be / (root directory)
ip — configures network interfaces, routes
iptables — configures NAT/firewall rules
apk is just Alpine's package manager (like apt on Ubuntu, brew on Mac).

Building the container

4. Dirs + Alpine minirootfs (the "image")

Correction: Alpine's sh (ash) does NOT support bash brace expansion mkdir -p {base,upper,work,merged} — it created one literal dir named {base,upper,work,merged} instead of 4 dirs. Fix: list dirs explicitly.

limactl shell raw-container sh -c '
cd ~/raw-container
mkdir -p base upper work merged
'

(the literal {base,upper,work,merged} dir created by the failed brace expansion was removed with rm -rf before retrying — one-time cleanup, not part of the fix itself)

Downloaded + extracted Alpine minirootfs (aarch64, matches VM arch):

curl -L -o alpine-minirootfs.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/aarch64/alpine-minirootfs-3.23.4-aarch64.tar.gz
sudo tar -xzf alpine-minirootfs.tar.gz -C base

What/why: A "container image" (like what docker pull downloads) is really just... a folder full of files that looks like a tiny Linux filesystem (/bin, /etc, /usr, etc). That's exactly what just got downloaded into base/, a stripped-down copy of Alpine's root filesystem, plain files, nothing fancy. upper/, work/, merged/ stay empty for now, they come into play next.

5. Mount OverlayFS

Note: lima's guest username was rajiv.guest (not rajiv as on host) — path was /home/rajiv.guest/raw-container/.... Used $PWD to avoid hardcoding.

limactl shell raw-container sh -c '
cd ~/raw-container
sudo mount -t overlay overlay -o lowerdir=$PWD/base,upperdir=$PWD/upper,workdir=$PWD/work $PWD/merged
mount | grep merged
'

Result:

overlay on /home/rajiv.guest/raw-container/merged type overlay (rw,relatime,lowerdir=...,upperdir=...,workdir=...,uuid=on)

What/why: This is the trick behind "container image layers." base/ (the Alpine files downloaded earlier) stays completely untouched and read-only. Anything the container creates or edits lands in upper/ instead. merged/ is just a view combining both, looks like one normal filesystem, but writes get quietly diverted to upper/ under the hood. Exactly why you can run a container, install a package, delete the container, and the original image is still pristine, nothing was ever actually written into it.

6. Namespaces + chroot

limactl shell raw-container sudo unshare --pid --uts --ipc --net --mount --fork chroot /home/rajiv.guest/raw-container/merged /bin/sh -c 'mount -t proc proc /proc; cat /etc/os-release; ps; sh'

Verified:

/etc/os-release reports Alpine (host VM is also Alpine here, so less dramatic than an Ubuntu-host demo, but root fs is confirmed isolated — different filesystem tree, not the host's).
ps shows the shell as PID 1 inside the namespace — confirms PID namespace isolation.
pwd = / — confirms chroot.

What/why: This is the actual "create the container" command, everything before this was just prep. Two things happen together:

unshare --pid --uts --ipc --net --mount — spins up a new, empty set of namespaces (process list, hostname, IPC, network, mounts). The new shell can only see its own version of each of these, not the VM's.
chroot merged /bin/sh — tells that new shell "your / is merged/, nothing outside it exists for you."

Put together, that's a process that (a) thinks it's alone on the machine, own PID tree, own network, and (b) can only see the Alpine files prepared earlier, not the VM's real filesystem. That's literally a container, no separate binary, no daemon, just a regular /bin/sh process launched with different kernel-level boundaries.

7. Host-visible PID of the chrooted shell

Ran unshare in one terminal, checked ps aux in a second (host PID differs from in-container PID 1).

limactl shell raw-container ps aux | grep sh

Found the chain:

3118/3119  sudo unshare --pid --uts --ipc --net --mount --fork chroot ...
3120       unshare --pid --uts --ipc --net --mount --fork chroot ...
3123       sh                                                          <- this is it

Host PID = 3123 — this is what nsenter -t <PID> targets for network namespace work below.

What/why: Good mental-model check here, the container process didn't get some "new secret identity." From inside it sees itself as PID 1, but the VM still sees it as a completely ordinary process, PID 3123, same as anything else. Namespaces only change what a process can see, not what it fundamentally is. Needed this host-side PID for the next steps, since nsenter and cgroups get configured from outside the container, targeting it by its real PID.

8. cgroups — memory limit

limactl shell raw-container sh -c '
sudo mkdir -p /sys/fs/cgroup/raw-container
echo 100M | sudo tee /sys/fs/cgroup/raw-container/memory.max
echo 3123 | sudo tee /sys/fs/cgroup/raw-container/cgroup.procs
cat /sys/fs/cgroup/raw-container/memory.max
'

Result: memory.max = 104857600 (100MB) applied to PID 3123 (cgroup v2, unified hierarchy — Alpine mounts this by default).

What/why: Namespaces (steps 6-7) control visibility, what the process can see. cgroups control consumption, how much it can use. /sys/fs/cgroup isn't a real disk folder btw, it's a special kernel interface, creating a directory there creates a resource-limit "group," and writing files into it configures/applies the limits. Made a group, capped its memory at 100MB, then added PID 3123 (the container) to that group, from that point the kernel enforces the cap on that process (and anything it spawns). Genuinely the same mechanism Docker uses for docker run --memory.

9. Networking — veth pair, NAT, DNS

Create veth pair on host:

limactl shell raw-container sudo ip link add veth-host type veth peer name veth-guest

What/why: Since the container got its own network namespace (--net in step 6), it started out with no network at all, not even a way to talk to the VM it's running inside. A veth pair is basically a virtual patch cable with two ends: veth-host stays in the VM's normal network, veth-guest will go inside the container. Whatever goes into one end comes out the other.

Move veth-guest into container's net namespace (host PID 3123 — see step 7):

limactl shell raw-container sudo ip link set veth-guest netns 3123

What/why: This basically hands one end of the virtual cable into the container's isolated network namespace, so it's visible from inside now.

Assign IPs, bring interfaces up. Host side:

limactl shell raw-container sudo ip addr add 10.0.0.1/24 dev veth-host
limactl shell raw-container sudo ip link set veth-host up

Container side (inside the / # shell, PID 3123):

ip addr add 10.0.0.2/24 dev veth-guest
ip link set veth-guest up
ip link set lo up

What/why: A network cable with no IP addresses on either end is useless, so this gives each end an address on the same tiny private subnet (10.0.0.0/24) so they can talk to each other, and brings the interfaces up (like plugging the cable in, basically). lo = loopback (127.0.0.1), had to bring that up too since the container's namespace starts with it disabled.

Verified link: ping -c 2 10.0.0.1 from container → host, 0% loss.

Note: default route (10.0.0.1 via veth-guest) was auto-added by the kernel when the IP was assigned — ip route add default via 10.0.0.1 errored "File exists", which is fine, route was already correct.

Host outbound interface is eth0 (lima VM's uplink) — confirmed via ip route show on host before writing NAT rules (don't assume the interface name, check it).

Enable IP forwarding + NAT on host:

limactl shell raw-container sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
limactl shell raw-container sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
limactl shell raw-container sudo iptables -A FORWARD -i eth0 -o veth-host -j ACCEPT
limactl shell raw-container sudo iptables -A FORWARD -o eth0 -i veth-host -j ACCEPT

What/why: The container can now reach the VM (10.0.0.1), but the internet has no clue what 10.0.0.2 even is, that private address is meaningless outside this VM. NAT (MASQUERADE) makes the VM rewrite the container's outgoing traffic to look like it's coming from the VM itself (its eth0 address), then routes replies back. ip_forward=1 is needed for the VM's kernel to relay traffic between two interfaces at all (off by default, since a machine usually isn't a router). The iptables -A FORWARD rules explicitly allow that relaying, without them traffic gets forwarded but then just dropped by the default firewall policy.

Verified: ping -c 2 8.8.8.8 from container → internet, 0% loss.

DNS — container had no resolver config by default:

echo "nameserver 8.8.8.8" > /etc/resolv.conf
apk update

What/why: Ping worked fine with a raw IP (8.8.8.8), but hostnames like dl-cdn.alpinelinux.org need DNS to resolve first. /etc/resolv.conf is the standard Linux file that names which DNS server to ask, the container's copy (inside merged/) was empty, so nothing could resolve names yet. Pointed it at Google's public DNS (8.8.8.8) and that fixed it.

Result: OK: 27447 distinct packages available — full networking (isolation + NAT + DNS) confirmed working.

10. Getting hermes-agent into the container

Attempt 1 — copy source (rsync). Worked, but wasteful (502MB copy) and stale vs host.

Attempt 2 — bind mount instead. Cleaner idea — live source, no duplication.

# undo the copy
limactl shell raw-container sudo rm -rf /home/rajiv.guest/raw-container/merged/opt/hermes-agent
limactl shell raw-container sudo mkdir -p /home/rajiv.guest/raw-container/merged/opt/hermes-agent

# bind mount host source into container's mount namespace (PID 3123 already has its own from --mount)
limactl shell raw-container sudo nsenter -t 3123 -m mount --bind /Users/rajiv/.hermes/hermes-agent /home/rajiv.guest/raw-container/merged/opt/hermes-agent

What/why a bind mount: unlike a symlink (which just points to a path, meaningless once chrooted since the container can't see outside merged/ at all), a bind mount makes one real directory appear at a second location, live. nsenter -t 3123 -m means "run this mount command as if inside PID 3123's mount namespace," so the mount takes effect just for the container's view, not the whole VM's.

Hit a wall here: pip install -e . failed with error: Cannot update time stamp of directory 'hermes_agent.egg-info'. Root cause turned out to be lima's virtiofs mount (macOS↔VM bridge) doesn't support utimes on directories, and setuptools needs to touch egg_info/ during an editable build. Real limitation of bind-mounting a host macOS dir for build tooling, not something worth fighting.

Decided to drop the bind mount entirely. Unmounted it, installed hermes-agent fresh from PyPI instead (it's published there anyway), no host coupling at all, honestly fits the "container has its own image" model better too.

limactl shell raw-container sudo nsenter -t 3123 -m umount /home/rajiv.guest/raw-container/merged/opt/hermes-agent

11. Python + venv inside the container

Installed system python3 + build deps via apk:

apk add python3 py3-pip gcc musl-dev python3-dev libffi-dev

What/why: The Alpine minirootfs image has nothing installed beyond bare essentials, no Python at all. gcc/musl-dev/python3-dev/libffi-dev are compiler + headers, needed since some Python packages install from source and need to compile small C extensions.

Hit uv venv failing twice here:

uv venv venv --python 3.12 → Failed to discover managed Python installations / Could not detect either glibc version nor musl libc version, uv just couldn't probe libc on this minimal Alpine image, even with --python $(which python3) given explicitly (No interpreter found at path).
Fix was to skip uv venv entirely, use stdlib venv directly: python3 -m venv <path>. Worked fine, uv pip install still usable afterward if I wanted it, just not uv venv for env creation here.

Also, first venv attempt (python3 -m venv venv inside /opt/hermes-agent, i.e. inside the bind-mounted dir) failed with Read-only file system, another symptom of that same virtiofs bind-mount limitation. Fix: put the venv in a container-local path instead, never inside a host-bind-mounted dir.

python3 -m venv /root/hermes-venv

What/why a venv at all: A "virtual environment" is just an isolated folder of Python packages + its own python binary, separate from the system Python. Keeps hermes-agent's specific dependency versions from clashing with anything else, standard Python practice really, unrelated to containers themselves, would've done this on a normal machine too.

12. Install hermes-agent (PyPI) — hit the cgroup limit

/root/hermes-venv/bin/python -m pip install hermes-agent

First attempt: process got Killed mid-resolve. Confirmed via dmesg on host — real cgroup enforcement, not a fluke:

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/raw-container,task=python,pid=12280
Memory cgroup out of memory: Killed process 12280 (python) total-vm:115608kB, anon-rss:96536kB

Turns out 100MB (set in step 8) was too tight for pip's dependency resolution. Bumped it up:

limactl shell raw-container sudo sh -c 'echo 512M > /sys/fs/cgroup/raw-container/memory.max'

Retried install — succeeded.

What/why: This is literally the cgroup memory cap from step 8 doing its job, kernel killed the pip process the instant it crossed 100MB, exactly as designed. Not a bug, actually a nice demonstration the resource limit is real and enforced, same mechanism Docker's --memory flag relies on. Just needed a more realistic limit for an actual package install.

13. Run hermes gateway

/root/hermes-venv/bin/hermes gateway run

Result — running successfully inside the raw container:

┌─────────────────────────────────────────────────────────┐
│           ⚕ Hermes Gateway Starting...                 │
├─────────────────────────────────────────────────────────┤
│  Messaging platforms + cron scheduler                    │
│  Press Ctrl+C to stop                                   │
└─────────────────────────────────────────────────────────┘

(Warnings about no messaging platforms/allowlists configured are expected — unrelated to containerization, just default hermes config.)

What/why this counts as "in a container": This hermes process is running with its own PID namespace (thinks it's PID 1 or close to it), its own network namespace (reachable at 10.0.0.2, NAT'd to the internet), its own filesystem (the Alpine merged/ view, not the VM's real one), and a hard 512MB memory ceiling enforced by the kernel. That combination is basically what a container is, just built each piece by hand instead of typing docker run.

Outcome

Built a container from raw Linux primitives (namespaces via unshare, OverlayFS, cgroups v2, veth+NAT+DNS, chroot), zero docker/podman involved, and got hermes gateway run running inside it, isolated and resource-capped. Felt pretty good honestly!

Here's how it all stacked up in the end:

Biggest real-world lesson: bind-mounting a macOS host dir into a Linux VM (virtiofs) breaks tools that need directory utimes (setuptools editable installs). Not a container-specific issue, just a virtiofs/9p limitation. Worth remembering for any lima/VM workflow, not just this one exercise.

Considered alternative

Did wonder mid-exercise whether to just use podman instead, but decided to finish the raw build since the whole point was learning the primitives, not shipping something. Podman/docker are still the right call for actual day-to-day use, this was purely for understanding.

Cleanup / how to stop everything

Option A — pause for today, resume another day (keep the VM + downloaded image)

limactl stop raw-container

Stopping the VM kills the running container (namespaces, cgroup, veth) along with it, that's fine, those don't need to persist anyway. What does stay on disk: the VM itself and ~/raw-container/{base,upper,work,merged} inside it (the Alpine minirootfs, whatever files got installed).

To resume:

limactl start raw-container

Then redo from step 5 (mount overlay, doesn't survive a stop) through step 6 (unshare/chroot into a fresh container). Steps 1-4 (install lima, start VM, install tools, download minirootfs) don't need repeating, all still sitting on disk.

Option B — permanently stop, clean, delete everything

limactl stop raw-container
limactl delete raw-container

This wipes the VM and its entire disk, the downloaded Alpine image, base/upper/work/merged, hermes venv, everything built in this exercise. Nothing left on the Mac from it except this note. lima itself (the tool) stays installed though; brew uninstall lima separately if you want that gone too, not needed really, harmless to keep around.