Skip to main content

Docker inside NixOS Containers

Okay but why?

I don't like Docker. I mean, I like the image, updates, packaging bits, but I very much dislike the networking. I don't want containers to use the host as a router, I want to use my router as a router, and also have control over east-west firewalls.

Enter: first-class NixOS systemd-nspawn containers. I absolutely love these, you can create them ephemerally for an extra clean environment, you can put them onto VLAN'd Linux bridges, have complete control over the IP network & host firewall, I've even got one that forces all of its traffic over a wireguard tunnel.

The thing is, though, there's a bit of trickery to make Docker work inside a systemd-nspawn container, so I'm documenting it here for future use.

Okay fine, but why not Podman?

I actually tried Podman first! But, at least on NixOS 23.11, stopping NixOS containers from the main host had an issue stopping the container that caused the shutdown time for it to be about 90 seconds. I found Docker didn't have this problem, so used it instead.

Just show me the config!

Okay here it is. In the below example, we're starting a simple hello-world container from docker hub.

containers.hello-world = {
  autoStart = true;
  ephemeral = true;
  bindMounts = {
    "/path/to/config/dir" = {
      hostPath = "/path/to/config/dir";
      isReadOnly = false;
    };
  };
  hostBridge = "br0";                    # Use a bridge on your system
  localAddress = "192.168.1.10/24";      # Configuring a static IP For the container
  privateNetwork = true;
  extraFlags = [                         # These extra flags are required for docker usage
    "--system-call-filter=keyctl"
    "--system-call-filter=bpf"
  ];
  config = { config, pkgs, lib, ... }: {
    system.stateVersion = "23.11";
    networking = {
      defaultGateway = "192.168.1.1";
      nameservers = [ "192.168.1.1" "1.1.1.1" ];
      nftables.enable = true;            # Open firewall as needed
      firewall.allowedTCPPorts = [ 12345 54321 ];
      firewall.allowedUDPPorts = [ 12345 54321 ];   
    };
    virtualisation.oci-containers = {
      backend = "docker";                # As of 23.11 Podman causes long container shutdown times, docker is quick.
      containers.hello-world = {
        image = "hello-world";
        extraOptions = [ "--net=host" ]; # Let docker have the whole host NIC
      };
    };
  };
};

Example with Plex

Plex has some specific needs:

  1. It needs access to the host GPU, passed through via /dev/dri
  2. It needs access to create /dev/dri nodes for the container
  3. It needs read/write access to a configuration directory
  4. It needs read only access to consumable video directories
containers.plex = {
  autoStart = true;
  ephemeral = true;
  allowedDevices = [
    {
      node = "/dev/dri/card0";
      modifier = "rwm";
    }
    {
      node = "/dev/dri/renderD128";
      modifier = "rwm";
    }
  ];
  bindMounts = {
    "/dev/dri" = {
      hostPath = "/dev/dri";
      isReadOnly = false;
    };
    "/var/lib/plex" = {
      hostPath = "/var/lib/plex";
      isReadOnly = false;
    };
    "/path/to/Movies".hostPath = "/path/to/Movies";
    "/path/to/TV".hostPath = "/path/to/TV";
  };
  hostBridge = "brDMZ";
  localAddress = "10.0.1.28/24";
  privateNetwork = true;
  extraFlags = [
    "--system-call-filter=keyctl"
    "--system-call-filter=bpf"
  ];
  config = { config, pkgs, lib, ... }: {
    system.stateVersion = "23.11";
    networking = {
      defaultGateway = "10.0.1.1";
      nameservers = ["10.0.1.2" "10.0.1.1"];
      nftables.enable = true;
      firewall.allowedTCPPorts = [ 32400 3005 8324 32469 ];
      firewall.allowedUDPPorts = [ 1900 5353 32410 32412 32413 32414 ];
    };
    virtualisation.oci-containers = {
      backend = "docker";
      containers.plex = {
        image = "lscr.io/linuxserver/plex:latest";
        environment = {
          PUID = "193";
          PGID = "800";
          TZ = "America/Regina";
          VERSION = "latest";
          #PLEX_CLAIM = "ONLY USED DURING SETUP";
        };
        volumes = [
          "/var/lib/plex:/config"
          "/path/to/Movies:/path/to/Movies"
          "/path/to/TV:/path/to/TV"
        ];
        extraOptions = [
          "--net=host"
          "--device=/dev/dri:/dev/dri"
        ];
      };
    };
  };
};
Back to top