My home network uses IPv4 and IPv6 ranges. Several machines run docker and all have default ranges, which of course overlap. I can setup docker networks with fixed IP ranges, but for one case that does not solve everything since it creates ad-hoc networks.

My setup

For CI/CD I use a Drone CI docker runner for some workloads. For each build step, it spawns a new docker container, so it's quite powerful and flexible. This is a tradeoff against requiring access to the docker socket to manage docker containers.

It is a security issue to give docker containers access to the docker socket, so to mitigate that slightly, I run a VM only to run docker for drone runner.

So I have a machine (apollo)¹, which runs drone CI and other services in docker containers. Then I have another machine (heracles) where I run the drone runner. Both machines use the same subnet (192.168.1.0/24). On apollo I run docker containers and on orion I run a virsh VM which runs Docker inside. Docker by default recycles the same subnets (in this case 172.16.0.0/12 in blocks of /16). I have created non-overlapping docker networks on apollo and orion, so they are stable even between reboots. The VM itself needs their own subnet as well and that is again non-overlapping: 192.168.100.0/24.

Diagram 1 shows this convoluted network setup.

,--[apollo]--------------.     ,---[heracles]-----------.
|   192.168.1.1/24       |     |    192.168.1.2/24      |
|   runs docker          |     |    runs virsh/qemu VM  |
|                        |     |                        |
|,-[various]------------.|     |,--[orion]-------------.|
||  172.16.0.0/16       ||     ||   192.168.100.0/24   ||
||  docker containers  <++--.  ||   runs docker        ||
||                      ||  |  ||                      ||
|`----------------------'|  |  ||,-[drone runner]-----.||
`-----------------------'   |  |||  172.17.0.0/16     |||
                            |  |||  docker container  |||
                            |  |||                    |||
                            |  ||+-[runner steps]-----+||
                            |  |||  172.16.0.0/12(!)  |||
                            `--+++> docker containers |||
                               ||`--------------------'||
                               |`----------------------'|
                               `------------------------'

The problem

As you can see in the above diagram, the part I did not talk about yet is whenever the drone runner spawns a new instance for a build job. It will assign a separate network to this instance from the pool of 172.16.0.0/12, which of course overlaps with the docker network on apollo (172.16.0.0/16), hence the (!) mark there.

I was aware of the /etc/docker/daemon.json configuration bip that configures a network range for docker:

{
    "bip": "172.18.0.1/16"
}

Sadly, that only works for the default docker network and not for non-default networks.

A solution

I recently became aware of an option called default-address-pools, which does exactly what I needed; it creates a pool of subnets (in this case: 10.1.0./16) out of which it takes smaller subnets (in this case, /24) to assign to ad-hoc networks.

{
    "bip": "172.18.0.1/16",
    "default-address-pools": [
      {"base": "10.1.0.0/16", "size": 24}
    ]
}

Diagram 2 shows non-overlapping subnets, even with dynamic drone runner instances.

,--[apollo]--------------.     ,---[heracles]-----------.
|   192.168.1.1/24       |     |    192.168.1.2/24      |
|   runs docker          |     |    runs virsh/qemu VM  |
|                        |     |                        |
|,-[various]------------.|     |,--[orion]-------------.|
||  172.16.0.0/16       ||     ||   192.168.100.0/24   ||
||  docker containers  <++--.  ||   runs docker        ||
||                      ||  |  ||                      ||
|`----------------------'|  |  ||,-[drone runner]-----.||
`-----------------------'   |  |||  172.17.0.0/16     |||
                            |  |||  docker container  |||
                            |  |||                    |||
                            |  ||+-[runner steps]-----+||
                            |  |||  10.1.X.0/24       |||
                            `--+++> docker containers |||
                               ||`--------------------'||
                               |`----------------------'|
                               `-----------------------'

Alternate solutions

There are other solutions possible, that I did not look into:

1. Using a range outside 172.16.0.0/12 for my fixed docker subnets.
2. Only using IPv6 for these as they do not need to be reached from the outside.

The downside of using other IPv4 ranges is obviously that I need to re-number/re-route everything, so I did not feel like it at this time.

The downside of only using IPv6 is that not all services support that, so I have to have IPv4 networks as well anyway. Part of my containers can be IPv6-only though, it is something I'm considering.