Docker Network Conflicts on Synology NAS

This is one of those blog posts where I write about a technical problem that probably only affects a few people. But I’ll write it down anyway, so I can refer to it later if I ever encounter the problem again.

Table of Contents

• The Environment
• The Problem
• Troubleshooting
• The Solution

The Environment

I’m running a Synology NAS (DS920+) with DSM 7.3-81180 and Docker, or rather, “Container Manager,” as Synology calls it, installed via the Synology Package Center. I also use Hyper Backup to back up my NAS to another NAS in an offsite location, and vice versa.

The network route between these looks like this: NAS1 -> fritz.box -> Site2Site VPN -> fritz.box (offsite) -> NAS2.

My network uses the 192.168.10.x subnet, and the offsite location uses the 192.168.70.x subnet. (These are not the real IPs, but they are close enough for this example.)

The Problem

Two days ago I started getting messages from Hyper Backup stating that the backup jobs were failing because the destination NAS was unreachable. Unfortunately, the GUI did not provide more information about the error and incorrectly identified certificate errors as the cause.

Troubleshooting

I dug deeper into the NAS with SSH and found out that the destination NAS was indeed not reachable via its local IP address (192.168.70.x). Pinging it failed and ip route get 192.168.70.100 showed a very unexpected 192.168.70.100 dev docker-ece8b792 src 192.168.64.1 route instead of going through the VPN. What’s immediatey suspicious here is the docker-ece8b792 interface, which is a Docker network interface. Listing the Docker networks with docker network ls showed that there was indeed a Docker network with the ece8b792 ID and which container was using it. To get even more details about the network, I ran:

sudo docker network inspect $(sudo docker network ls -q) 2>/dev/null | grep -B5 "192.168.64"

which resulted in this output:

        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.64.0/20",
                    "Gateway": "192.168.64.1"
--
        "Containers": {
            "<redacted>": {
                "Name": "container-name",
                "EndpointID": "<redacted>>",
                "MacAddress": "01:23:45:67:89:02",
                "IPv4Address": "192.168.64.2/20",

So now we have a culprit: Docker created a network 192.168.64.0/20 (192.168.64.0 through 192.168.79.255) which overlaps with the offsite VPN network, causing all traffic to that subnet to be routed to the Docker network instead of going through the VPN.

But why does it do so? The container configuration was a very basic one:

services:
  container-name:
    build: .
    container_name: container-name
    env_file: .env
    restart: unless-stopped

No custom network configuration at all.

Somewhere here I remembered that I had fixed a similar problem years ago. This was confirmed by the /etc/docker/daemon.json file I found on the NAS:

admin@nas:/$ sudo cat /etc/docker/daemon.json

{
  "default-address-pools": [
    {
      "base": "172.16.0.0/12",
      "size": 24
    },
    {
      "base": "10.0.0.0/8",
      "size": 24
    }
  ]
}

But… it turns out that this is not used on DSM 7.3 anymore!

A quick sudo docker info | grep -A5 "Default Address Pools" confirmed that and did not show any default address pools.

The Solution

After some digging around, I found out that Synology has changed the way Docker is configured on DSM 7.3 and now uses a different configuration file: /var/packages/ContainerManager/etc/dockerd.json.

So to fix it I added the default address pools to that file instead:

sudo tee /var/packages/ContainerManager/etc/dockerd.json << 'EOF'
{
  "data-root": "/var/packages/ContainerManager/var/docker",
  "log-driver": "db",
  "registry-mirrors": [],
  "seccomp-profile": "unconfined",
  "storage-driver": "btrfs",
  "default-address-pools": [
    {"base": "172.16.0.0/12", "size": 24},
    {"base": "10.0.0.0/8", "size": 24}
  ]
}
EOF

Then restarted the Container Manager service:

sudo systemctl restart pkg-ContainerManager-dockerd

Now we also see the address pools correctly with sudo docker info | grep -A5 "Default Address Pools":

 Default Address Pools:
   Base: 172.16.0.0/12, Size: 24
   Base: 10.0.0.0/8, Size: 24

After a docker compose down and docker compose up -d of the affected container, a new Docker network was created with an IP range from the correct address pool and the offsite NAS was reachable again.