Over the past week or so, I’ve been trying out using Docker to deploy a Django site on a VPS. My preferred setup for that is to have Caddy running on the host, not in any container, as a reverse proxy. (It’s a single, static binary; I don’t see any joy in wrapping that in a container.) In the past when I’ve hosted similar things, I just ran gunicorn in a python virtualenv on the host as well, and bound it to the loopback. The current thing I’m building is a little bit more painful to run that way on my VPS, so I thought I’d finally cave and give Docker a try in “production.” While there was quite a bit to like about it, there was also an unpleasant surprise.

My standard build, which I keep swearing I’ll put in an Ansible playbook next time I do it, includes a basic set of firewall rules. On a ubuntu host, here’s what they look like in ufw

$ sudo ufw status verbose
tatus: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
443                        ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
443 (v6)                   ALLOW IN    Anywhere (v6)

That means that I expect iptables to reject any incoming connections that aren’t in the list, and allow only ports 22 (ssh) and 443 (https).

That’s worked well for me, for quite a while.

This time, since I wanted to run gunicorn to serve my django application, redis as a message broker, celery as a task queue and postgres as my database, I thought docker compose might be a nice way to save time and make it easier to stand up my application. For the most part, it did. But when I was testing my setup, I started seeing strange log entries from django about hostnames that might need to be added to AUTHORIZED_HOSTS and requests that were using the HTTP CONNECT verb in an attempt to turn my server into a proxy for other parts of the web.

My normal expectation is that caddy would not allow either of these things through. I even explicitly disallowed the CONNECT verb as an experiment and turned on some additional logging.

test.example.com {
        log {
                output file /var/log/caddy/test.example.com-access.log
        }
        @disallowedMethods {
                method CONNECT
        }
        respond @disallowedMethods "HTTP Method Not Allowed" 405
        root * /srv/example.com/public
        file_server
        @notStatic {
                not path /static/*
        }
        encode zstd gzip
        reverse_proxy @notStatic :8000
}

With the extra logging, I was able to see that the requests, which seemed to be looking for holes in some IPTV software I’d never heard of, whose name made me think something much darker was happening, were in fact not coming through caddy’s reverse at all. They were going directly to gunicorn.

I double-checked my ufw rules using the status command above and saw that default deny was still enabled, and that ports 22 and 443 were the only ones allowed through. I even restarted the ufw service to make sure I wasn’t mistaken about its status.

After more head-scratching than I should have needed, I resorted to inspecting the iptables directly. When docker brought the container that was running gunicorn up, the ports configuration implied that gunicorn would be EXPOSEd on 0.0.0.0:8000:

  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    depends_on:
      - db
      - redis

I did not expect that docker would quietly add a rule to the system’s iptables to allow port 8000 through the firewall. It did. So, to fully bury the lede… my PSA is that if you’re using docker compose to set up services on a machine connected to the internet, make sure to bind them only to the loopback:

  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "127.0.0.1:8000:8000"
    depends_on:
      - db
      - redis

I’m trying on Kev Quirk’s “100 Days To Offload” idea. You can see details and join yourself by visiting 100daystooffload.com.

This is day 10. I’m way behind!