Reverse-Proxying HTTPS Traffic to a Server in my Basement Using Caddy and Wireguard

I’ve been experimenting with new web services lately, some of which I’d like to open up to my mobile devices. Punching a hole through my home network firewall is unappealing, though, as is standing up a new VPS for each one I’d like to kick around. I already feel like I’ve got too many VPS instances. So, naturally, the way to fix it is to add one more.

This feels like the ideal use case for the Caddy 2 proxy server connected to a VM running on my LAN via a WireGuard VPN that starts whenever the VM on the LAN is booted.

Obviously, this is not for anything critical. It’s just a nice, low friction way to try things out without jumping through too many hosting hoops.

By coincidence, my old favorite OS for hosting containers, CoreOS container linux, has announced its end-of-life. So I need to decide on a new option: either going with Fedora Core OS, moving to one of the container linux forks, or something else entirely. Unfortunately, Fedora CoreOS is not a good choice for this yet, due to difficulty with wireguard. So I’ll be using alpine as a docker host, mostly because I’ve been curious about it for a while.

The Web Service (aka my motivation)

I have been using self-hosted BitWarden as a password manager for several months now, and am mostly happy with it. Currently, I’m hosting it on a VPS that all my devices can use. The main thing I dislike about the server is that the default self-hosted service is quite heavy. Every six weeks or so, it consumes all the resources on my VPS and the system needs to be rebooted. Usually I notice this because the clients are incapable of saving passwords locally when the service is offline.

There is an unofficial compatible service whose resource usage looks more appealing given the small number of users my instance will have. I’d like to kick the tires on it without setting it up on a VPS somewhere and without punching a hole in my firewall for my mobile stuff to be able to reach the server and save passwords.

I’ll dedicate another post to details of the bitwarden setup. This is about getting wireguard working as a reverse proxy.

VPS Setup

CentOS 8 would normally be my host OS of choice for my VPS. I’m familiar with Red Hat’s tooling, it’s stable, it’s well supported by any VPS provider I could want to use, and it’s new enough that getting things like WireGuard and Caddy to run should not be difficult.

Unfortunately, the golang compiler in EPEL is not able to build Caddy, so the project’s EL8 COPR is not currently available.

Because the specific distribution is not important for this exercise, and building Caddy for EL8 outside the COPR infrastructure would mark about the fourth shaved yak of the weekend, I’m using fedora 31.

OS Install

I chose the smallest VPS from Digital Ocean. It’s got 1 CPU, 1GB RAM, 1TB monthly transfer and 25GB of storage, and would cost me $5 per month at their current pricing if I decide to leave this running once the trial is done. (If anyone reading would like a referral link that gets $100 of free hosting for a couple months, and somehow can’t find one, this one supports a podcast that I enjoy listening to but am otherwise unaffiliated with.)

I used their default Fedora 31 image. Once the VM was ready, I connected via ssh, created a new account for myself, forbade root login in sshd and patched it up. That required a reboot because there was a kernel upgrade.

I really appreciate that this service lets me include an ssh public key that gets installed on all of the VPSes I create by default. It’s a much nicer workflow than the usual dance with a temporary password I use elsewhere.

Wireguard Setup

The topology that makes the most sense to me here is to have my local VM running the back-end service be the wireguard client and have the reverse proxy be the wireguard server. This lets me avoid opening/forwarding any ports as I add back-end stuff I want to expose.

My server will listen for wireguard connections on an arbitrary high-numbered UDP port and will only pass traffic between RFC1918 addresses of systems on the VPN. This is very close to the simplest possible wireguard configuration.

Server

For Fedora 31, you need to activate a COPR and enable DKMS so that DNF will rebuild the kernel extension whenever the kernel is upgraded. For Fedora 32 (currently in beta) that will be unnecessary as the tools will be included in the Fedora-maintained repositories and the upstream kernel will include the extension.

sudo dnf copr enable jdoss/wireguard
sudo dnf install wireguard-dkms wireguard-tools

Once the tools are installed, it’s relatively quick to set up. The wg-quick utility is very low friction, especially if you follow a couple of conventions. The path of least resistance is to keep a configuration file named for the virtual interface with the .conf extension in /etc/wireguard

sudo -s
mkdir -p /etc/wireguard
umask 077
cd /etc/wireguard

Then generate a private key, dump the associated public key into a file for future reference, and generate a pre-shared key.

wg genkey | tee /etc/wireguard/privatekey | wg pubkey > /etc/wireguard/publickey
wg genpsk >psk

Populate /etc/wireguard/wg0.conf with these values:

[Interface]
Address    = 10.20.30.1/24
PrivateKey = CONTENTS_OF_/etc/wireguard/privatekey
#PostUp     = firewall-cmd --zone=public --add-port 51820/udp && firewall-cmd --zone=public --add-masquerade
#PostDown   = firewall-cmd --zone=public --remove-port 51820/udp && firewall-cmd --zone=public --remove-masquerade
ListenPort = 51820

[Peer]
PublicKey    = PUBLIC_KEY_FROM_CLIENT
PresharedKey = CONTENTS_OF_/etc/wireguard/psk
AllowedIPs   = 10.20.30.2/32

The commented out lines are useful if you need the default fedora firewall to open the port when the wg interface is started; on this box, I’m using DigitalOcean’s firewall for that purpose, so I had to add a permit rule for UDP port 51820 on the public interface.

Once this is populated, wg-quick up wg0 will create the wg0 interface, assign its IP address and mask, add a route and start listening for VPN traffic. In order for connections to be permitted, peers need to be listed in this configuration file, have the private key corresponding to the public key listed, use the appropriate preshared key to authenticate traffic, and be have their wireguard interfaces configured to use one of the listed AllowedIPs.

Client

Setting up the client side is very similar to the server. Installing wireguard required four new packages for alpine:

apk add wireguard-lts wireguard-tools-wg-quick wireguard-tools-wg bash

Generating a keypair on the client looks just like it does on the server, except that the preshared key is copied over from the server:

sudo -s
mkdir -p /etc/wireguard
umask 077
cd /etc/wireguard
wg genkey | tee /etc/wireguard/privatekey | wg pubkey > /etc/wireguard/publickey

And the wg0.conf is quite similar:

[Interface]
Address    = 10.20.30.2/24
PrivateKey = CONTENTS_OF_/etc/wireguard/privatekey

[Peer]
PublicKey    = PUBLIC_KEY_FROM_SERVER
PresharedKey = CONTENTS_OF_/etc/wireguard/psk
Endpoint = server.example.com:51820
AllowedIPs   = 10.20.30.1/32
PersistentKeepAlive = 25

wg-quick up wg0 will bring up the interface and you should be able to connect in both directions at that point.

The wg command will show the state of the connection.

# wg
interface: wg0
  public key: orEcFvOLSO2tKG8RIv0bIq/qVZEBigNUnP8W4XJ8jgM=
  private key: (hidden)
  listening port: 51820

peer: ysHtcocvxaUY635TZsV6cMWaXUhRCxSI1MgDhVJxLUk=
  preshared key: (hidden)
  endpoint: client.example.com:38927
  allowed ips: 10.20.30.2/32
  latest handshake: 6 seconds ago
  transfer: 6.74 KiB received, 11.36 KiB sent

Caddy 2 Setup

Configuring Caddy 2 as a TLS reverse proxy is very straightforward.

Installation:

sudo dnf copr enable @caddy/caddy
sudo dnf install caddy

Then a two-line configuration file is enough for caddy to go get a certificate from LetsEncrypt and proxy TLS back to the VM in my basement over wireguard. In /etc/caddy/Caddyfile

server.example.com
reverse_proxy 10.20.30.2:80

systemctl start caddy is then enough for it to do the right thing.

Conclusion

Adding a new service is now just a matter of standing up a new VM on my LAN’s VM server and connecting it to the wireguard VPN, then adding a block in the caddy 2 configuration. It lowers the impedance to trying out new self-hosted things without growing my VPS zoo every time I want to kick the tires on something. So far, I like it a great deal.