I’m blogging today’s yak shaving exercise because I found web searches challenging for some of this. And hopefully writing them down here will both help improve those searches and help me remember. Also, I am determined to blog more in 2024. This post really is mostly about snags in my Alpine Linux upgrade adventure, but I’d like to share some back-story first.

How I Got Here

My password management setup is a little over-complicated, and I can only blame myself for that. 15+ years ago, I got to meet some of the developers behind 1Password. They were smart and the product was solid and well-priced. At some point, they decided to move to a subscription-only service. That was initally OK with me; while I prefer to just buy software up front, I won’t try to instruct someone else on how to run their software business. Then I started using Linux more frequently on the desktop, and their story there was sub-optimal at the time. I limped along using their Windows client with WINE. After that, they started requiring you to store data on their server in order to sync between devices. That’s when I needed to switch.

I made the switch to Bitwarden. It’s nice software with a UI good enough that my 1Password-loving family will use it, and all of the source code is available under free licenses. They even published a version of their server suitable for self-hosting with relatively easy instructions. I cheerfully pay for a family subscription with them, but I do insist on hosting my own data. The official self-hostable server is quite heavy for a family-sized group of people, though, so I made the switch to bitwarden_rs in 2019. That has since been renamed to vaultwarden and it’s really great. Easy to run using docker, easy to back up, etc. I host it on a server in my basement, and have a VPS that my vaultwarden server connects to using wireguard. That VPS serves as a reverse proxy and has a valid TLS certificate that makes all the various official mobile clients and browser extensions that work with the mobile apps and browser extensions my user community family wants to use. Apart from occasionally making sure my backup script is working, the system very seldom requires any attention from me.

Today, it needed some attention (or I thought it did). I rarely use LinkedIn, but someone from Kentucky is repeatedly trying to log into my account. Notifications of this person’s attempts inspired me to rapidly change my password while I was traveling for our winter holidays, add TOTP authentication, and store recovery codes for said two-factor auth in my bitwarden password database. Today, they tried again while I was at my desk. I signed in to verify my settings, and wanted to double-check that my two factor recovery codes were available from my desktop PC. That bitwarden client was showing an error during synchronization. I wanted to fix that.

The Initial Problem

In the past, when something like that has happened, it’s been because my vaultwarden image was out-of-date and the upstream client had advanced past the API version implemented there. So my first thought was to upgrade vaultwarden. That is generally a simple process:

tar cjvf /root/bw-data-`date +%Y%m%d`.tar.bz2 /bw-data
docker pull vaultwarden/server:latest
docker stop bitwarden
docker rm bitwarden
docker run -d --name bitwarden -v /bw-data/:/data/ -p 80:80 vaultwarden/server:latest

The “bitwarden” name is a wart because I used to run the official bitwarden server in a container, and I didn’t feel like changing my monitoring script to handle a new name.

That didn’t work this time. Evidently the latest vaultwarden image needs a newer docker version. Docker was already updated as far as I could support on that Alpine system. I didn’t feel like dealing with this today, but on the list of things I care about working, my password manager ranks very near the top. And I had just broken the sync server. I could have rolled back easily, but I was working under the belief that my desktop client was not being synchronized properly. That hadn’t spread to the browser extensions yet, but if those or the mobile clients stopped working, it was going to be a real headache. The desktop client felt like the canary in the coal mine. (Spoiler: it wasn’t. I just needed to log out and back in, in the end.)

My Alpine VM Was OLD

I was running alpine 3.11. That shipped in 2019 and stopped being officially supported in 2021. I suppose it’s a minor miracle that it only now caused me a problem. I decided to make sure I had a good backup, and set to upgrading. I should have just set up a new VM with the latest stable version. This server is livestock… All I care about is the wireguard configuration and the data used by that vaultwarden image. For reasons I can only attribute to sheer orneriness, I wanted to upgrade this VM instead of creating a new one and copying over those two data items.

First Volley: Update /etc/apk/repositories and Get Going

As a first attempt, I decided that I’d like to keep this on the latest stable version, so I edited /etc/apk/repositories

https://dl-cdn.alpinelinux.org/alpine/latest-stable/main
https://dl-cdn.alpinelinux.org/alpine/latest-stable/community

to always point to latest-stable and to use the TLS version of the distro’s CDN.

After that, I ran apk update. It failed with a signature error. Understandably, the distro has rolled their package signing keys since 2019. They backported the package that contains them all the way back to 3.12, but I was on 3.11. Had I realized that sooner, I could have upgraded the OS to 3.12, upgraded the package containing the signing keys, then went straight to latest-stable.

Manually Upgrade Signing Keys

Instead, I pointed a browser on my desktop machine to the package file containing the keys and upgraded just that file, deciding that TLS was enough of a chain of trust for me here:

apk add --allow-untrusted ./alpine-keys-2.4-r1.apk
apk update

Then I tried apk upgrade, and got an unresolvable conflict attempting to upgrade libffi. Evidently, some existing packages depended on it and had been completely removed from the distro in the intervening years, so they couldn’t be upgraded to packages that depended on the new version of libffi. I determined which packages were no longer part of the distro by running:

# dump list of available packages
apk search -q |sort -u >/tmp/avail
# see which installed packages are no longer available
comm -23 /etc/apk/world /tmp/avail

Two python packages stood out as the likely culprits, so I manually removed those. They weren’t things I explicitly depended on, so it felt safe to assume that whatever packages wanted them would now use the same tools in whatever way newer releases provided those tools:

apk del py-pip python-dev

With that, apk upgrade did its job.

latest-stable docker doesn’t work

After the reboot, I was hopeful. I tried to start my container, and got a very confusing message:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown.

Running any container produced a similar error, including docker run hello-world. After quite a bit of digging around, it seems that docker 24.01 in its default configuration wants an interface that the LTS kernel does not provide. If the LTS kernel were going to be an ongoing concern, I might have preferred to downgrade docker to a version that works with that. But LTS has changed meanings recently, and it didn’t seem worth spending any effort on following that. And besides, Alpine now offers a kernel that’s optimized for running as a VM guest, the way I was using it.

Upgrading to linux-virt

So I thought I’d upgrade to that:

apk add linux-virt
sync
reboot

But the system booted back into the LTS kernel. Alpine uses a new-to-me bootloader called syslinux. And in its default configuration I never see a menu. Partly because it runs in a VM on a headless server, and partly because it’s just configured that way out of the box.

By this point, I had decided that:

  • I was no longer ornery enough to pursue this upgrade any further
  • A rollback was probably the safest thing if this took more than about 5 more minutes

So I decided to give linux-virt one try by making it the default kernel and see what happened. I changed /etc/update-extlinux.conf so that default=virt, ran:

update-extlinux
reboot

and started getting ready to roll back.

It came up and just worked. At which point I discovered that simply logging out of the desktop client and logging back in would have let me refrain from shaving these yaks until a different day. I’m not unhappy, though. I learned new things about how Alpine Linux worked on a day that’s snowy and a school/federal holiday. As more time went by, this would’ve become harder to fix and it could’ve reared its head on a day where I’d have a harder time absorbing the effort.

I’m making a new start on Kev Quirk’s “100 Days To Offload” idea for 2024. You can see details and join yourself by visiting 100daystooffload.com.

This is day #1.