hamster.dance on NixOS

Published 10 Dec 2020, last modified 15 Dec 2020

I recently migrated my virtual private server (VPS)—the rented computer that runs hamster.dance—to a new operating system. It had been running on CentOS 8, and that was working pretty well overall, but I’ve been pretty enthusiastic about NixOS lately, I could think of a few ways it might make managing the server a little easier, and the web site is a hobby for me rather than critical infrastructure, so I figured I might as well try. I put together a checklist to follow in order to perform the migration, and prepared a bunch of Nix configuration code ahead of time. I can’t say I pulled off the migration without a hitch, but I learned a few things along the way (not all of them specifically related to NixOS) and have even made a few little improvements already.

A screenshot of neofetch displaying statistics of my NixOS system on a command line

Lesson 1: I still really have a lot to learn about networking

When I first started the NixOS installation on my VPS, I met with a nasty surprise: no network connectivity. My hosting provider gave me Virtual Network Computing (VNC) access to the VPS—a browser window that showed exactly what would have been on a physical monitor if I could connect one to the VPS, and allowed me to type commands into it—but none of the commands I typed into the VPS could reach the Internet, and I couldn’t transfer any files over from my own computer. This halted the whole NixOS installation because the install process requires a connection to at least nixos.org, and possibly to various other software sources, to fetch the correct software to establish the initial user-specified configuration.

It turns out that because of how the network is configured at the server farm in Dallas, Texas where my VPS is located, a lot of these familiar installation images that I’ve used to install Linux operating systems on my own computers without a problem can’t automatically determine the correct way to connect to the network when they’re first started. The hosting provider’s tech support told me I’d have to manually configure the correct network settings, but I didn’t know what commands I had to actually type to make that happen. It turns out you can get a computer science degree like I’ve got without ever covering basic network administration in a class.

I was also confused by some unfamiliar features of the installation image that I got from my hosting provider, which was for an older version of NixOS than I had ever used, and I wondered if that might be contributing to the problem, so I headed over to the #NixOS community IRC channel on freenode and described my problem. Someone there who understands basic networking much better than I do suggested that all I should need to do to get the installer working correctly is add an IP address and default route using two ip commands:

# ip address add 142.11.211.68/24 dev ens3
# ip route add default via 142.11.211.1 dev ens3

I did this, and then for good measure I restarted the NixOS network setup service:

# systemctl restart network-setup.service

Then, after I had actually installed NixOS with a basic configuration, I realized that the IP configuration I put in place hadn’t persisted on the installed system because I hadn’t actually included it in the Nix files that define the system configuration. So I added some code to the main /etc/nixos/configuration.nix file, the one that I don’t keep in source control and that mostly just defines a few things that are very specific to the computer I’m using:

networking.interfaces.ens3.ipv4.addresses = [
  { address = "142.11.211.68"; prefixLength = 24; }
];
networking.defaultGateway = {
  address = "142.11.211.1"; interface = "ens3";
};

Not really the most daunting technical problem one could solve, but it took me hours to work out the solution!

Lesson 2: Directory structure is an important part of defining system configurations

Certain parts of my NixOS system configuration didn’t work right when I first started it up because certain directories weren’t in place. In particular, the custom Nix derivation I wrote for Bibliogram, the Node.js-based alternative Instagram front-end I’ve been running, stored its SQLite database (the cache of images and such fetched from Instagram) into a subdirectory under /var/, but I deployed the configuration at first without actually creating the directory. I had to rebuild the derivation, and added a mkdir -p command into the derivation code so I wouldn’t have to manually create the directory if I were deploying to a fresh system again. Then I found that Bibliogram still wasn’t working properly because when I manually created the directory I gave it the wrong ownership and permissions. The lesson here is that accounting for directory structure is an important part of keeping a NixOS system configuration truly declarative and automated. I’ll have to research the correct or “idiomatic” way to ensure directories exist with the right ownership & permissions in Nix.

Lesson 3: Instagram’s corporate overlords are not to be trusted

I mentioned before that I’ve been running a copy of Bibliogram on my site. It’s very nice piece of software for me because it has allowed me to view the photographs of people who insist on using Instagram instead of a more pleasant alternative like PixelFed, without having to be logged into Instagram or sift through ads or have Instagram rearrange stuff instead of just showing me someone’s posts in chronological order. My site is just one of several that has hosted a running copy of Bibliogram, but it has been the most-used part of my site. Unfortunately, on CentOS I had been running a version of Bibliogram I awkwardly containerized myself, which sometimes halted because I hadn’t given it enough memory or something, didn’t always restart gracefully when that happened, and was not kept up to date very well. I was hoping that because my new NixOS derivation of Bibliogram would make it easier to apply updates and integrate Bibliogram into me system services, I would start seeing less of the broken behavior I was starting to see from my deployment. In particular, I’d started to see a problem where scrolling past the first page of posts on any user profile would fail to bring up more posts, instead loading an eye message about an incomplete response from Bibliogram.

It turns out this broken response was not something I could fix with an updated copy of Bibliogram. As developer Cadence explained on Bibliogram’s issue tracker, Instagram is now generally blocking cloud IP addresses from accessing its API. There is not any code solution for this yet and it may spell the end of publicly-hosted Bibliogram deployments like mine. I’m keeping my copy of Bibliogram online for now, but if Cadence declares (as I probably would) that they can’t or won’t recover from this actively hostile move by Instagram, I’ll sadly accept that it’s over and take bibliogram.hamster.dance down for good.

Lesson 4: So much server software is easier to deploy on NixOS!

Almost anything that has already been defined as a service in the NixOS codebase is shockingly easy to deploy on a NixOS system, often requiring just a few lines in the system configuration files. I had long wanted to try AWStats so I could collect some basic usage statistics about my site without sending visitor data to a third party or embedding unnecessary JavaScript on my site, but the setup process was a little complicated on CentOS. On NixOS I needed just a few lines of Nix code in my system configuration to set it up. Then, when I found that AWStats was displaying IP addresses logged by my web server, Nginx, the NixOS approach to Nginx made it a lot easier to anonymize these IP addresses with ipscrub than it otherwise would have been (although because I had never used custom logging or extra modules with Nginx before, this was still a learning experience for me, with a few false starts that briefly crashed my web server). And now I’m so pleased with the anonymized usage statistics I’m getting that I don’t mind keeping them publicly available.

Even my email server configuration was made easier by NixOS. On CentOS I had Postfix and Dovecot installed for sending and receiving email, and after days of tinkering with their settings got them to work together in an idiosyncratic configuration that allowed me to access my email from a mobile client on my phone, but then I did something or rather (I don’t even know what it was) that reverted my system to a state where I couldn’t remotely access my email this way anymore, and I have up on it. On NixOS, I once again spent a lot of time tinkering with my email server settings to get them just right, but now that they’re correct for my use case I have the confidence of knowing I have them clearly defined in one file in a source code repository so I won’t lose them.

And now that it’s so easy to try out services that the community has defined as NixOS modules, I’m excited to explore some software I hadn’t run on NixOS. I’m trying out Molly Brown for serving a Gemini portion of my web site and may eventually try out other services like Gitea or the Nix flake version of Funkwhale.

I’m excited about what I might build on this new NixOS system!


Tags:

There are no comments on this article.

Leave a comment