Add post about my migration to ARM

pages/01.blog/2022-11-28_raspberry_pi_server/item.md

@@ -0,0 +1,241 @@
+---
+title: Migrating to Raspberry Pi servers
+taxonomy:
+  tag:
+    - NixOS
+    - Raspberry Pi
+    - ARM
+date: '2022-11-28 00:00'
+summary:
+  size: 200
+---
+
+# Migrating from x86 to ARM (Raspberry Pi)
+
+I’ve run my services on used Dell and HP servers for a few years now. I’ve
+decided a few months ago that I want to replace my x86 Dell server with a bunch
+of Raspberry Pis.
+
+## Reasoning
+
+### Noise
+
+I live in a small apartment and the server is stationed in my living room. For
+a long time I had the server hidden under my couch. This worked but the noise
+was quite notable. A few years ago I bought a rack which is specifically
+designed to block noise which helped quite a bit. However it still produced
+some noise because of the three fans in its walls. In addition I thought that it
+might still leak some sound from the server fans but this turned out to be not
+true. The volume stayed the same after I powered of the server.
+
+So I hoped that I would be able to power off the fans once I switched to
+passively cooled Raspberry Pis. As it turned out this isn’t possible. The heat
+inside the rack gets too much without the fans running. The firewall got up to
+over 70 degrees Celsius. It might work out but it’s better for the longevity of
+the devices and the performance to keep the fans running. The Raspis are far
+less likely to throttle when properly cooled. Luckily they are not that loud
+and not as high pitched as the server fans.
+
+### Heat and power consumption
+
+The heat is the more pressing of the two issues. Especially during the summer
+it gets quite warm in my flat, which is under the roof. Adding a server to it
+really doesn’t help to cool it down.
+
+The temperature in my flat is usually around 23 - 24 degrees. With the
+Raspberry Pis it’s now around 21 - 22 degrees. So already quite a bit lower and
+I can really feel the difference in air temperature coming out of the rack. We
+will see how well it works during the hot days. After all the devices inside
+the rack are still producing some heat.
+
+As for the power consumption, I never really measured the Dell server. However
+it was a dual socket system so easily around 100 W. In an initial test a Pi
+used between 3 - 4 Watts of power while idling and a bit under 8 Watts under
+full load. This was before I disabled Bluetooth and WiFi on all of them so they
+should consume now around 2 - 3 Watts in idle[^1] .
+
+### Repairs and transparent costs.
+
+I know you can get used servers fairly cheap or even for free (the Dell machine
+is such a server). However if you really want to be on the safe side you should
+have two. The second one being for spare parts in case something breaks.
+Otherwise you have to be lucky to find the replacement parts when you need
+them.
+
+Granted, with the current situation you have basically the same problem with
+the Pis since they are so hard to get. But if you can get them it’s much easier
+to keep a spare Raspberry Pi around than a whole server.
+
+With my current setup one Pi is running one service. This means that if I want
+to add a new service it's going to cost me about one Raspberry Pi. Which makes
+it much easier for me to decide if it’s worth it or what a replacement might
+cost me.
+
+### Learning
+
+Obviously the setup with the x86 server was much more flexible. I used Proxmox
+as the hypervisor so I just could spin up a new VM, test something and if I
+didn’t need it, I could just destroy it again. The backup and snapshot
+functionally are very helpful as well.
+
+The Raspberry Pi setup doesn’t have this luxury but it forces me to try new
+things and solve the various problems. After all I’m doing this not just to
+improve my privacy but to learn something new as well. I'm especially looking
+forward to do all this with NixOS since it has various ways how you can quickly
+test something without having to install it permanently.
+
+[^1]: https://chipwired.com/raspberry-pi-power-use/
+
+## The setup
+
+### Hardware
+
+I got a very nice rack mount from a friend. It has fourteen slots from which
+I’m currently using ten. One holding the spare Pi. The rack mount is open
+source, the source can be found in the foot note[^2].
+
+[^2]: https://github.com/novski/EMB
+
+![rack.jpg](rack.jpg)
+
+### Software
+
+For the OS I chose NixOS because it is currently the distribution that I like
+using the most. In addition it uses very little resources. This way I don’t
+waste any resources on something I don’t really need.
+
+Currently I’m hosting the following services:
+- Nextcloud
+- TT-RSS + RSS Bridge
+- Docker-Mailserver
+- Grav (this blog)
+- Gitea
+- Plex
+- PiHole
+- Heimdall
+
+#### Docker and Host Services
+
+![diagram.png](diagram.png)
+
+All of these services are running in a Docker container. I might switch some of
+them back to native applications but currently I don't want to mess around with
+applications and their dependencies. What I did however is migrate away from
+`docker-compose` and use the NixOS `virtualisation.oci-containers` option to
+define the containers. A lot of it is very similar to `docker-compose` but it
+is missing a few features. For example, the automatic network configuration is
+missing. To solve this I had to write a `system.activationScript` to create the
+network at boot if it was missing[^3].
+
+In addition I stopped using DBs in containers and migrated back to using the OS
+packages. It’s IMO just easier to administrate and backup them this way. For
+all of my DBs I implement a backup solution with Restic. A systemd service
+triggered by a timer dumps the DB data into Restic via `stdin` and alerts me
+via Telegram if something went wrong [^4].
+The option `--add-host=host.docker.internal:host-gateway` is very important if
+you want to access your database running on the host from a container. With this
+option your database is reachable under “host.docker.internal” from within the
+container. So you don’t have to use the IP which might change.
+
+Since it's very easy to use Nginx with Let's Encrpyt, I stopped using Traefik
+for this task[^5]. Some applications are even using Nginx to forward requests
+to PHP-FPM running inside the container[^6].
+
+Long story short, I only use containers for the application itself. Everything
+else is running on the host configured through Nix. This way I don’t have to
+care too much about the application setup and can update as many packages as
+possible through the system package manager.
+
+[^3]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/nextcloud/default.nix#L78
+
+[^4]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/Restic-server-mysql-client/default.nix
+
+[^5]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/nginx-proxy/default.nix#L12
+
+[^6]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/nginx-fpm/default.nix
+
+#### Raspberry Pi specific changes
+
+Since my Raspberry Pis are running on SD cards (64GB SanDisk High Endurance
+Monitoring) I decided to implement some changes in order to hopefully increase
+the lifespan of the SD cards a bit.
+
+I mount `/var/log` to a `tmpfs` meaning a RAM disk. This way logs don’t get
+written to the SD cards but stay in RAM. With the Raspberry Pi 4s RAM isn’t
+that much of a problem anymore so this is an easy solution. In addition I
+enable `volatile` logs in systemd. As I understood it, this is basically the
+same but for all the logs which get stored in the systemd journal[^7]. The
+drawback is that the logs are lost when you restart the Pi. To help with this I
+send all the logs to my NAS which acts as a syslog server. This has the
+additional advantage that in case someone fucks with my servers and deletes the
+logs, they are stored on a different device as well[^8]. By the way lnav is a
+great way to look through the logs on the NAS[^13].
+
+Another change that I implement is to store as much data as possible on my NAS
+and access it through NFS. Docker can mount NFS shares directly but it is a bit
+of a pain to write the option for it. This works much better with
+`docker-compose`[^9]. I don’t store any database data on the NFS share because
+this might lead to corruption. At some point I might play around with ISCI but
+with my current setup this involves a lot of manual steps which I don’t want to
+do. The data from TT-RSS gets stored on the SD card as well because it uses a
+lock file which doesn’t work with my NFS setup. Luckily it isn’t that much
+data.
+
+All the relevant on device data gets backed up with Restic and stored on my NAS,
+an external harddrive and in an offsite OpenSwift bucket.
+
+I noticed that the clock in the Raspberry Pis doesn’t keep its time after a
+reboot. Therefore I point them all to my firewall which acts as the local NTP
+server for them. In addition I had to make sure that the services are only
+starting after the network is online. Otherwise the logs would report a wrong
+time stamp[^10].
+
+Since the Pis don’t have that much storage and are a bit constrained when it
+comes to power I’m building there config on my notebook and push it to each Pi
+with a simple script[^11]. I had to tweak the order a bit so that the Pi-Hole
+gets updated at the end. It’s not that helpful if the main DNS server goes down
+while you’re updating all the systems :).
+
+The last change that I had to make was building the Docker image for TT-RSS for
+the ARM64 architecture because it isn't provided by the upstream project. I
+hacked it together with Githubt Actions which builds the image once a day[^14].
+
+[^7]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/log-to-ram/default.nix
+
+[^8]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/syslog/default.nix
+
+[^9]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/ebe8adcea559fa7c0c23336ac20bb50854f5fc25/modules/nextcloud/default.nix#L46
+
+[^10]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/5f90eb6068a1da2a460aedc175255e64e3c037a4/modules/common/default.nix#L14-L29
+
+[^11]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/5f90eb6068a1da2a460aedc175255e64e3c037a4/scripts/remote_switch.sh
+
+## Conclusion
+
+All in all I’m very happy with the migration to the Raspberry Pis. They work
+quite well for my use case.
+- The performance with the Nextcloud server could be better but when I compare
+it to my other services this feels to me like it is more a problem with
+Nextcloud than the Pis. It just shows itself on the Pi much more because of the
+limited hardware.
+- Storing the data on NFS causes some latency which is most notable on the Git
+server when browsing the UI after I haven’t accessed the server for a bit.
+- Transcoding a file with Plex is as well not that perfomant but something that
+I can easily fix by transcoding it on my notebook once before I store it on the
+NAS.
+
+Other than that it works really great. I can even build custom images for each
+server so that in case of a SD card failure I can just build the required
+image, flash it to a new SD card and depending on the system restore the DB and
+be ready to go again[^12]. However this is more a feature of NixOS than my Pi
+setup. You can do it with ISOs as well. NixOS was in general a huge help with
+this project because it is so easy to configure and reconfigure a system.
+However it can sometimes take a lot of time to get something working especially
+if it is a software which you haven't used outside of NixOS because you don't
+know if you're missing a config in NixOS or using the application the wrong way.
+
+[^12]: https://git.2li.ch/Nebucatnetzer/nixos/src/commit/5f90eb6068a1da2a460aedc175255e64e3c037a4/flake.nix#L34
+
+[^13]: https://lnav.org/
+
+[^14]: https://github.com/Nebucatnetzer/tt-rss-aarch64/blob/main/.github/workflows/build-image.yml