Cron, Running Podman containers with systemd, Awesome Selfhostables: ttftw 2023w13

Three things from this week.

When you’ve got a home server or a lab computer that you want to keep on all the time there are a lot of ways to set up the jobs you want it to do. My current lab machine is a pretty capable 5th gen AMD Ryzen 7. I decided to try Debian Testing as the OS and I installed with a desktop environment even though I expect that most of the time I won’t be using that interface.

The three things on my mind this week are all about some basic choices to set up this kind of a server. First I’ll look at how I run periodic jobs, then long running or always-on services, and finally I’ll share some of the stuff I want to try out on my own home server.

This week’s ttftw got way more into the weeds than I usually do. But if you like it you’ll like it.

Running any old program as a cron job

Cron is so basic and functional that it feels like there must be something wrong with just relying on it as-is. When it works it works. It can be hard to debug when it doesn’t. Logs from cron used to show up under /var/log/messages but that’s all very old-timey now. Instead on modern systemd you now use journalctl with some random gibberish flags that you have to copy and paste from a forum because they change every week. Right now it seems to be:

sudo journalctl --system

...

Mar 23 19:00:01 lapis CRON[5452]: pam_unix(cron:session): session opened for user robr(uid=1000) by (uid=0)
Mar 23 19:00:01 lapis CRON[5453]: (robr) CMD ((python3 /home/robr/somescript.py) >> /home/robr/cronrun.txt 2>&1)

...

Mar 26 13:30:01 lapis CRON[18126]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Mar 26 13:30:01 lapis CRON[18127]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin/invoke-rc.d anacron start >/dev/null; fi)
Mar 26 13:30:01 lapis CRON[18126]: pam_unix(cron:session): session closed for user root
Mar 26 13:33:57 lapis systemd[1]: Started anacron.service - Run anacron jobs.
Mar 26 13:33:57 lapis anacron[18132]: Anacron 2.3 started on 2023-03-26
Mar 26 13:33:57 lapis anacron[18132]: Normal exit (0 jobs run)
Mar 26 13:33:57 lapis systemd[1]: anacron.service: Deactivated successfully.

...

Couple notes here:

1. I didn’t use a filter so there’s a mountain of logs. A simple --unit filter wouldn’t show me all of the logs that are tagged CRON, anacron, and the systemd. Using journalctl --boot restricts to the current boot at least. Press g for the pager to jump to the start and shift-g jumps to the bottom. Search with /, use n for next match and p for previous match.
2. I didn’t use --user. Running journalctl --user may show things specific to your user but I don’t think cronjobs specifically show up there.
3. My cronjob includes >> /home/robr/cronrun.txt 2>&1. That’s an attempt to dump output to a file called cronrun.txt in my home directory. I think it works sometimes.

Create a cron job by running crontab -e, this edits the current user’s cron table. The structure of the crontab file is described in man 5 crontab (or tldr crontab for easy ones). Depending on your setup you might not want to run all your jobs as the user you usually log in as. Also remember that the user’s environment variables might not be present and ~ doesn’t get expanded to the user’s directory. Lines in the crontab can be comments, environment settings, or cron commands. Cron commands are explained most easily with this diagram that everyone copies:

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <command to execute>

Those first five fields can sometimes be replaced with more readable tokens like @daily or the once-per-boot @reboot. There could be differences in some cron implementations so it helps to double-check the man page on your system since they should match whatever is actually installed.

The <command to execute> part is actually run using /bin/sh, even though your user’s shell might be bash or something else. This week I found that when I had the command python foo in there my job failed to run. When I changed it to (python foo) then things worked… I stopped debugging at that point. Usually I make a little wrapper so that the command to execute doesn’t need to have any arguments or rely on much in the environment.

Testing Cron commands

There are a ton of websites out there that will help create the cron expression to schedule your job. Suppose you want a way to mess around with those expressions on your own machine? Well cron doesn’t include any good way to test out the language that it uses but there are some libraries out there. There’s a little Python one called cron-converter.

pip install cron-converter

Then you can give a string like '0 10 * 3,5,7 1,3' and see, for example, the next 10 times when that cron expression should run.

$ python -c "from cron_converter import Cron; c=Cron('0 10 * 3,5,7 1,3'); s=c.schedule(); print('\n'.join(['{:%A, %B %d %H:%M:%S}'.format(s.next()) for i in range(10)]))"
Monday, March 27 10:00:00
Wednesday, March 29 10:00:00
Monday, May 01 10:00:00
Wednesday, May 03 10:00:00
Monday, May 08 10:00:00
Wednesday, May 10 10:00:00
Monday, May 15 10:00:00
Wednesday, May 17 10:00:00
Monday, May 22 10:00:00
Wednesday, May 24 10:00:00

Don’t forget that cron uses UTC. If your timezone is UTC-5 then add 5 hours to your local time when you make the schedule. Also, cron may take special action when the system clock changes that look like DST updates of less than 3 hours.

Finally, as the man page says, the crontab syntax does not make it possible to define all possible periods one can imagine. To have a task run in a time period that cannot be defined using crontab syntax, have the program itself check the date and time information and continue execution only if the period matches the desired one.

Using podman containers with systemd

Cron is a simple way to run a script on a given schedule. My lab machine is meant to run a bunch of ongoing services too. Think things like a personal news aggregator, a file backup server, maybe a git server, personal internal web servers or things like that. Last time I put together a machine with that kind of variety I installed everything directly on the machine’s OS. Sometimes there could be conflicts between different versions of libraries. It was hard to upgrade the OS too. These are the kinds of problems that OCI containerization (a.k.a. docker containers) can help with. And loads of open source projects treat OCI containers as a first-class installation option.

If I were running this on a cluster I’d use Kubernetes but with a single machine I can use Docker Desktop or Podman. Lately I’ve been steering more toward Podman. Just like with Docker Desktop, you can start up a container with a command that pulls an image from a remote server. Unlike docker it doesn’t assume the domain name for the registry. So a simple demo to run a web server might look like this command adapted from Podman’s “Getting Started” guide:

podman run --name=webby --detach --publish 38988:80/tcp docker.io/library/httpd

You can see the container is running with podman ps. You can also see containers that are still around but aren’t running with podman ps --all. To stop and delete this container I’d use:

podman stop webby
podman rm webby

Run this and visit localhost:38988 and you’ll get the “It works!” from Apache webserver, using the latest tagged image on Docker Hub. Using podman run is an easy way to pull the image and run a server in a container. Then the container sticks around but when the computer reboots the server needs to run again. Enter our friend systemd.

Podman has a handy subcommand generate which will create boilerplate like systemd unit files or the YAML for a Kubernetes pod. So if I wanted to run that webserver all the time I’d use podman generate systemd to make the systemd unit file then install and activate it. The name I included in the earlier example makes it easier to follow and easier to upgrade. Something like this:

podman generate systemd --files --name webby

This will create a file in the current directory with a name like container-webby.service. The one I just made has this in it

# container-webby.service
# autogenerated by Podman 3.4.4
# Sun Mar 26 11:04:53 PDT 2023

[Unit]
Description=Podman container-webby.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/mnt/wslg/runtime-dir/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start webby
ExecStop=/usr/bin/podman stop -t 10 webby
ExecStopPost=/usr/bin/podman stop -t 10 webby
PIDFile=/mnt/wslg/runtime-dir/containers/overlay-containers/ef227d0c18e5709f5a450d6b41e4afb5181590856c58edbb2f34c30531725c7d/userdata/conmon.pid
Type=forking

[Install]
WantedBy=default.target

Since I’ve decided to run these jobs as my own user for now I’m installing them using systemd --user. I might change my mind about that but it means that in general I should expect services to run only when my user is logged in. Maybe that means I want to enable an auto-login for my user. After my user logs out the services keep running because I used loginctl enable-linger as suggested in the podman-generate examples. That page is a good reference with some more detailed explanations than I’m going to get into.

Here’s how I install a service file so that it lands in ~/.config/systemd/user where systemd wants user files (at least where it wants them this week).

mkdir -p ~/.config/systemd/user
mv container-webby.service ~/.config/systemd/user/
systemctl --user unmask container-webby.service
systemctl --user enable container-webby.service

The output from enabling the service should indicate a new symlink from default.target.wants/container-webby.service to container-webby.service under ~/.config/systemd/user/.

Tearing down the service and reversing the installation should be approximately the reverse of the commands. Masking isn’t necessary, I just run unmask as a habit before enabling.

systemctl --user disable container-webby.service
rm ~/.config/systemd/user/container-webby.service

There are a lot of ways this could be refined. One interesting detail I came across while writing is that podman-generate can spin up a fresh container every time with --new. This increases the overlap with one-shot tasks I’m handling with cron. It’s also possible to generate configs for a pod with several containers included.

With some time I’ll get more practice and I’ll have to decide if I want to build things in the form of a set of services managed by systemd, a few pods managed primarily from Podman, or if I do decide that I want to move to running things under Kubernetes after all. It takes some patience and some planning to keep things manageable, useful, and predictable. As things stand, I really value being able to start up a service, try it out, and tear it down. The approach I’ve got now strikes a good balance for me but if you want to follow my route I’d suggest that personal previous experience is an important input when deciding how to set up services on your own lab machines.

With all that in mind, where to find some good suggestions about what to run?

The Awesome Selfhosted list

Lots of “Awesome” lists have sprung up over the past few years and some are more awesome than others. The The Awesome Selfhosted list has some legit awesome projects in there. When I skimmed the headings I found some good reminders of why I wanted to run my own home server again. Some of the things I like are also in the Awesome Sysadmin.

Here are some of the services I’m looking forward to checking out, in no particular order

• Gitea or another frontend to a git code host
• Tiny Tiny RSS to catch some RSS feeds
• ArchiveBox to cache the stuff I’ve read instead of leaving a million browser tabs open
• Paste bins like dpaste or SnyPy
• Network infra beyond dnsmasq or a Pi-hole. Both are great but I might want to set up some more general VPN and DNS along with a DHCP server outside of the home routers.
• Datasette from Simon Willison
• Home Assistant worked out great in my experiments so far, now I’d like build a more complete integration with it.

Then there are a couple categories that will just take some deeper thinking and patient evaluation, like some kind of home media management and a wiki or note-taking system. There are just a lot of choices so I want to think a while about what I actually want from that kind of software.

Next time

Writing three things a week has been fun but these things all turned out to be rather large things. Next week look forward to three much smaller things. Maybe I’ll just give some reviews on TikTok clips or something.

• ttftw