diff --git a/_drafts/journald-1.md b/_drafts/journald-1.md new file mode 100644 index 0000000..4d03357 --- /dev/null +++ b/_drafts/journald-1.md @@ -0,0 +1,61 @@ +--- +layout: post +title: Jumping into journald +date: 2021-05-31 +--- + +On many Linux systems, systemd-journald runs as a daemon at boot and collects your logs. You can access +them through journalctl but it turns out journald is a lot more complicated then just sending something to a text file. + +While you'll mostly see entries as a terse error message on one line, journald stores a lot more information. +For example, here's an error from my audio server, [pipewire](https://pipewire.org/). Note that some fields are reordered to make it +easier to explain +```shell +artemis@starlight ~> journalctl --user -xeu pipewire.service -o export +... + +_BOOT_ID=eba612c42f634b58b0484c42756ba712 +_UID=1000 +_GID=1000 +_CAP_EFFECTIVE=0 +_MACHINE_ID=b3bee68a0f884e6c982529efec408a61 +_HOSTNAME=starlight +_TRANSPORT=journal +_SELINUX_CONTEXT=kernel +_AUDIT_SESSION=2 +_AUDIT_LOGINUID=1000 +_SYSTEMD_OWNER_UID=1000 +_SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/session.slice/pipewire.service +_SYSTEMD_UNIT=user@1000.service +_SYSTEMD_SLICE=user-1000.slice +_SYSTEMD_USER_SLICE=session.slice +_SYSTEMD_USER_UNIT=pipewire.service +_PID=4046 +_COMM=pipewire +_EXE=/nix/store/4qp4npwqabf3mnsy230w3z1nqdjl1gxr-pipewire-0.3.26/bin/pipewire +_CMDLINE=/nix/store/4qp4npwqabf3mnsy230w3z1nqdjl1gxr-pipewire-0.3.26/bin/pipewire +_SYSTEMD_INVOCATION_ID=a9477826e73747ac810f958f894c302e +_SOURCE_REALTIME_TIMESTAMP=1622336991484040 +__CURSOR=s=ac8b24ddf2634038b49168edc5d6e544;i=b4233a3;b=4950abcba7ec46629fce878fb239a6e6;m=1408502e8be;t=5c381c41568ad;x=2af33cc0b675a511 +__REALTIME_TIMESTAMP=1622336991488173 +__MONOTONIC_TIMESTAMP=1376621095102 +PRIORITY=4 +SYSLOG_IDENTIFIER=pipewire +CODE_FILE=../src/pipewire/impl-node.c +CODE_LINE=957 +CODE_FUNC=dump_states +MESSAGE=(PipeWire ALSA [.electron-wrapped]-110) client too slow! rate:256/48000 pos:4451017216 status:triggered +``` + +An entry consists of freeform variables with binary (though generally ASCII/US english) values. Values starting with an underscore +are "trusted" and generated by journald while others are sent by the process along with the primary message. This helps provide +context about what exact process failed and what state it was in during that failure. Unfortunately, the [official descriptions](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html) of what these fields mean can be a bit obtuse. + +While working on my prototype for a system-journald replacement, [rjournald](https://github.com/artemist/rjournald) I've discovered what many of these mean through context or reading the systemd code: +- **_BOOT_ID** is a UUID that changes every time you start your system. This is generated by the Linux kernel and is accessable at `/proc/sys/kernel/random/boot_id`. I've found it useful to help figure out if I've rebooted my system since an error occured +- **_UID** tells you what user executed the process. You can get this for your user with the command `id`. +- **_GID** tells you which group the process was using. While a user can have several groups, a process executes under one primary group ID +- **_CAP_EFFECTIVE** provides what [capabilities](https://linux.die.net/man/7/capabilities) a process can use. Capabilities give fine-grained privelaged access to processes without requiring them to be the root user. For example, binding to port 80 or 443 requires the CAP_NET_BIND_SERVICE capability. If _CAP_EFFECTIVE=0 then you know you've missed that capability. +- **_MACHINE_ID** is a unique ID to the system which you can find in `/etc/machine-id`. This should be set on the first boot of your system by systemd and helps you figure out if the logs could be from before a system was reinstalled +- **_HOSTNAME** is the name of the system. You probably set this in install. I'm not quire sure where systemd gets this, as there's multiple different places to set the hostname, including systemd-hostnamed (which lets you set non-ascii hostnames for some programs), `/etc/hostname` (which is where systemd will read your hostname and tell it to the kernel), and `/proc/sys/kernel/hostname` (what the kernel thinks your hostname is). +- **_TRANSPORT** How the log got to journald. As of writing I could find 6 transports: journal (using the native journald protocol), stdout (a process's standard output or error redirected to systemd), syslog (the legacy linux logging system), kernel (kernel messages you can get through the `dmesg` command), audit (logs the kernel generates about programs' activities), and driver (error messages from within journald). I'll discuss a few in more depth later