Arch Linux images with mkosi

count: false

# Arch Linux images with mkosi

### Should Arch Linux move to mkosi for all our image artifacts?

#### Jelle van der Waa [<jelle@archlinux.org>](mailto:jelle@archlinux.org)

---

# Overview

* $ whoami / definitions
* porting to mkosi
* reproducible images
* continuous integration
* conclusion

---

# $ whoami

* Arch Linux Developer since 2014
* DevOps team member since 2018
* Software Engineer @ Red Hat
* Interested in Reproducible Builds & Packaging

???

* Started packaging, end up with too many hats

---

# Arch Linux

* Rolling release distribution
* Pacman package manager
* Packaging uses sysusers/tmpfiles.d 
* Adopted systemd early on

???

---

# mkosi

* Started by Lennart Poettering
* Currently maintained by Daan de Meyer & Jörg Behrmann
* `.ini` based configuration format
* Rootless building of images (namespaces)

---

# Alternatives

* osbuild
* kiwi
* both support Arch Linux

---

# Images

* VM image (40Gb)
* Cloud-init enabled image
* Vagrant images 
	* libvirt
	* virtualbox backend
* Archiso
* iPXE
* Container image
	* hosted on dockerhub & quay

???

---

# Current situation

* Self written Bash scripts
* Releases every two weeks
* Requires root privileges
* Build & tested in a GitLab custom libvirt runner

https://gitlab.archlinux.org/archlinux/arch-boxes

???

---

# Porting to mkosi

* mkosi.conf - package definitions
* mkosi.extra - custom files
* mkosi.repart - custom partitioning

???

* Package definitions, required packages, can be overlayed in profiles
* extra, default systemd

---

# Porting to mkosi

* Profiles
   * vagrant
   	* libvirt
   	* virtualbox
   * images
   	* basic
   	* cloud-init

---

# Challenges

* btrfs swapfile creation support
* installing a user authorized_keys

???

* not possible to chown as user when not using root

---

# Output

* mkosi only supports outputting a raw disk image
* mkosi.postout scripts to get the desired output format image

---

# Output example Vagrant

mkosi.postout
```bash
#!/bin/bash

IMAGE_NAME="Arch-Linux-x86_64-libvirt-latest.box"

cp "${SRCDIR}/Vagrantfile.libvirt" "${OUTPUTDIR}/Vagrantfile"

qemu-img convert -f raw "${OUTPUTDIR}/${IMAGE_ID}.raw" \
         -O qcow2 "${OUTPUTDIR}/box.img"
DISK_SIZE=$(qemu-img info --output=json "${OUTPUTDIR}/box.img" | \
            jq '."virtual-size" / 1024 / 1024 / 1024 + 0.5 | round')

echo '{"format":"qcow2","provider":"libvirt","virtual_size":'"${DISK_SIZE}"'}' \
     > "${OUTPUTDIR}/metadata.json"
tar -C "${OUTPUTDIR}" -czf "${OUTPUTDIR}/${IMAGE_NAME}" \
       Vagrantfile metadata.json box.img
```

---

# Reproducible Builds

> “A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts.”

https://reproducible-builds.org

---

# Reproducible Arch Linux Packages

* Started in 2019
* 90% Reproducible so far
* Reproducible minimal container image

https://reproducible.archlinux.org

https://lists.reproducible-builds.org/pipermail/rb-general/2024-March/003301.html

---

# Reproducibility issues

* Timestamps (SOURCE_DATE_EPOCH)
* Recorded hostname, version information, kernel used etc.
* Locale, sorting, build paths
* Installation scriplets

???

---

# Reproducible images

* Some prior work here done by [edgelesssys](https://github.com/edgelesssys/reproducible-mkosi)
* SourceDateEpoch option - clamp mtimes
* Seed option - reproducible partition UUID's

---

# Reproducible images

* Start analyzing images with diffoscope

---

# Reproducible images

```bash
diffoscope imageA.raw imageB.raw
```

---

# Reproducible images

```bash
diffoscope imageA.raw imageB.raw
```

```
[11111.1] Out of memory: Kill process 2603 (diffoscope) score 761 or sacrifice child
```

---

# Reproducible images

* Reduce the problem scope
* Output the image to a directory
* Diffoscope the directories

---

# Reproducible images

```bash
export SOURCE_DATE_EPOCH=1662046009
mkosi -d arch -p systemd --format directory \
      --source-date-epoch $SOURCE_DATE_EPOCH \
      -o foo \
      -m https://archive.archlinux.org/repos/2024/06/30/
mkosi -d arch -p systemd --format directory \
      --source-date-epoch $SOURCE_DATE_EPOCH \
      -o bar \
      -m https://archive.archlinux.org/repos/2024/06/30/
sudo diffoscope foo bar --html-dir output
xdg-open output/index.html
```

---

# Reproducible images

Two differences with a systemd package:

* var/cache/ldconfig/aux-cache (glibc)
* var/lib/pacman/local

```
20	%INSTALLDATE%   20	%INSTALLDATE%
21	1722251133      21	1722251028
```

---

# Reproducible images

lib/libalpm/add.c
```c
	/* make an install date (in UTC) */
	newpkg->installdate = time(NULL);
```

Workaround
```bash
sed -i -e '/INSTALLDATE/{n;s/.*/0/}' \
          "${BUILDROOT}"/var/lib/pacman/local/*/desc
```

Patch
https://gitlab.archlinux.org/pacman/pacman/-/merge_requests/213

---

# Reproducible bootable image

```bash
mkosi -d arch -p systemd -p linux -p base -p grub \
              -p openssh -p sudo -p reflector \
	      -p btrfs-progs -p udev \
	      --source-date-epoch 1662046009 \
	      --format directory -o bar \
	      -m https://archive.archlinux.org/repos/2024/06/30/
```

---

# Reproducible bootable image

* UKI unreproducible
	* diffoscope can't unpack EFI binaries
* systemd-boot
	* efi/loader/random-seed unreproducible (duh!)

???

* UKI => leftover /var/cache/ldconfig/aux-cache
* random-seed has been removed in mkosi
* ukify-inspect is a useful tool

---

# Reproducible filesystems

* Filesystems are created by systemd-repart
* Some mkfs programs support creating a filesystem with files

```bash
mkfs.btrfs --rootdir image-contents test.img
```

#### vfat
```bash
mkfs.vfat test.img
mvcopy -s -p -Q -m -i test.img boot efi ::
```

???

Creating a filesystem directly with the files included can be done without mounting so unprivileged and possibly easier to make reproducible

---

# Reproducible filesystems testing

```bash
export SOURCE_DATE_EPOCH=0

fallocate test1.img
fallocate test2.img

mkfs.ext4 test1.img
mkfs.ext4 test2.img
```

```bash
diff <(dumpe2fs test1.img) <(dumpe2fs test2.img)
< Filesystem created:       Tue Sep 17 11:02:05 2024
---
> Filesystem created:       Tue Sep 17 11:02:01 2024
```

---

# Reproducible image filesystems

* vfat/ext4/f2fs/erofs can be made reproducible
* btrfs/xfs needs work
* dosfstools requires an upstream patch

https://salsa.debian.org/reproducible-builds/reproducible-website/-/merge_requests/156

---

#  Reproducible image filesystems

* Non-reproducible EFI partition

```bash
drwx------ 1 root root 32 Aug 29 19:59 EFI
drwx------ 1 root root 60 Aug 29 19:59 loader
```

https://github.com/systemd/systemd/pull/34180

---

#  Reproducible images

* Build on GitHub CI irreproducible
* Build a reproducible image locally
* Build on Arch server / Laptop reproducbible (xfs/btrfs)

???

The Arch container uid/gids for /srv/ftp are different from a normal install. This somehow leaks into the image

---

# Reproducible images (bonus)

Almost reproducible Fedora on Arch Linux

???

* Some strange difference in the root partition

---

# Continuous Integration (CI)

* mkosi project provides a GitHub action
* GitLab CI by default uses a container
* Latest mkosi (main) does not build in a privileged container
* Arch boxes project uses a custom libvirt runner

---

# Conclusion

* Easy to port to mkosi
* Reproducible image
* Make mkfs.btrfs reproducible(!)
* ArchISO
	* ISO support needed in systemd-repart

---

# Links

https://github.com/jelly/arch-mkosi-boxes/

---

# Questions