## Boring Infrastructure ### Building a secure signing environment <p><img data-src="images/archlinux.png" height="128px"></p> <p>FrOSCon 2024</p> <p>David Runge</p>

### Overview * [Background 🧐](#background-🧐) * [State of the package 📦](#state-of-the-package-📦) * [Motivation 🦾](#motivation-🦾) * [Signstar 💫](#signstar-💫) * [Future work 🌟](#future-work-🌟)

## Background 🧐

### About * Freelance software developer * Arch Linux Package Maintainer (2017)/ Developer (2019)/ Main Signing Key (2021) * Pro-audio, Rust, Python, infrastructure, packaging, installation process

### Obligations *"I use Arch btw"* 🤪

### Pacman * pacman - **pac**kage **man**ager (C) * makepkg - package build tool (Bash)

### Distribution packaging * [PKGBUILD](https://man.archlinux.org/man/PKGBUILD.5) - build script for a package, hosted at https://gitlab.archlinux.org/archlinux/packaging/packages/ * [devtools](https://gitlab.archlinux.org/archlinux/devtools) - collection of tools to build packages in a *"clean chroot"* from PKGBUILD scripts (Bash) * [dbscripts](https://gitlab.archlinux.org/archlinux/dbscripts) - set of scripts to maintain the *binary* package repositories (Bash)

### User repositories * [Arch User Repository (AUR)](https://aur.archlinux.org/) - platform to provide and discuss user provided PKGBUILDs * [Unofficial user repositories](https://wiki.archlinux.org/title/Unofficial_user_repositories) - set of user-maintained *binary* package repositories (often based on PKGBUILDs from the AUR) * [AUR helpers](https://wiki.archlinux.org/title/AUR_helpers) exist to help build from source

### Package repositories * artifacts: prebuilt (binary) package files and repository sync databases (see talks from [FrOSCon 2023](https://media.ccc.de/v/froscon2023-2909-oxidizing_the_arch_linux_packaging_infrastructure) / [ASG! 2023](https://media.ccc.de/v/all-systems-go-2023-207-oxidizing-the-arch-linux-packaging-infrastructure)) * optional: detached OpenPGP signatures for package files and repository sync databases * artifact signing depends on owner of repository * artifact verification based on detached signatures per repository depends on user (`SigLevel` in [pacman.conf](https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING))

<p><img data-src="images/mermaid-repository-interaction.png" height="600px"></p>

### Pacman signature verification 🔏 * system-wide, mutable GnuPG keyring in `/etc/pacman.d/gnupg/` 🙀 * signature verification in the official repositories is based on PGPKI (aka. OpenPGP ["Web of Trust"](https://en.wikipedia.org/wiki/Web_of_trust)) * packager and [trust anchor](https://en.wikipedia.org/wiki/Trust_anchor) certifcates for Arch Linux are maintained in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) * same keyring for all repositories 🥲 * no distinction between verification of sync databases and package files

### State of the package 📦

### Arch Linux Packagers * five [main signing keys](https://archlinux.org/master-keys/) * at least three required to uphold PGPKI * 64 individual [package maintainers](https://archlinux.org/people/package-maintainers/) * access to `[extra]` repository * each requires at least three distinct [third-party identity certifications](https://openpgp.dev/book/certificates.html#third-party-identity-certifications) issued by main signing keys for sufficient trust level in the PGPKI * 24 [developers](https://archlinux.org/people/developers/) with access to `[core]` repository

### Use of OpenPGP * Upstream source verification * Package source verification * Binary package verification * Deployment artifacts (e.g. installation media, virtual machine images) * *Repository sync database verification*

### Packaging workflow * Packagers build packages in a *"clean chroot"* on their own machine or build server and sign them * Packagers are encouraged to use [OpenPGP card](https://wiki.archlinux.org/title/OpenPGP_card) hardware tokens to prevent exfiltration of their private key material * Packagers each have SSH access to the central repository server * Repository sync databases are not signed❗

<p><img data-src="images/mermaid-package-maintainer-access.png" height="600px"></p>

### Other artifacts 📀💻 * Virtual machine images are built automatically and signed in CI * Installation media is build manually by a single developer on a monthly basis, signed and uploaded

<p><img data-src="images/mermaid-virtual-machine-images.png" height="600px"></p>

<p><img data-src="images/mermaid-installation-media.png" height="600px"></p>

## Motivation 🦾

## Prevent key exfiltration 🔐 * Packagers should use hardware tokens and encrypted backups, but we can't know for certain whether they do! * Private key material held in CI can be exfiltrated * Guarding one central system is easier than guarding `n` systems

## Automate all the things 🤖 * Package build automation and mass rebuilds on secluded infrastructure (no direct packager access) * Signing of repository sync databases * Automated building and signing of installation media * Secure signing of virtual machine images

## Signstar 💫

## Requirements 📓 * Security * Maintainability * Robustness (boring is good!) * Integration with existing packaging ecosystem * Standard's compliant implementation of OpenPGP

## Threat model (HSM) - The given HSM is a tamper-proof device, which fully prevents private key exfiltration. - Credentials required for administrative actions towards the HSM can be stored securely offline until they are needed for the (re)configuration of the HSM (e.g. initial setup, or creation of new keys and users). - Backups of the HSM are stored securely offline and allow restoring a device in case of disaster.

## Threat model (Host 1/n) - A dedicated physical host, managing all access to the HSM, is used for its configuration and for granting access to signing operations. - The host is not accessible (e.g. via SSH) by administrators during normal operations, but only via rescue environment. - Two separate, immutable and read-only operating system images are used to segregate administrative credentials from those used for unprivileged actions (e.g. retrieving backups, signing).

## Threat model (Host 2/n) - To add or alter administrative credentials on the host, its rescue system is used and at least two persons are required (one granting access to a rescue system, another to provide a payload), to prevent a single system administrator compromise to lead to a full system compromise. - Credentials for unprivileged actions are rotated on each (re)configuration or update of the system and are never persisted outside of it. - A backup is created after each (re)configuration of the HSM.

## Threat model (Host 3/n) - If an administrator with access to the administrative credentials of the HSM is compromised, the credentials must be rotated and the HSM be reconfigured accordingly. - An immutable, read-only and single-purpose OS image is granted access to (one or more) *Operator* credentials for the HSM and can be (re)configured transparently and reproducibly in a fast manner.

## Threat model (Host 4/n) - A public transparency log is appended to by the signing facilities for each requested signature, which provides insights into all signing operations done by the system. - Transparency log monitoring is done by a system outside of the scope of the Signstar project.

## Threat model (Client 1/n) - Clients to the signing service can each be provided with credentials to the service in a dedicated scope (e.g. hardware backed and service specific) and have no access to the actual *Operator* (or any other) credentials of the HSM. - Some or all signstar clients may run in untrusted environments or may run untrusted or unsafe code (e.g. build servers during build time), which may lead to them being compromised.

## Threat model (Client 2/n) - Known compromised clients can either be shutdown and/ or be disconnected from the signing service by re-configuring the service with altered or removed credentials for the respective clients.

## Hardware Security Module * Nitrokey's [NetHSM](https://www.nitrokey.com/products/nethsm)! 🎉 * [Open-source](https://github.com/nitrokey/nethsm/) * [Documented API](https://nethsmdemo.nitrokey.com/api_docs/index.html) * (auto-generated) [Rust bindings](https://crates.io/crates/nethsm-sdk-rs) 🦀 * [container image](https://hub.docker.com/r/nitrokey/nethsm) for testing purposes

## Custom integration * High-level, extensively documented and tested [nethsm](https://crates.io/crates/nethsm) library * Dedicated command-line tool ([nethsm-cli](https://crates.io/crates/nethsm-cli)) to interact with the NetHSM API (currently minus [namespaces](https://docs.nitrokey.com/nethsm/administration#namespaces)) * [OpenPGP support](https://gitlab.archlinux.org/archlinux/signstar/-/merge_requests/19) based on [rPGP](https://crates.io/crates/pgp) thanks to [Wiktor Kwapisiewicz](https://github.com/wiktor-k/) 🥳

### Network separation <p><img data-src="images/mermaid-signstar-network.png" height="600px"></p>

### Setup Overview <p><img data-src="images/mermaid-signstar-overview.png" height="600px"></p>

### Signing <p><img data-src="images/mermaid-signstar-signing.png" height="600px"></p>

### Deployment <p><img data-src="images/mermaid-signstar-deployment.png" height="600px"></p>

## First goals 🎯 * Signing of repository sync databases * Signing of installation media * Signing of virtual machines * Signing of packages in a custom build service environment

## Next steps 👣 * Implement missing basic features and components * signstar-configure * signstar-request-signature * signstar-sign * Write [Arch Linux RFC](https://rfc.archlinux.page) * Buy device(s) * Deployment

## Future work 🌟

## Missing features * [Namespaces](https://docs.nitrokey.com/nethsm/administration#namespaces) for further privilege separation * Configuration format for configuration and signing tool * Payload format for lightweight signature requests * Dedicated executables for signature request, signing and configuration

## Seeding * Provide administrative credentials based on [Shamir's Secret Sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing) * Minimize lifetime requirement of administrative credentials during runtime

## Deployment * Image-based A/B system for signing service host * Make HSM credentials only available in dedicated boot phases ([systemd-measure](https://man.archlinux.org/man/systemd-measure.1))

## Threat model improvements * Host security and physical access * Administrator compromise * Shared secret compromise

## Attestation log * Hook into signing process * Integrate with existing services (e.g. [rekor](https://github.com/sigstore/rekor))

## Secure boot * Arch Linux does not provide a signed shim, but really should * Signing scheme is somewhat involved and specialized * May require use of dedicated [PKCS#11 driver](https://github.com/Nitrokey/nethsm-pkcs11)

## Threshold signing * Arch Linux has a [reproducible builds](https://reproducible.archlinux.org/) effort * Provide signature only if `n` out of `m` build machines are able to reproduce an identical artifact * Gatekeep reproducibility * profit! 🥳

## ???

## Funding 💰 * Arch Linux is community driven * We are actively looking for sponsors

## Slides https://pkgbuild.com/~dvzrv/presentations/froscon2024/

## Contact * [Signstar on Arch Linux's GitLab](https://gitlab.archlinux.org/archlinux/signstar) * [David Runge \<dvzrv@archlinux.org\>](mailto:dvzrv@archlinux.org) * [#archlinux-projects](ircs://irc.libera.chat/archlinux-projects) on [Libera Chat](https://libera.chat/) or [arch-projects](https://lists.archlinux.org/listinfo/arch-projects) mailing list * [@dvzrv@chaos.social](https://chaos.social/@dvzrv)

## Kthxbye 👋