## Boring Infrastructure ### Building a secure signing environment <p><img data-src="images/archlinux.png" height="128px"></p> <p>All Systems Go! 2024</p> <p>David Runge</p>

### Overview * [Background 🧐](#background-🧐) * [State of the package 📦](#state-of-the-package-📦) * [Motivation 🦾](#motivation-🦾) * [Signstar 💫](#signstar-💫) * [Future work 🌟](#future-work-🌟)

## Background 🧐

### About * Freelance software developer * Arch Linux Package Maintainer (2017)/ Developer (2019)/ Main Signing Key (2021) * Pro-audio, Rust, Python, infrastructure, packaging, installation process

### Obligations *"I use Arch btw"* 🤪

### Pacman * pacman - **pac**kage **man**ager (C) * makepkg - package build tool (Bash)

### Distribution packaging * [PKGBUILD](https://man.archlinux.org/man/PKGBUILD.5) - build script for a package, hosted at https://gitlab.archlinux.org/archlinux/packaging/packages/ * [devtools](https://gitlab.archlinux.org/archlinux/devtools) - collection of tools to build packages in a *"clean chroot"* from PKGBUILD scripts (Bash) * [dbscripts](https://gitlab.archlinux.org/archlinux/dbscripts) - set of scripts to maintain the *binary* package repositories (Bash)

### User repositories * [Arch User Repository (AUR)](https://aur.archlinux.org/) - platform to provide and discuss user provided PKGBUILDs * [Unofficial user repositories](https://wiki.archlinux.org/title/Unofficial_user_repositories) - set of user-maintained *binary* package repositories (often based on PKGBUILDs from the AUR) * [AUR helpers](https://wiki.archlinux.org/title/AUR_helpers) exist to help build from source

### Package repositories * artifacts: prebuilt (binary) package files and repository sync databases (see talks from [FrOSCon 2023](https://media.ccc.de/v/froscon2023-2909-oxidizing_the_arch_linux_packaging_infrastructure) / [ASG! 2023](https://media.ccc.de/v/all-systems-go-2023-207-oxidizing-the-arch-linux-packaging-infrastructure)) * (optional) [detached OpenPGP signatures](https://openpgp.dev/book/signing_data.html#detached-signatures) for package files and repository sync databases * artifact signing depends on owner of repository * artifact verification based on detached signatures per repository depends on user (`SigLevel` in [pacman.conf](https://man.archlinux.org/man/pacman.conf.5#PACKAGE_AND_DATABASE_SIGNATURE_CHECKING))

<p><a href="mermaid/repository-packaging.mermaid"><img data-src="images/repository-packaging.png"></a></p>

<p><a href="mermaid/repository-update.mermaid"><img data-src="images/repository-update.png" height="600px"></a></p>

### Pacman signature verification 🔏 * system-wide, mutable GnuPG keyring in `/etc/pacman.d/gnupg/` 🙀 * signature verification in the official repositories is based on PGPKI (aka. OpenPGP ["Web of Trust"](https://en.wikipedia.org/wiki/Web_of_trust)) * packager and [trust anchor](https://en.wikipedia.org/wiki/Trust_anchor) certificates for Arch Linux are maintained in [archlinux-keyring](https://gitlab.archlinux.org/archlinux/archlinux-keyring) * same keyring for all repositories 🥲 * no distinction between verification of sync databases and package files

### State of the package 📦

### Arch Linux Packagers * five [main signing keys](https://archlinux.org/master-keys/) * at least three required to uphold PGPKI * 64 individual [package maintainers](https://archlinux.org/people/package-maintainers/) * access to `[extra]` repository * each requires at least three distinct [third-party identity certifications](https://openpgp.dev/book/certificates.html#third-party-identity-certifications) issued by main signing keys for sufficient trust level in the PGPKI * 24 [developers](https://archlinux.org/people/developers/) with access to `[core]` repository

### Use of OpenPGP * Upstream source verification 👌 * Package source verification ✅ * Binary package verification ☑️ * Deployment artifacts (e.g. installation media, virtual machine images) ✔️ * *Repository sync database verification* 🫠

### Packaging workflow * Packagers build packages in a *"clean chroot"* on their own machine or build server and sign them * Packagers are encouraged to use [OpenPGP card](https://wiki.archlinux.org/title/OpenPGP_card) hardware tokens to prevent exfiltration of their private key material * All packagers have SSH access to the central repository server 😬 * Repository sync databases are not signed❗

<p><a href="mermaid/legacy-package-maintainer-workflow.mermaid"><img data-src="images/legacy-package-maintainer-workflow.png" height="600px"></a></p>

### Other artifacts 📀💻 * Virtual machine images are built automatically and signed in CI * Installation media is built manually by a single developer on a monthly basis, signed and uploaded

<p><a href="mermaid/legacy-vm-images-workflow.mermaid"><img data-src="images/legacy-vm-images-workflow.png" height="600px"></a></p>

<p><a href="mermaid/legacy-installation-media-workflow.mermaid"><img data-src="images/legacy-installation-media-workflow.png"></a></p>

## Motivation 🦾

## Prevent key exfiltration 🔐 * Packagers should use hardware tokens and encrypted backups, but we can't know for certain whether they do! * Private key material held in CI can be exfiltrated * Guarding one central system is easier than guarding `n` systems

## Automate all the things 🤖 * Package build automation and mass rebuilds on secluded infrastructure (no direct packager access) * Signing of repository sync databases * Automated building and signing of installation media * Secure signing of virtual machine images

## Goals 🎯 * Signing of virtual machines * Signing of installation media * Signing of repository sync databases * Signing of packages in a custom build service environment

## Signstar 💫

## Requirements 📓 * Security * Maintainability * Robustness (boring is good!) * Integration with Arch Linux's existing packaging ecosystem * Standards compliant implementation of OpenPGP (newsflash: GnuPG is not ⚡)

## Setup 🧪 * [Hardware Security Module](https://en.wikipedia.org/wiki/Hardware_security_module) * Rust based integration 🦀 * Physical host * custom, image-based operating system * strong authentication * coherent, dedicated tooling for all tasks * Client integration * Digital signatures

## Hardware Security Module * Nitrokey's [NetHSM](https://www.nitrokey.com/products/nethsm)! 🎉 * [Open-source](https://github.com/nitrokey/nethsm/) * [Documented API](https://nethsmdemo.nitrokey.com/api_docs/index.html) * (auto-generated) [Rust bindings](https://crates.io/crates/nethsm-sdk-rs) 🦀 * [container image](https://hub.docker.com/r/nitrokey/nethsm) for testing purposes 🫙

## Custom integration * High-level, extensively documented and tested [nethsm](https://crates.io/crates/nethsm) library * Dedicated command-line tool ([nethsm-cli](https://crates.io/crates/nethsm-cli)) to interact with the NetHSM API * OpenPGP support (since nethsm 0.4.0 and nethsm-cli 0.2.0) based on [rPGP](https://crates.io/crates/pgp) thanks to [Wiktor Kwapisiewicz](https://github.com/wiktor-k/) 🥳 * Fully integration tested against upstream container image

## Threat model: HSM 1/n - The given HSM is a tamper-proof device, which fully prevents private key exfiltration. - Backups of the HSM are stored securely offline and allow restoring a device in case of disaster.

## Threat model: HSM 2/n - Credentials required for administrative actions towards the HSM can be stored securely offline until they are needed for the (re)configuration of the HSM (e.g. initial setup, or creation of new keys and users). - A collection of administrative credentials required for the (re)configuration of the HSM are distributed as shares of a shared secret, divided using [Shamir's Secret Sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing) (SSS) to dedicated individuals.

## Physical host 🖥️ * [UEFI](https://en.wikipedia.org/wiki/UEFI): [Secure Boot](https://en.wikipedia.org/wiki/UEFI#Secure_Boot) with [Unified Kernel Image](https://uapi-group.org/specifications/specs/unified_kernel_image/) (UKI) * [TPM2](https://en.wikipedia.org/wiki/Trusted_Platform_Module): Encrypted `/var` partition and secret for [WireGuard](https://www.wireguard.com/) tunnel * Physical connection to one or more HSMs

### Network separation <p><a href="mermaid/signstar-network.mermaid"><img data-src="images/signstar-network.png"></a></p>

## Image-based OS 💽 * [SignstarOS](https://gitlab.archlinux.org/archlinux/signstar/-/tree/c73fcc18489575c7b9474e3638927947328f9b5a/resources/mkosi/signstar) 💫 (based on Arch Linux) * read-only rootfs, LUKS encrypted `/var` using [systemd-repart](https://man.archlinux.org/man/systemd-repart.8) on first boot * A/B updates using [systemd-sysupdate](https://man.archlinux.org/man/systemd-sysupdate.8) * VPN tunnel for diverting logs and metrics over [WireGuard](https://wireguard.com) * Read-only configuration (without passphrases) for HSM * built using [mkosi](https://github.com/systemd/mkosi) (thanks to [Daan](https://github.com/daandemeyer) for the [help](https://github.com/systemd/mkosi/blob/5eab779e6e6b717fb7e533dabc551180f17a49c8/docs/root-verity.md) 🙏)

### Deployment <p><a href="mermaid/signstar-deployment.mermaid"><img data-src="images/signstar-deployment.png"></a></p>

### Provisioning <p><a href="mermaid/signstar-hsm-provisioning.mermaid"><img data-src="images/signstar-hsm-provisioning.png"></a></p>

### Logging <p><a href="mermaid/signstar-logging.mermaid"><img data-src="images/signstar-logging.png" height="600px"></a></p>

### Metrics <p><a href="mermaid/signstar-metrics.mermaid"><img data-src="images/signstar-metrics.png" height="600px"></a></p>

## Strong authentication 📨🦾 * OpenSSH * hardened configuration (clients with TPM2 backed keys, hello! 👋) * system-wide `authorized_keys` * forced commands instead of login shell

### Authentication overview <p><a href="mermaid/signstar-authentication.mermaid"><img data-src="images/signstar-authentication.png"></a></p>

## Force commands ```sshd_config Match user signer AuthorizedKeysFile /etc/ssh/signer.authorized_keys ForceCommand /usr/bin/signstar-download-signature ```

## Dedicated tooling 🪛🔨🖥️⬇️ * `signstar-download-signature`: Download signature for message * `signstar-download-backup`: Download encrypted HSM backup * `signstar-download-key-certificate`: Download key certificates (public keys)

## Dedicated tooling 🪛🔨🖥️⬇️ * `signstar-download-metrics`: Download HSM device metrics * `signstar-download-secret-share`: Download share of a secret divided using [Shamir's Secret Sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing) (SSS) * `signstar-download-wireguard`: Download public key of [WireGuard](https://www.wireguard.com/) setup

## Dedicated tooling 🪛🔨🖥️⬆️ * `signstar-upload-backup`: Upload a backup file to restore the HSM to * `signstar-upload-secret-share`: Upload HSM administrative credentials as shares of a secret divided using SSS * `signstar-upload-update`: Upload HSM firmware updates

## Dedicated tooling 🪛🔨🖥️🔄 * `signstar-configure-user`: Create and configure users for SignstarOS during image creation * `signstar-configure`: Configure per-user configurations and the HSM in a running SignstarOS based on provided administrative credentials

## Threat model: Host 1/n - A dedicated physical host managing all access to the HSM is used for its configuration and for granting access to signing operations. - The host is not directly accessible by system users of the image-based OS (i.e. via login shell over SSH) during runtime, but only via rescue environment. - Dedicated host credentials are used to collect the public key for a WireGuard connection, which is setup during first boot and is used exclusively for securely diverting system logs and metrics to a hardcoded host.

## Threat model: Host 2/n - A custom image-based operating system with a read-only root filesystem and an encrypted `/var` partition (with key enrolled to local TPM2 chip) is used on the host. Relevant secrets such as SSH host keys and current Operator user credentials are kept in the encrypted partition, preventing offline exfiltration on harddrive theft or copy.

## Threat model: Host 3/n - The cryptographic key material used for secure boot and verity signing of the OS image, as well as that used for signing OS image artifacts are kept in offline backups and are only used with dedicated hardware tokens when building images of the OS image. A compromise of the holder of either or both secure boot/verity signing or artifact signing key may lead to the creation of a compromised OS image. It is therefore advised to ensure proper deployment of update artifacts using quality gates.

## Threat model: Host 4/n - Administrative credentials for the configuration of the HSM are provided to the host OS as shares of a shared secret, divided using SSS. They are held in tmpfs and removed once the administrative action is finished or a predefined timeout is reached.

## Threat model: Host 5/n - Passphrases for unprivileged user actions on the NetHSM are rotated on each (re)configuration of the system and are never persisted outside of the host. - A backup is created after each successful (re)configuration of the NetHSM.

## Kitteh promise, we're halfway there! 🐈🤞

## Threat model: Host 6/n - If a shareholder of the shared secret to the administrative credentials of the HSM is compromised, loses access to the share or leaves the organization, a reconfiguration of the administrative credentials takes place, rotating all credentials and making new shares for a shared secret available for download to current shareholders.

## Threat model: Host 7/n - If `n` out of `m` shareholders are compromised, an attacker needs physical access to the NetHSM or the backups to be able to either remove keys or use them for cryptographic operations.

## Threat model: Host 8/n - During runtime the OS grants access to unprivileged user credentials (e.g. *Operator*, *Metrics* and *Backup*) of the HSM to pre-configured system users. - If credentials for a pre-configured system user account are compromised, an attacker may use the account solely in the context of its role (e.g. to sign artifacts, to retrieve metrics or encrypted backups). - Credentials for the host's system users can be changed transparently, reproducibly and in a fast manner by upgrading the host's OS.

## Threat model: Host 9/n - Logs and metrics of the host are aggregated securely and are accessible for continuous monitoring of the OS and its critical facilities. - A public transparency log is appended to by the signing facilities for each requested signature, which provides insights into all signing operations done by the system. - Transparency log monitoring is done by a system outside of the scope of the Signstar project.

# 🍦

## Client host 💻 * [UEFI](https://en.wikipedia.org/wiki/UEFI): [Secure Boot](https://en.wikipedia.org/wiki/UEFI#Secure_Boot) with [Unified Kernel Image](https://uapi-group.org/specifications/specs/unified_kernel_image/) (UKI) * [TPM2](https://en.wikipedia.org/wiki/Trusted_Platform_Module): Hardware backed SSH key

## Client tooling 🪛🔨💻⬆️ * `signstar-request-signature`: Create custom lightweight payloads for messages, to request signatures for

### Signing <p><a href="mermaid/signstar-signing.mermaid"><img data-src="images/signstar-signing.png"></a></p>

## Threat model: Client 1/n - Clients to the signing service can each be provided with credentials to the service in a dedicated scope (e.g. hardware backed and service specific) and have no access to the actual *Operator* (or any other) credentials of the HSM. - Some or all signstar clients may run in untrusted environments or may run untrusted or unsafe code (e.g. build servers during build time), which may lead to them being compromised.

## Threat model: Client 2/n - Known compromised clients can either be shutdown and/ or be disconnected from the signing service by re-configuring the service with altered or removed credentials for the respective clients.

## Digital signature 📝 - OpenPGP for now - Maybe SSH and/ or PKCS#7 in the future?

## Threat model: DS 1/n - Digital signatures contain metadata about the environment in which they were created. This helps in identifying signature requester, signed payloads and information about the involved hosts. - Digital signatures are requested by designated clients to the service host.

## Threat model: DS 2/n - If the client host or its credentials for the service host are compromised, either the client host is deactivated and/ or its credentials for the service host are removed or changed. All identified malicious signatures are gathered and affected artifacts are rebuilt or resigned.

## Threat model: DS 3/n - Depending on severity of the breach, all artifacts in direct circulation still signed by the affected key can be rebuilt and signed by a different key. Afterwards the affected key material or its trust level can be revoked, ensuring that end-users no longer trust signatures made by this key. Finally, a public report lists all known affected artifacts and signatures and outlines the chosen mitigation steps.

## Future work 🌟

## Next steps 👣 * Implement missing basic features and components * Write [Arch Linux RFC](https://rfc.archlinux.page) * Buy device(s) * (Test) deployment * Nice to have

## Missing features * Configuration format for configuration and signing tool * Finish payload format for lightweight signature requests * Logging and metrics integration * Dedicated executables for signature request, signing and configuration

## Threat model improvements * Host security and physical access * Administrator compromise * Shared secret compromise

## Attestation log 🪵 * Hook into signing process * Integrate with existing services (e.g. [rekor](https://github.com/sigstore/rekor))

## Secure boot * Arch Linux does not provide a signed shim, but really should * Signing scheme is somewhat involved and specialized * May require use of dedicated [PKCS#11 driver](https://github.com/Nitrokey/nethsm-pkcs11)

## Threshold signing * Arch Linux has a [reproducible builds](https://reproducible.archlinux.org/) effort * Provide signature only if `n` out of `m` build machines are able to reproduce an identical artifact * Gatekeep reproducibility * profit! 🥳

# 😽

## Slides https://pkgbuild.com/~dvzrv/presentations/all-systems-go-2024/

## Contact * [Signstar on Arch Linux's GitLab](https://gitlab.archlinux.org/archlinux/signstar) * [David Runge \<dvzrv@archlinux.org\>](mailto:dvzrv@archlinux.org) * [#archlinux-projects](ircs://irc.libera.chat/archlinux-projects) on [Libera Chat](https://libera.chat/) or [arch-projects](https://lists.archlinux.org/listinfo/arch-projects) mailing list * [@dvzrv@chaos.social](https://chaos.social/@dvzrv)

## ???

## Kthxbye 👋