## Repository management (with repod) <p><img data-src="images/archlinux.png" height="128px"></p> <p>Arch Summit 2023</p> <p>David Runge</p>

### Overview * Motivation * Repod * Further

### Previous presentations * [Hacking in Parallel 2022](https://pretalx.c3voc.de/hip-berlin-2022/talk/HCNL3V/)

## Motivation 🦾

## Dbscripts 🕸️ * Good integration tests! 👍 * Validation in Bash is **hard** (and not pretty) 🥲 * There are no unit or integration tests for metadata files * Repository management with dbscripts is limiting and dangerous (no transactions, no concurrency, no rollback)

### Package repositories * Architecture-specific directory structure * Packages, signatures and repository sync databases * Exposed to user systems by a web server * Pacman downloads files and synchronizes against local state

### What does a repository contain? 👀 A dummy repository called `repo`: ```sh . |-- /srv/ftp/repo | `-- x86_64 | |-- package-1.0.0-1-x86_64.pkg.tar.zst -> ../../pool/package-1.0.0-1-x86_64.pkg.tar.zst | |-- package-1.0.0-1-x86_64.pkg.tar.zst.sig -> ../../pool/package-1.0.0-1-x86_64.pkg.tar.zst.sig | |-- repo.db | `-- repo.files `-- /srv/ftp/pool |-- package-1.0.0-1-x86_64.pkg.tar.zst `-- package-1.0.0-1-x86_64.pkg.tar.zst.sig ```

### Repository sync databases 💽 * Live alongside packages in a repository * Contain information about packages in a repository * Two per repository (metadata, metadata + file info) * Detached signatures may be provided * Used by pacman to install and update packages from a repository

### What do sync databases contain? 👀 Default sync database: ```sh . |-- package-1.0.0-1 | `-- desc [..] ``` Files sync database: ```sh . |-- package-1.0.0-1 | |-- desc | `-- files [..] ```

### Repository sync database metadata * `desc` - information on packages in the version found in a repository * `files` - list of files provided by a package found in a repository

### Desc file example zstd 1.5.5-1: ```text %FILENAME% zstd-1.5.5-1-x86_64.pkg.tar.zst %NAME% zstd %BASE% zstd %VERSION% 1.5.5-1 %DESC% Zstandard - Fast real-time compression algorithm %CSIZE% 493009 %ISIZE% 1500453 %MD5SUM% 2ba620ed7816b97bcad1a721a2a9f6c4 %SHA256SUM% 1891970afabc725e72c6a9bb2c127d906c1d3cc70309336fbe87adbd460c05b8 %PGPSIG% iQEzBAABCgAdFiEE5JnHn1PJalTlcv7hwGCGM3xQdz4FAmQ79ZMACgkQwGCGM3xQdz4V+Qf/Yz7Y+3WwSDKtspwcaEr3j95n1nN5+SAThl/OHe94WwmInDWV09GwM+Lrw6Y1RFDK1PI1ZLON3hOo/81udW0uCHJ4n0bnU/2x3B4UW82dcBqFBjiEqNEF1x6KcQGf9PE9seZndsiAxVzrbEH9u48RIHx0SuwWnzlryCoHPYTgYsPrpkH0IzLUerP2Lc8rjUR2eAKn6zoomb3mR74dPNMn2yx9gS0l+79EshQR8kWtOVvTv7xgRriWeJMBNoTTvDfiDq5B8395vPaBmSfrU0O3tvVF3eDAGtpxIb8hqfhtRqy3XqTcRrYaoj44KtJraGCbq5DrsImEdx5byS7qBhoheQ== %URL% https://facebook.github.io/zstd/ %LICENSE% BSD GPL2 %ARCH% x86_64 %BUILDDATE% 1681646714 %PACKAGER% Jelle van der Waa <jelle@archlinux.org> %PROVIDES% libzstd.so=1-64 %DEPENDS% glibc gcc-libs zlib xz lz4 %MAKEDEPENDS% cmake gtest ninja ```

### Files file example zstd 1.5.5-1: ```text %FILES% usr/ usr/bin/ usr/bin/pzstd usr/bin/unzstd usr/bin/zstd usr/bin/zstdcat usr/bin/zstdgrep usr/bin/zstdless usr/bin/zstdmt usr/include/ usr/include/zdict.h usr/include/zstd.h usr/include/zstd_errors.h usr/lib/ usr/lib/cmake/ usr/lib/cmake/zstd/ usr/lib/cmake/zstd/zstdConfig.cmake usr/lib/cmake/zstd/zstdConfigVersion.cmake usr/lib/cmake/zstd/zstdTargets-none.cmake usr/lib/cmake/zstd/zstdTargets.cmake usr/lib/libzstd.so usr/lib/libzstd.so.1 usr/lib/libzstd.so.1.5.5 usr/lib/pkgconfig/ usr/lib/pkgconfig/libzstd.pc usr/share/ usr/share/doc/ usr/share/doc/zstd/ usr/share/doc/zstd/zstd_manual.html usr/share/licenses/ usr/share/licenses/zstd/ usr/share/licenses/zstd/LICENSE usr/share/man/ usr/share/man/man1/ usr/share/man/man1/unzstd.1.gz usr/share/man/man1/zstd.1.gz usr/share/man/man1/zstdcat.1.gz usr/share/man/man1/zstdgrep.1.gz usr/share/man/man1/zstdless.1.gz ```

### Package building 📦 <p> <img data-src="images/diagram_pkg_build.svg"> </p>

### Repository server 🎛️ <p> <img data-src="images/diagram_repo_server.svg"> </p>

### User machine 🖥️ <p> <img data-src="images/diagram_user_machine.svg"> </p>

### Relationships 🤝 * Package building ⬇️ * Repository management ⬇️ * Package management

## Repod <img data-src="images/repod.png" height="128px">

## Repod * [repod](https://gitlab.archlinux.org/archlinux/repod), transaction based and very configurable, to eventually replace dbscripts * fully validating using pydantic models 🐍 * client-server model * generates sync databases via intermediate JSON based state, that contains **all** metadata * provides [repod-file](https://man.archlinux.org/man/repod-file.1) as CLI tool for interacting with packages and local repositories

## -ETOOMANY🐍 * repod required custom parsers/ specs for `.BUILDINFO`, `.MTREE`, `.PKGINFO`, `.SRCINFO`, `desc`, `files` * most file types have a floating spec, that only has a reference implementation (in `makepkg`/ `repo-add`/ `pacman`)

## -ECOMPATIBILITY * previously existing parsers with various degrees of compatibility in several languages (e.g. [python-srcinfo](https://github.com/kyrias/python-srcinfo) and [srcinfo.rs](https://github.com/Morganamilo/srcinfo.rs)) * issues with package metadata found only after publishing them (automated repod job validates all repository sync databases and core packages)

## Further 🌟

## Reuse specifications 📓 * make use of [central specifications](https://gitlab.archlinux.org/archlinux/alpm) for `.BUILDINFO`, `.PKGINFO`, `.MTREE`, `.SRCINFO`, `desc` and `files` * create central Rust-based specifications for the state repository format and use them * remove custom internal specifications

## Future work 🐙 * many [milestones](https://gitlab.archlinux.org/archlinux/repod/-/milestones) left * high test coverage and integration testing is essential * optimization * RiiR?

## Funding 💰 * [Prototype Fund](https://prototypefund.de) tried twice (👎) * [NLnet](https://nlnet.nl) tried twice (👎)

## Contact 📱 <style> .verticalhorizontal { display: table-cell; height: 100px; text-align: center; width: 100px; vertical-align: middle; } </style> <table border="0px" text-align="center" vertical-align="middle"><tr><td class="verticalhorizontal"><a href="https://gitlab.archlinux.org/archlinux/repod">gitlab.archlinux.org/archlinux/repod</a></td><td class="verticalhorizontal"><img data-src="images/website.png" height="99px"></td></tr></table> <table border="0px" text-align="center" vertical-align="middle"><tr><td class="verticalhorizontal"><p><a href="mailto:dvzrv@archlinux.org">dvzrv@archlinux.org</a></p></td><td class="verticalhorizontal"><img data-src="images/email.png" height="99px"></td></tr></table> <a href="ircs://irc.libera.chat/archlinux-projects">#archlinux-projects</a> on <a href="https://libera.chat/">Libera Chat</a> or <a href="https://lists.archlinux.org/listinfo/arch-projects">arch-projects</a> mailing list