Arch Linux Infrastructure overview

# Arch Linux Infrastructure overview

How Arch keep's rolling

#### Jelle van der Waa [<jelle@archlinux.org>](mailto:jelle@archlinux.org)

---

# Overview

1. About me
2. Infrastructure overview
3. Monitoring
4. Management
5. Questions

---

# $ whoami

* Joined Arch Linux in 2010
* Developer @ Arch Linux (since 2017)
* Involved in the security/devops team

---

# Server overview

* Six hetzner boxes (~ 8 core, 32-64 GB ram)
* Eight servers donated by PIA

---

# Services overview

---

# Archweb

* Provides the main website
* Build using Django (Python framework)
* To do lists for rebuilds
* Developer dashboard
* Provides several services (mirrors, etc.)

---

# Archweb - Signoffs

![signofss](./img/signoffs.png)

---

# Archweb - To do lists

![signofss](./img/todolist.png)

---

# Mirroring

* Two tier system
* Tier 1 syncs from archlinux.org (rsync)
* Tier 2 syncs from Tier 1

---

# Archweb mirror monitoring

* Periodically fetches lastsync to determine mirrors status
* Pings mirrors from several locations

https://www.archlinux.org/mirrors/kernel.org/212/

---

# Archweb package updates

* A django managment command imports packages into the website
* Feeds the RSS feed
* Used for Developer dashboard
* Todo lists (updated version in [staging])
* Imports PGP keys for displaying on the website

---

# Build server

* 24 core, 128 GB dedicated server
* Provides a buildserver for Developer/Trusted Users
* Ability to provide hosting for public_html for Dev's/TU's
* Second buildserver located in Sinagpore (sgp.mirror.pkgbuild.com)

---

# Communcation

* mail (mail.archlinux.org)
* quassel (quassel.archlinux.org)
* matrix (matrix.archlinux.org)

---

# AUR

* The Arch User Repository
* Written in PHP/Python
* Hosted on one dedicated server (luna)

???

- cgit / performance issues
- small mysql db still! 300 MB

---

# DBscripts

* Repository managament
* Handles addition/removal/moving of packages to the repository

---

# Wiki

* Mediawiki
* Private fork with some theming/branding
* 18.833 pages
* 30.000 users
* Biggest Database (7GB)

---

# BBS (FluxBB)

* Forum
* 88.300 registered users
* 1.700.000 posts
* ~ 5 GB DB

---

# Bugtracker (flyspray)

* Currently using flyspray
* Want to migrate to bugzilla

---

# Backups

* Borg is used for backups
* ~3 TB disk

---

# Extra mirrors

Provides mirrors underrepresented countries hosted on the PIA boxes

---

# Security Tracker

* Tracks CVE for Arch Linux packages
* Separate team maintains https://security.archlinux.org

---

# Security tracker

* Python-flask miminal webservice
* Tracks CVE's, Generates Advisory's
* JSON API for advisories, see arch-audit

???

Mention that we need people in our security tracking team.

---

# Archive

https://archive.archlinux.org

* 2.1 TB of archived packages
* Used for reproducible builds and downgrades
* Tooling https://github.com/archlinux/archivetools

???

Used to grow endlessly, had to write a custom script to remove unreferenced files
for reproducible builds. Partly uploaded to archive.org

---

# Arch Vagrant

* Automated building of VBox images for vagrant

---

# Mailman

* self hosted mailman hosted for all our mailing lists

---

# Git repo

* Git repo for various projects

---

# Misc

* Staging ground for archweb
* Patchwork instance
* Kanboard
* Torrent tracker
* Phrik

---

# Monitoring

* Zabbix
* Grafana dasbhoard

???

Demo dashboard

# Monitoring - Services

* custom scripts for monitoring systemd units
* monitor cpu/memory usage per service
* monitor failed units

???

Both are handled by a Python script with the dbus module to talk
to systemd.

---

# Monitoring - Rest

* Perl scripts for monitoring
* Borg monitoring
* MySQL user/db status
* Arch Audit for security issues

???

We've written custom tooling to monitor various things of interest.

---

# Monitoring - Results!

* Heavy MySQL usage for the AUR
* Quicker security patching
* Service monitoring spotted issue in cgit

???

Heavy MySQL stats ~ 300% CPU, resolved by nginx caching.
Heavy CPU usage for cgit, due to not git gc'ing.

---

# Management

* Ansible is used to manage most of the servers
* Managed by a small devops team (8 persons)

Mailing list
https://lists.archlinux.org/listinfo/arch-devops

Ansible code
https://git.archlinux.org/infrastructure.git/

---

# Ansible role

* pacman is supported!
* basic role to deploy our basic server
* deploys ssh/tools/users/pubkeys

---

# Future

* Ansible more Arch!
* LDAP
* Bugzilla
* Migrating from Luna

---

# Help out

* join #archlinux-devops
* contribute ansible roles

---

# Questions