aurutils 3.0.0

After 2 years of intermittent but hard work, aurutils 3.0.0 has been released. Thanks to all contributors - especially to my github collaborators @rafasc and @maximbaz.
$ git shortlog --no-merges -s '2.3.1..' | sort -rn
193 Alad Wenter
60 Rafael Ascensão
16 James Alad Bork
9 Maxim Baz
3 morganamilo
3 bartus
2 Morten Linderud
2 Ethan Sommer
2 Daniel Houck
2 aramirezverbio
1 Thom Wiggers
1 Szymon Mikulicz
1 nl6720
1 Matt Schreiber
1 Justin Wong
1 Felix Yan
1 columbarius
1 bartoszek
1 Avinash H. Duduskar
With the amount of changes, details spread over pull requests, git commit logs, and the documentation, I've written this post to detail the most important changes in aurutils 3.0.0. If you've used aurutils 2.3 before, I highly recommend reviewing the Transition table. Happy hacking!

Updates (aurutils 3.1):

  1. New features
    1. aur-sync: store commits viewed by the user
    2. aur-sync: default location for ignored packages
    3. aur-sync: enable --provides by default
    4. aur-fetch: local modifications and git rebase
    5. aur-fetch: removed support for tar snapshots
    6. aur-fetch: move diff output to aur-sync
    7. aur-chroot: static pacman configuration
    8. aur-build: new command-line
    9. aur-repo: new output options
    10. aur-query: parallel transfers
    11. aur: override commands with AUR_EXEC_PATH
    12. Reduced dependencies
    13. Releases signed with signify
  2. Transition table
  3. Detailed changelog

New features

aur-sync: store commits viewed by the user

In previous releases, aur-sync and aur-fetch would display package diffs relative to the git commit before new, upstream (AUR) changes were merged. If you decided to exit the viewer or file manager before the build process (for example, due to changes you did not agree to), that same diff would be lost on later aur-sync calls. With aurutils 3.0, aur-sync keeps a list of "seen" commits.

A seen commit is git HEAD of a package repository after exiting the file manager (with a non-zero exit status, or a confirmation by the user if AUR_CONFIRM_PAGER is set). Commit hashes are kept seperately from AURDEST in XDG_DATA_HOME/aurutils/view. The directory structure is as follows (example):
└── view
├── alatools
├── astroid-git
├── auracle-git
├── aur-talk-git
├── aurutils
├── aurutils-git
├── ...
where each file pkgbase has a commit hash (e.g. 6427ee3d091759d2497ff44a0498738fa4684a53) for a matching package repository. Absence of a file in this directory implies no revision of the package has been reviewed yet. This can be used by file managers such as nnn or ranger to indicate you should review all files in such a repository.

File review
        with ranger
Example implementation by @FichteFoll

Note that a package diff is only displayed with a matching seen commit in XDG_DATA_HOME/aurutils/view. The following script populates this directory for all packages in the local repository, using the git HEAD commit as the seen commit. (Any missing git repositories are cloned in turn.)

cd "${AURDEST:-~/.cache/aurutils/sync}"
aur repo --table | cut -f3 | sort -u | while IFS= read -r pkgbase; do
  if [[ -d $pkgbase/.git ]]; then
    env -C "$pkgbase" git rev-parse HEAD > "${XDG_DATA_HOME:-~/.local/share}/aurutils/view/$pkgbase"

Alternatively, aur sync -o --rebuild-all will retrieve and update all git repositories for packages in the local repository.

aur-sync: default location for ignored packages

aur-sync now supports a default location for packages it should ignore (aur sync --ignore) in XDG_CONFIG_HOME/aurutils/sync/ignore. Each package name should be specified on a seperate line.To add a comment (for example, the reason why aur-sync should ignore a particular package), add one after a number sign (#).

As of aurutils 3.0.0, it is not possible to version ignores (i.e. foo=1.2.3); this may be implemented in a later release.

aur-sync: enable --provides by default

In aurutils 2.3, the --provides flag allowed aur-sync to use aur-repo-filter for finding virtual dependencies (provides) in the target local repository. With aurutils 3.0, aur-sync by default searches all pacman sync repositories for virtual dependencies. You can disable this functionality with the --no-provides flag, or specify pacman repositories to search in with --provides-from.

aur-fetch: local modifications and git rebase

Local modifications to package repositories can now be preserved with git commit. There are several steps to take note of: In all of the above steps, we assumed that git-rebase applies cleanly. If local modifications cause a merge conflict, it has be to solved manually.

aur-fetch: removed support for tar snapshots

tar snapshots are no longer supported. In particular, the --git and --tar options were removed from aur-fetch and aur-sync. This is due to the reduced use of tar snapshots over time, and additional efforts required to implement persistent diffs for this format (see aur-sync: store commits viewed by the user for details).

aur-fetch: move diff output to aur-sync

To give aur-fetch a more clearly defined purpose, package diffs are now directly generated inside aur-sync. In particular, the --logdir, --verbose and --format flags were removed. However, you can still use aur-fetch to generate diffs without relying on aur-sync by using the new --results FILE option.

The output is as follows:
fetch is the default: only git fetch is run with no other changes. fetch_head represents the new FETCH_HEAD object. When cloning a new repository with git clone, the value 0 represents a non-existing object. If a git rebase or git reset was done, prev_head represents the previous git HEAD.

aur-sync uses this information together with the stored commits in XDG_DATA_HOME/aurutils/view (see aur-sync: store commits viewed by the user for details) to generate package diffs.

aur-chroot: static pacman configuration

Note: With aurutils 3.1, the aur-build pacman configuration path moved from /usr/share/devtools/pacman-aur.conf to /etc/aurutils/pacman-<repository_name>.conf. Consequently, the --suffix option was removed from aur-build and aur-sync. multilib-devel is now also (when applicable) installed next to base-devel; the multilib-devel group only contains replacements for a subset of packages in base-devel. See github pull 786. The section below is updated accordingly.

(and by extension, aur build --chroot) no longer generates a pacman configuration based on local repositories configured on the host. Instead, it uses a static configuration file where local repositories are configured. For aur-chroot, the default file path is /usr/share/devtools/pacman-extra.conf. The extra suffix in pacman-extra.conf can be changed with the --suffix option, for example --suffix=multilib for /usr/share/devtools/pacman-multilib.conf.

The full path can be set with the --pacman-conf option. aur-build defaults to --pacman-conf=/etc/aurutils/pacman-<repository_name>.conf. Note that the pacman.conf inside the container may not reflect this file fully, due to additional processing by arch-nspawn (see FS#38641 for details).

There are some restrictions imposed on the pacman configuration. In particular, repositories defined in seperate files (and refered to by Include statements) are ignored by devtools. Other directives such as IgnorePkg can also be problematic for container use. To avoid issues, I suggest taking /usr/share/devtools/pacman-extra.conf as a template instead of /etc/pacman.conf.

aur-build: new command-line

Previously, aur-build included --clean, --syncdeps as default makepkg arguments for building packages. To override these options, you could use the end-of-options separator (--) as follows.
    aur build
would run makepkg --clean --syncdeps,
    aur build -- -r -sc
would run makepkg --clean --syncdeps --rmdeps, and
    aur build -- -r
would only run makepkg --rmdeps. aurutils 3.0 brings the aur-build command-line closer to makepkg: The above example then becomes:
    aur build
which runs makepkg,
    aur build --clean -s
which runs makepkg --clean --syncdeps, and
    aur build --clean -sr
which runs makepkg --clean --syncdeps --rmdeps. Additional makepkg options that are not wrapped by aur-build can be specified with the --margs option, for example:
    aur build --clean -sr --margs --skippgpcheck
Note: -S is now an alias for --sign, and -s an alias for --syncdeps. The -c option remains an alias for --chroot.

aur-repo: new output options

Note: aurutils 3.1 removed the aur repo --ini option (displaying a partial pacman.conf output of local repositories) and added aur repo --path-list. The section below is updated accordingly.

For extended information on local repositories, aur-repo now supports the --table option. --table is similar to aur depends --table and outputs four columns: pkgname, depends, pkgbase, and pkgver.
$ aur repo --table | column -t
alatools python-xdg alatools 2018.12-2
alatools python3 alatools 2018.12-2
auracle-git pacman auracle-git r366.8739929-1
auracle-git libcurl.so=4-64 auracle-git r366.8739929-1
auracle-git libsystemd.so=0-64 auracle-git r366.8739929-1
auracle-git meson auracle-git r366.8739929-1
auracle-git git auracle-git r366.8739929-1
auracle-git perl auracle-git r366.8739929-1
auracle-git systemd auracle-git r366.8739929-1
auracle-git gtest auracle-git r366.8739929-1
auracle-git gmock auracle-git r366.8739929-1
aur-talk-git python aur-talk-git r21.31cb5c0-1
aur-talk-git python-lxml aur-talk-git r21.31cb5c0-1
aur-talk-git python-html2text aur-talk-git r21.31cb5c0-1
aur-talk-git git aur-talk-git r21.31cb5c0-1
aurutils-git git aurutils-git 3.0.0rc.r20.g63bdcc4-1
aurutils-git jq aurutils-git 3.0.0rc.r20.g63bdcc4-1
aurutils-git pacutils aurutils-git 3.0.0rc.r20.g63bdcc4-1
aurutils-git curl aurutils-git 3.0.0rc.r20.g63bdcc4-1
aurutils-git git aurutils-git 3.0.0rc.r20.g63bdcc4-1

aur repo --repo-list now outputs a simple list of local repository names. To print local repository paths, use aur repo --path-list.

aur-query: parallel transfers

To improve performance on large aurweb queries, aur-query (previously: aur-rpc) now supports parallel GET requests with curl --parallel. You can regulate this behavior with the AUR_QUERY_PARALLEL and AUR_QUERY_PARALLEL_MAX (curl --parallel-max, defaults to 15) environment variables. As a sample case, we wish to check packages that are both in the AUR and the official repositories. For this we use a combination of expac and aur-vercmp --all on 11291 packages. The test environment is a low-end VM (VirtualBox, 1024MB RAM, 1 CPU) and an LTE connection.

Without parallel transfers:

archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=0 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=0 aur vercmp --all
Time (mean ± σ): 12.115 s ± 0.224 s [User: 1.955 s, System: 2.609 s]
Range (min … max): 11.841 s … 12.569 s 10 runs

With parallel transfers and 15 threads:
archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=1 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=1 aur vercmp --all
Time (mean ± σ): 5.598 s ± 0.209 s [User: 1.795 s, System: 2.618 s]
Range (min … max): 5.304 s … 6.017 s 10 runs

With parallel transfers and 50 threads:
archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=1 AUR_QUERY_PARALLEL=50 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=1 AUR_QUERY_PARALLEL_MAX=50 aur vercmp --all
Time (mean ± σ): 6.729 s ± 0.302 s [User: 3.040 s, System: 2.746 s]
Range (min … max): 6.348 s … 7.258 s 10 runs
As the above test indicates, a higher amount of threads does not necessarily result in higher performance.

Note: curl --parallel support is experimental.

aur: override commands with AUR_EXEC_PATH

The path to aurutils scripts can now be set at runtime with the AUR_EXEC_PATH environment variable. While originally made for testing new changes, you can use this to easily modify or extend core aurutils functionality. This is similar to placing a custom aurfoo program higher in PATH in aurutils 1.5 and previous versions.

This path can also be set at build-time with the AUR_LIB_DIR macro.

Reduced dependencies

diffstat, expac, parallel and wget are no longer used. Duties handled by expac are now handled solely by pacutils, and wget was replaced by curl. diffstat was used to generate diff summaries for tar snapshots, and latter were removed (see aur-fetch: removed support for tar snapshots.)

Releases signed with signify

Due to keyserver issues with gpg, releases are signed with signify starting with aurutils 2.3.7. They can be verified with the following public keys:
aurutils 2.3.x
untrusted comment: signify public key (aurutils 2.3)
aurutils 3.0.x
untrusted comment: signify public key (aurutils 3.0)
aurutils 3.1.x
untrusted comment: signify public key (aurutils 3.1)
See also https://github.com/AladW/aurutils#signatures.

Transition table

Old command New command Notes
aur jobs - The nag banner in parallel is already removed in community/parallel, and aur-jobs had no other functionality.
aur build --build-command cmd args... MAKEPKG=cmd aur build --margs args... -
aur build aur build -sc aur-build now has an empty set of default arguments. Further arguments can be specified either directly (-s, -r, -L, -A) or with the --margs option. See aur-build: new command-line for details.
aur build -s aur build -S The -s alias for --sign was renamed to -S. -s is now an alias for --syncdeps instead, matching makepkg -s.
aur build -- makepkg args... aur build --margs makepkg args... There are two differences to note here: the first is that the end-of-options separator -- is no longer used for specifying makepkg options. The second is that --margs does not reset makepkg options specified elsewhere (default, or as aur-build arguments). See aur-build: new command-line for details.
aur chroot --no-build aur chroot --update or aur chroot -U aur-chroot steps are now enabled selectively, instead of disabled.
aur chroot --no-prepare aur chroot --build or aur chroot -B aur-chroot steps are now enabled selectively, instead of disabled.
aur chroot --database repo1 --database repo2... - aur-chroot now uses a static pacman configuration. See aur-chroot: static pacman configuration for details.
aur fetch --log-dir, aur fetch --verbose, aur fetch --format aur fetch --results FILE Package diffs are now left to other tools, in particular aur-sync. Diffs can be generated directly from the --results output. See aur-fetch: move diff output to aur-sync for details.
aur rpc aur query -
AUR_LOCATION=https://... aur rpc --rpc-ver 5 AUR_QUERY_RPC=https://.../rpc/?v=5 aur query -
aur repo aur repo --path When no arguments were specified to aur-repo, it would print the path of the local repository. Printing the path now requires the --path argument; without arguments, aur-repo prints a usage text.
aur sync --provides aur sync, aur sync --provides-from See aur-sync: use --provides by default for details.
aur sync --git, aur sync --tar aur sync Support for retrieving tar snapshots was removed. See aur-fetch: removed support for tar snapshots for details.
aur sync --print aur sync --no-build -
aur sync --no-ver-shallow aur sync --no-ver-argv -
aur fetch aur fetch --sync=reset, aur fetch --sync=auto aur-fetch now only runs git fetch by default. See aur-fetch: local modifications and git rebase for details.
aur vercmp-devel
Due to limitations in aur-vercmp-devel, the script was moved to the manual page aurvcs. This page will be extended in later releases with alternative approaches to updating VCS packages. See PR#617 for details.

Detailed changelog

This changelog can also be found in the aurutils package or on github.

Copyright © 2020, Alad Wenter