ekeyrevoked

About — Contact — RSS — Index

aurutils 3.0.0

After 2 years of intermittent but hard work, aurutils 3.0.0 has been released. Thanks to all contributors - especially to my github collaborators @rafasc and @maximbaz.

$ git shortlog --no-merges -s '2.3.1..' | sort -rn
   193  Alad Wenter
    60  Rafael Ascensão
    16  James Alad Bork
     9  Maxim Baz
     3  morganamilo
     3  bartus
     2  Morten Linderud
     2  Ethan Sommer
     2  Daniel Houck
     2  aramirezverbio
     1  Thom Wiggers
     1  Szymon Mikulicz
     1  nl6720
     1  Matt Schreiber
     1  Justin Wong
     1  Felix Yan
     1  columbarius
     1  bartoszek
     1  Avinash H. Duduskar

With the amount of changes, details spread over pull requests, git commit logs, and the documentation, I've written this post to detail the most important changes in aurutils 3.0.0. If you've used aurutils 2.3 before, I highly recommend reviewing the Transition table. Happy hacking!

Updates:

Clarified aur repo --path and added aur vercmp-devel to the transition table. Thanks to @Strykar and @hashworks. [29/10/2020 18:10 GMT]
Fixed --clean short option in aur-build: new command-line. Thanks to @FichteFoll. [30/10/2020 19:50 GMT]
Changed "EOF" to end-of-options separator. Thanks to @Earnestly. [30/10/2020 19:50 GMT]
Added a section on signify. (Releases signed with signify) [03/11/2020 14:40 GMT]
Clarified populating the view directory for aur-sync. (github issue 791) [11/11/2020 18:40 GMT]
Clarified restrictions on aur-chroot static pacman configuration and new default paths in aurutils 3.1. [11/11/2020 20:51 GMT]
Removed aur repo --ini from transition table. Updated aur-repo: new output options section. (github issue 793) [17/11/2020 13:38 GMT]
Added updated file paths. Updated aur-fetch: move diff output to aur-sync section. (github issue 798) [08/12/2020 16:03 GMT]

New features
Transition table
Detailed changelog

New features

aur-sync: store commits viewed by the user

In previous releases, aur-sync and aur-fetch would display package diffs relative to the git commit before new, upstream (AUR) changes were merged. If you decided to exit the viewer or file manager before the build process (for example, due to changes you did not agree to), that same diff would be lost on later aur-sync calls. With aurutils 3.0, aur-sync keeps a list of "seen" commits.

A seen commit is git HEAD of a package repository after exiting the file manager (with a non-zero exit status, or a confirmation by the user if AUR_CONFIRM_PAGER is set). Commit hashes are kept seperately from AURDEST in XDG_DATA_HOME/aurutils/view. The directory structure is as follows (example):

  .
  └── view
      ├── alatools
      ├── astroid-git
      ├── auracle-git
      ├── aur-talk-git
      ├── aurutils
      ├── aurutils-git
      ├── ...

where each file pkgbase has a commit hash (e.g. 6427ee3d091759d2497ff44a0498738fa4684a53) for a matching package repository. Absence of a file in this directory implies no revision of the package has been reviewed yet. This can be used by file managers such as nnn or ranger to indicate you should review all files in such a repository.

Example implementation by @FichteFoll

Note that a package diff is only displayed with a matching seen commit in XDG_DATA_HOME/aurutils/view. The following script populates this directory for all packages in the local repository, using the git HEAD commit as the seen commit. (Any missing git repositories are cloned in turn.)

#!/bin/bash
cd "${AURDEST:-~/.cache/aurutils/sync}"
aur repo --table | cut -f3 | sort -u | while IFS= read -r pkgbase; do
if [[ -d $pkgbase/.git ]]; then
env -C "$pkgbase" git rev-parse HEAD > "${XDG_DATA_HOME:-~/.local/share}/aurutils/view/$pkgbase"
fi
done

Alternatively, aur sync -o --rebuild-all will retrieve and update all git repositories for packages in the local repository.

aur-sync: default location for ignored packages

aur-sync now supports a default location for packages it should ignore (aur sync --ignore) in XDG_CONFIG_HOME/aurutils/sync/ignore. Each package name should be specified on a seperate line.To add a comment (for example, the reason why aur-sync should ignore a particular package), add one after a number sign (#).

As of aurutils 3.0.0, it is not possible to version ignores (i.e. foo=1.2.3); this may be implemented in a later release.

aur-sync: enable --provides by default

In aurutils 2.3, the --provides flag allowed aur-sync to use aur-repo-filter for finding virtual dependencies (provides) in the target local repository. With aurutils 3.0, aur-sync by default searches all pacman sync repositories for virtual dependencies. You can disable this functionality with the --no-provides flag, or specify pacman repositories to search in with --provides-from.

aur-fetch: local modifications and git rebase

Local modifications to package repositories can now be preserved with git commit. There are several steps to take note of:

aur fetch --sync=rebase runs git fetch to fetch new revisions, git reset --hard HEAD to discard uncommited changes and git rebase to merge local commited changes. If rebase.autoStash is true (git config rebase.autoStash true), then git reset is skipped. The --sync=rebase option applies to all packages specified on the aur-fetch command-line.
aur fetch --sync=reset discards any local changes with git reset --hard HEAD@{upstream}.
aur fetch --sync=auto chooses between --sync=reset and --sync=rebase depending if aurutils.rebase is true (git config aurutils.rebase true). This way, you can decide for each package repository if local changes should be preserved or not. For details on the --sync=auto flag, see man 1 aur-fetch.
aur sync uses aur fetch --sync=auto.

In all of the above steps, we assumed that git-rebase applies cleanly. If local modifications cause a merge conflict, it has be to solved manually.

aur-fetch: removed support for tar snapshots

tar snapshots are no longer supported. In particular, the --git and --tar options were removed from aur-fetch and aur-sync. This is due to the reduced use of tar snapshots over time, and additional efforts required to implement persistent diffs for this format (see aur-sync: store commits viewed by the user for details).

aur-fetch: move diff output to aur-sync

To give aur-fetch a more clearly defined purpose, package diffs are now directly generated inside aur-sync. In particular, the --logdir, --verbose and --format flags were removed. However, you can still use aur-fetch to generate diffs without relying on aur-sync by using the new --results FILE option.

The output is as follows:

    fetch:head:fetch_head:file://path
    rebase:prev_head:head:file://path
    reset:prev_head:head:file://path
    clone:0:head:file://path

fetch is the default: only git fetch is run with no other changes. fetch_head represents the new FETCH_HEAD object. When cloning a new repository with git clone, the value 0 represents a non-existing object. prev_head represents the previous git HEAD if a git rebase or git reset was done. aur-sync uses this information (together with stored commits in XDG_DATA_HOME/aurutils/view, see aur-sync: store commits viewed by the user) to generate package diffs.

Furthermore, aur-sync now also generates an orderFile (which defines the order of files displayed in git-diff) for each git repository in AURDEST. The path was changed accordingly to $XDG_CONFIG_HOME/aurutils/sync/orderfile. You can update existing paths with git config, for example:

    find ~/.cache/aurutils/sync -type d -exec git -C {} config diff.orderfile ~/.config/aurutils/sync/orderfile \;

See man git-diff for more information.

aur-chroot: static pacman configuration

Note: With aurutils 3.1, the aur-build pacman configuration path moved from /usr/share/devtools/pacman-aur.conf to /etc/aurutils/pacman-<repository_name>.conf. Consequently, the --suffix option was removed from aur-build and aur-sync. multilib-devel is now also (when applicable) installed next to base-devel; the multilib-devel group only contains replacements for a subset of packages in base-devel. See github pull 786. The section below was updated accordingly.

aur-chroot (and by extension, aur build --chroot) no longer generates a pacman configuration based on local repositories configured on the host. Instead, it uses a static configuration file where local repositories are configured. For aur-chroot, the default file path is /usr/share/devtools/pacman-extra.conf. The extra suffix in pacman-extra.conf can be changed with the --suffix option, for example --suffix=multilib for /usr/share/devtools/pacman-multilib.conf.

The full path can be set with the --pacman-conf option. aur-build defaults to --pacman-conf=/etc/aurutils/pacman-<repository_name>.conf. Note that the pacman.conf inside the container may not reflect this file fully, due to additional processing by arch-nspawn (see FS#38641 for details).

There are some restrictions imposed on the pacman configuration. In particular, repositories defined in seperate files (and refered to by Include statements) are ignored by devtools. Other directives such as IgnorePkg can also be problematic for container use. To avoid issues, I suggest taking /usr/share/devtools/pacman-extra.conf as a template instead of /etc/pacman.conf.

aur-build: new command-line

Previously, aur-build included --clean, --syncdeps as default makepkg arguments for building packages. To override these options, you could use the end-of-options separator (--) as follows.

    aur build

would run makepkg --clean --syncdeps,

    aur build -- -r -sc

would run makepkg --clean --syncdeps --rmdeps, and

    aur build -- -r

would only run makepkg --rmdeps. aurutils 3.0 brings the aur-build command-line closer to makepkg:

there are no fixed makepkg arguments;
common makepkg options are taken directly from the command-line (--syncdeps, --rmdeps, --log, --ignorearch). When using aur build --pkgver, these options also apply to the preparatory makepkg -od command;
the end-of-options separator (--) serves only to delimit options.

The above example then becomes:

    aur build

which runs makepkg,

    aur build --clean -s

which runs makepkg --clean --syncdeps, and

    aur build --clean -sr

which runs makepkg --clean --syncdeps --rmdeps. Additional makepkg options that are not wrapped by aur-build can be specified with the --margs option, for example:

    aur build --clean -sr --margs --skippgpcheck

Note: -S is now an alias for --sign, and -s an alias for --syncdeps. The -c option remains an alias for --chroot.

aur-repo: new output options

Note: aurutils 3.1 removed the aur repo --ini option (displaying a partial pacman.conf output of local repositories) and added aur repo --path-list. The section below was updated accordingly.

For extended information on local repositories, aur-repo now supports the --table option. --table is similar to aur depends --table and outputs four columns: pkgname, depends, pkgbase, and pkgver.

$ aur repo --table | column -t
alatools                      python-xdg             alatools                      2018.12-2
alatools                      python3                alatools                      2018.12-2
auracle-git                   pacman                 auracle-git                   r366.8739929-1
auracle-git                   libcurl.so=4-64        auracle-git                   r366.8739929-1
auracle-git                   libsystemd.so=0-64     auracle-git                   r366.8739929-1
auracle-git                   meson                  auracle-git                   r366.8739929-1
auracle-git                   git                    auracle-git                   r366.8739929-1
auracle-git                   perl                   auracle-git                   r366.8739929-1
auracle-git                   systemd                auracle-git                   r366.8739929-1
auracle-git                   gtest                  auracle-git                   r366.8739929-1
auracle-git                   gmock                  auracle-git                   r366.8739929-1
aur-talk-git                  python                 aur-talk-git                  r21.31cb5c0-1
aur-talk-git                  python-lxml            aur-talk-git                  r21.31cb5c0-1
aur-talk-git                  python-html2text       aur-talk-git                  r21.31cb5c0-1
aur-talk-git                  git                    aur-talk-git                  r21.31cb5c0-1
aurutils-git                  git                    aurutils-git                  3.0.0rc.r20.g63bdcc4-1
aurutils-git                  jq                     aurutils-git                  3.0.0rc.r20.g63bdcc4-1
aurutils-git                  pacutils               aurutils-git                  3.0.0rc.r20.g63bdcc4-1
aurutils-git                  curl                   aurutils-git                  3.0.0rc.r20.g63bdcc4-1
aurutils-git                  git                    aurutils-git                  3.0.0rc.r20.g63bdcc4-1

aur-query: parallel transfers

To improve performance on large aurweb queries, aur-query (previously: aur-rpc) now supports parallel GET requests with curl --parallel. You can regulate this behavior with the AUR_QUERY_PARALLEL and AUR_QUERY_PARALLEL_MAX (curl --parallel-max, defaults to 15) environment variables. As a sample case, we wish to check packages that are both in the AUR and the official repositories. For this we use a combination of expac and aur-vercmp --all on 11291 packages. The test environment is a low-end VM (VirtualBox, 1024MB RAM, 1 CPU) and an LTE connection.

Without parallel transfers:

archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=0 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=0 aur vercmp --all
  Time (mean ± σ):     12.115 s ±  0.224 s    [User: 1.955 s, System: 2.609 s]
  Range (min … max):   11.841 s … 12.569 s    10 runs

With parallel transfers and 15 threads:

archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=1 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=1 aur vercmp --all
  Time (mean ± σ):      5.598 s ±  0.209 s    [User: 1.795 s, System: 2.618 s]
  Range (min … max):    5.304 s …  6.017 s    10 runs

With parallel transfers and 50 threads:

archie@vbox:~$ hyperfine 'expac -S '%n\t%v' | env AUR_QUERY_PARALLEL=1 AUR_QUERY_PARALLEL=50 aur vercmp --all'

Benchmark #1: expac -S %nt%v | env AUR_QUERY_PARALLEL=1 AUR_QUERY_PARALLEL_MAX=50 aur vercmp --all
  Time (mean ± σ):      6.729 s ±  0.302 s    [User: 3.040 s, System: 2.746 s]
  Range (min … max):    6.348 s …  7.258 s    10 runs

As the above test indicates, a higher amount of threads does not necessarily result in higher performance.

Note: curl --parallel support is experimental.

aur: override commands with AUR_EXEC_PATH

The path to aurutils scripts can now be set at runtime with the AUR_EXEC_PATH environment variable. While originally made for testing new changes, you can use this to easily modify or extend core aurutils functionality. This is similar to placing a custom aurfoo program higher in PATH in aurutils 1.5 and previous versions.

This path can also be set at build-time with the AUR_LIB_DIR macro.

Reduced dependencies

diffstat, expac, parallel and wget are no longer used. Duties handled by expac are now handled solely by pacutils, and wget was replaced by curl. diffstat was used to generate diff summaries for tar snapshots, and latter were removed (see aur-fetch: removed support for tar snapshots.)

Releases signed with `signify`

Due to keyserver issues with gpg, releases are signed with signify starting with aurutils 2.3.7. They can be verified with the following public keys:

aurutils 3.0.x

untrusted comment: signify public key (aurutils 3.0)
RWQcfDbvctqX5i5yNDNqu607LC7mKArHMsq7ziT8ynx9bQwj5m5ZpMJY

aurutils 3.1.x

untrusted comment: signify public key (aurutils 3.1)
RWSiKm8qeKjPfppkN7lm/N4qENa3Racl7DRMfWK4JQS7bl2i/NuI3ZZG

Transition table

Old command	New command	Notes
`aur jobs`	-	The nag banner in parallel is already removed in community/parallel, and aur-jobs had no other functionality.
`aur build --build-command cmd args...`	`MAKEPKG=cmd aur build --margs args...`	-
`aur build`	`aur build -sc`	aur-build now has an empty set of default arguments. Further arguments can be specified either directly (`-s`, `-r`, `-L`, `-A`) or with the `--margs` option. See aur-build: new command-line for details.
`aur build -s`	`aur build -S`	The `-s` alias for `--sign` was renamed to `-S`. `-s` is now an alias for `--syncdeps` instead, matching `makepkg -s`.
`aur build -- makepkg args...`	`aur build --margs makepkg args...`	There are two differences to note here: the first is that the end-of-options separator -- is no longer used for specifying makepkg options. The second is that --margs does not reset makepkg options specified elsewhere (default, or as aur-build arguments). See aur-build: new command-line for details.
`aur chroot --no-build`	`aur chroot --update` or `aur chroot -U`	aur-chroot steps are now enabled selectively, instead of disabled.
`aur chroot --no-prepare`	`aur chroot --build` or `aur chroot -B`	aur-chroot steps are now enabled selectively, instead of disabled.
`aur chroot --database repo1 --database repo2...`	-	aur-chroot now uses a static pacman configuration. See aur-chroot: static pacman configuration for details.
`aur fetch`	`aur fetch --sync=reset`, `aur fetch --sync=auto`	aur-fetch now only runs git fetch by default. See aur-fetch: local modifications and git rebase for details.
`aur fetch --log-dir`, `aur fetch --verbose`, `aur fetch --format`	`aur fetch --results FILE`	Package diffs are now left to other tools, in particular aur-sync. Diffs can be generated directly from the --results output. See aur-fetch: move diff output to aur-sync for details.
`aur rpc`	`aur query`	-
`AUR_LOCATION=https://... aur rpc --rpc-ver 5`	`AUR_QUERY_RPC=https://.../rpc/?v=5 aur query`	-
`aur repo`	`aur repo --path`	When no arguments were specified to `aur-repo`, it would print the path of the local repository. Printing the path now requires the `--path` argument; without arguments, `aur-repo` prints a usage text.
aur repo --repo-list	aur repo --repo-list, aur repo --path-list	The output format of aur repo --repo-list was changed from tab-seperated repository name/path entries to only repository names. Printing the paths can be done with aur repo -path-list instead.
`aur sync --provides`	`aur sync`, `aur sync --provides-from`	See aur-sync: use --provides by default for details.
`aur sync --git`, `aur sync --tar`	`aur sync`	Support for retrieving tar snapshots was removed. See aur-fetch: removed support for tar snapshots for details.
`aur sync --print`	`aur sync --no-build`	-
`aur sync --no-ver-shallow`	`aur sync --no-ver-argv`	-
`aur vercmp-devel`	-	Due to limitations in aur-vercmp-devel, the script was moved to the manual page aurvcs. This page will be extended in later releases with alternative approaches to updating VCS packages. See PR#617 for details.

File paths

Old path	New path	Notes
$XDG_CONFIG_HOME/aurutils/fetch/orderfile ~/.config/aurutils/fetch/orderfile	$XDG_CONFIG_HOME/aurutils/sync/orderfile ~/.config/aurutils/sync/orderfile	This file defines the ordering of files in git diff output. (See man git-diff.) Diffs are now generated inside aur-sync, and the orderfile path is changed accordingly. See aur-fetch: move diff output to aur-sync for details.
-	/etc/aurutils/pacman-<repository>.conf	aur-chroot and aur-build --chroot now use a static pacman configuration. See aur-chroot: static pacman configuration for details.
-	`$XDG_CONFIG_HOME/aurutils/sync/ignore`	aur-sync now ignores packages in this file by default. Priorly, a path had to be given explicitly with --ignore-file, or the packages specified on the command-line with aur sync --ignore [package...].
$XDG_CACHE_HOME/aurutils/snapshot ~/.cache/aurutils/snapshot	-	Support for retrieving tar snapshots was removed. See aur-fetch: removed support for tar snapshots for details.

Detailed changelog

This changelog (and those for later releases) can also be found in the aurutils package or on github.

aur
- add AUR_EXEC_PATH environment variable
- add example aur-gc, aur-remove scripts to aur.1
aur-build
- add AUR_DEBUG, NO_COLOR, MAKEPKG environment variables
- add mollyguard for running as the root user
- add build: prefix to --results output
  - build:file:///path/to/package.tar.xz
- add --new, --prevent-downgrade (repo-add options)
- add Running message for all makepkg invocations
- replace --build-command with --margs and $MAKEPKG
- remove default makepkg arguments (#635)
  - specify common makepkg options as aur-build arguments (--syncdeps, --rmdeps, --ignorearch, --log, --noconfirm)
  - pass --syncdeps, --rmdeps to --pkgver makepkg command (#716)
  - make -r an alias for --rmdeps (was: alias for --root)
- use --margs to set makepkg options instead of the end-of-options separator (--)
- --makepkg-conf now sets the makepkg configuration for host builds (avoid quoting issues with setting --config in --margs)
- use makepkg --noextract if --pkgver is specified (#708)
- use print_all_package_names (/usr/share/makepkg/util/pkgbuild.sh) instead of makepkg --packagelist (#755)
- pass pacman.conf (--config) to aur-repo (#654)
- only remove intermediary package directory if empty (#602)
- remove --delta
- use PKGDEST instead of makepkg.conf to set package destination
aur-chroot
- add AUR_DEBUG environment variable
- add --packagelist
  - use print_all_package_names (/usr/share/makepkg/util/pkgbuild.sh)
- replace --no-prepare and --no-build (disable steps) with --update and --build (enable steps) respectively
- remove --database
- use static pacman configuration (defaults to /usr/share/devtools/pacman-extra.conf)
  - --suffix to override extra in the default path
  - if --suffix begins in multilib, install multilib-devel (instead of base-devel)
aur-fetch
- add AUR_DEBUG, NO_COLOR environment variables
- add --results (colon-delimited output)
- add --sync
  - only run git fetch by default;
  - run git reset if --sync=reset is specified;
  - run git rebase if --sync=rebase is specified;
  - run either git reset or git rebase if --sync=auto is specified
- exit 1 if git clone or git fetch failed
- remove support for tar archives
- remove diff output (--log-dir, --verbose, --format)
  - diffs can be generated from the extended --results output (done in aur-sync)
- remove setting orderfile (done in aur-sync)
- use git -C for git calls
aur-jobs (removed)
- remove script
aur-pkglist
- add AUR_DEBUG environment variable
- use curl for transfers instead of wget
aur-rpc (aur-query)
- add AUR_DEBUG, AUR_QUERY_RPC, AUR_QUERY_RPC_SPLITNO, AUR_QUERY_PARALLEL, AUR_QUERY_PARALLEL_MAX environment variables
- rename to aur-query
- set AUR RPC address with AUR_QUERY_RPC instead of a combination of --rpc-url, --rpc-ver and AUR_LOCATION
- set number of packages for splitting URIs with AUR_QUERY_RPC_SPLITINFO
- set AUR_QUERY_RPC_SPLITNO to set of number of packages for splitting URIs
- use curl for transfers instead of wget
  - enable parallel transfers (curl --parallel) with AUR_QUERY_PARALLEL=1
  - set maximum amount of parallel transfers (curl --parallel-max) with AUR_QUERY_PARALLEL_MAX
- exit 1 if a transfer failed
aur-repo-filter
- add AUR_DEBUG environment variable
- add --sysroot (pacsift --sysroot, pacinfo --sysroot)
- use pacinfo for package information instead of expac
aur-repo
- add AUR_DEBUG environment variable
- add --config
- add --ini
- add --quiet/-q (aur-vercmp -q)
- add --table
- add pacman configuration to --status-format (colon-delimited)
- print fully resolved path to local repository
- require --path to print the local repository path
- remove expac dependency
aur-search
- add AUR_DEBUG, NO_COLOR environment variables
- unset LC_ALL when printing Popularity
aur-srcver
- add AUR_DEBUG environment variable
- add --jobs
- do not use makepkg --log
- remove parallel dependency
aur-sync
- add AUR_DEBUG, NO_COLOR, AUR_CONFIRM_PAGER environment variables
  - use confirmation prompt after package review if AUR_CONFIRM_PAGER is set
- add --rebuild-all
- add mollyguard for running as the root user
- add default ignore file in $XDG_CONFIG_HOME/aurutils/sync/ignore
- do not fallback to PAGER or less (AUR_PAGER must be set or vifm installed)
- enable --provides by default
  - disable with --no-provides
  - specify repositories with --provides-from (comma-delimited)
- remove additional aur-build arguments after end-of-options separator (--) (#678)
- remove parallel dependency
- remove AURDEST_SNAPSHOT environment variable
- remove --git, --tar
- rename --print to --no-build/-o
- rename --no-ver-shallow to --no-ver-argv
- store commits viewed by the user (#379, #711)
- use aur-fetch --sync=auto
aur-vercmp
- add AUR_DEBUG, NO_COLOR environment variables
Makefile
- allow overriding AUR_LIB_DIR at build time