Full system integration/acceptance testing is done through autopkgtests. These test the actual installed binary distribution packages. They are run in QEMU or containers and thus can do intrusive and destructive things such as installing arbitrary packages, modifying arbitrary files in the system (including grub boot parameters), rebooting, or loading kernel modules.
The tests for systemd are defined in the Debian package's debian/tests directory. For validating a pull request, the Debian package is built using the unpatched code from that PR (via the checkout-upstream script), and the tests run against these built packages. Note that some tests which check Debian specific behaviour are skipped in "test upstream" mode.
systemd's GitHub project has webhooks that trigger autopkgtests on Ubuntu 16.04 LTS on three architectures:
- i386: 32 bit x86, little endian, QEMU (OpenStack cloud instance)
- amd64: 64 bit x86, little endian, QEMU (OpenStack cloud instance)
- s390x: 64 bit IBM z/Series, big endian, LXC (this architecture is not yet available in Canonical's OpenStack and thus skips some tests)
Please see the Ubuntu CI infrastructure documentation for details about how this works.
Manually retrying/triggering tests on the infrastructure
The current tests are fairly solid by now, but rarely they fail on infrastructure/network issues or race conditions. If you encounter these, please notify @martinpitt in the GitHub PR for debugging/fixing those -- transient infrastructure issues are supposed to be detected automatically, and tests auto-retry on those; and flaky tests should of course be fixed properly. But sometimes it is useful to trigger tests on a different Ubuntu release too, for example to test a PR on a newer kernel or against current build/binary dependencies (cgroup changes, util-linux, gcc, etc.).
This can be done using the generic retry-github-test script from Ubuntu's autopkgtest infrastructure: you need the parameterized URL from the configured webhooks and the shared secret (Ubuntu's CI needs to restrict access to avoid DoSing and misuse).
You can use Martin Pitt's retry-gh-systemd-test shell wrapper around retry-github-test for that. You need to adjust the path where you put retry-github-test and the file with the shared secret, then you can call it like this:
$ retry-gh-systemd-test <#PR> <architecture> [release]
release defaults to
xenial (aka Ubuntu 16.04 LTS). For example:
$ retry-gh-systemd-test 1234 amd64 $ retry-gh-systemd-test 2345 s390x zesty
Please make sure to not trigger unknown releases/architectures as they will cause a pending test on the PR which never gets finished.
Test the code from the PR locally
As soon as a test on the infrastructure finishes, the "Details" link in the PR "checks" section will point to the
log.gz log. You can download the individual test log, built .debs, and other artifacts that tests leave behind (some dump a complete journal or the udev database on failure) by replacing
/artifacts.tar.gz in that URL. You can then unpack the tarball and use
sudo dpkg -iO binaries/*.deb to install the debs from the PR into an Ubuntu VM of the same release/architecture for manually testing a PR.
Run autopkgtests locally
git clone https://anonscm.debian.org/git/autopkgtest/autopkgtest.git
Install necessary dependencies; on Debian/Ubuntu you can simply run
sudo apt install autopkgtest(instead of the above cloning), on Fedora do
yum install qemu-kvm dpkg-perl
Build a test image based on Ubuntu cloud images for the desired release/arch:
autopkgtest/tools/autopkgtest-buildvm-ubuntu-cloud -r xenial -a amd64
This will build
autopkgtest-xenial-amd64.img. This is normally being used through the
autopkgtestcommand (see below), but you can boot this normally in QEMU (using
-snapshotis highly recommended) to interactively poke around; this provides a easy throw-away test environment.
The most basic mode of operation is to run the tests for the current distro packages:
autopkgtest/runner/autopkgtest systemd -- qemu autopkgtest-xenial-amd64.img
But autopkgtest allows lots of different modes and options, like running a shell on failure (
-s), running a single test only (
--test-name), running the tests from a local checkout of the Debian source tree (possibly with modifications to the test) instead of from the distribution source, or running QEMU with more than one CPU (check the autopkgtest-virt-qemu manpage.
A common use case is to check out the Debian packaging git for getting/modifying the tests locally:
git clone https://anonscm.debian.org/git/pkg-systemd/systemd.git /tmp/systemd-debian/
and running these against the binaries from a PR (see above), running only the
logind test, getting a shell on failure, showing the boot output, and running with 2 CPUs:
autopkgtest/runner/autopkgtest --test-name logind /tmp/binaries/*.deb /tmp/systemd-debian/ -s -- \ qemu --show-boot --cpus 2 /srv/vm/autopkgtest-xenial-amd64.img