Previously, I’ve modified the blog post from 2022-01 to update the installation to bookworm. This is a rewrite, based on trixie.

Steps to install openQA (a local copy of openqa.debian.net)

The goal is to install openQA in a local virtual machine with the capacity to run 2 workers.

1. Use qemu with a recent netinst image (at time of writing: trixie 13.1.0 from here). The settings for qemu are:
   • Memory: 8192MiB (this is enough for openqa (1GiB), 1 big (4GiB) and 1 small worker (2GiB)
   • Storage: 80GiB HD, Disk bus=SATA
   • Add Hardware/Filesystem: Driver=virtio-9p, Source path=/media/fast Target path=/external_iso_files, Export filesystem as readonly mount=Checked
2. Install to the harddisk, using the default answers, except for the following:
   • Region: matching my own region and country, but still using en_US.UTF-8 as locale
   • Hostname: openqa
   • Proxy: http://192.168.122.1:3142 (my local apt-cacher-ng)
   • Software: GNOME + ssh server
3. Use dep822 for apt (user=root)
   apt --yes modernize-sources
rm /etc/apt/sources.list.bak
4. Prepare to use fasttrack.debian.net (user=root)
   apt-get install fasttrack-archive-keyring
5. Activate fasttrack.debian.net (user=root)
   cat << EOF > /etc/apt/sources.list.d/fasttrack.sources
Types: deb
URIs: https://fasttrack.debian.net/debian-fasttrack/
Suites: trixie-fasttrack
Components: main
Signed-By: /usr/share/keyrings/fasttrack-archive-keyring.gpg
EOF
6. Install required and preferred packages (user=root)
   apt-get update # Update the package list
#apt-get install auto-apt-proxy # Disabled -> fasttrack automagically maps from http to https -> more investigation required
apt-get install aptitude # For looking at package dependencies etc.
apt-get install --yes ansible ssh-askpass rsync # For the configuration of the openqa server
7. Install the Debian package openqa and friends (user=root)
   apt-get install --yes -t trixie-fasttrack openqa openqa-worker
8. Configure openqa (for non-ssl) (user=root)
   /usr/share/openqa/script/configure-web-proxy
a2dissite 000-default
a2ensite openqa
systemctl reload apache2
9. Configure the login procedure (user=root)
   cat << EOF > /etc/openqa/openqa.ini.d/50-odn.ini
[global]
download_domains = reproducible-builds.org debian.org
branding=plain
[auth]
method = Fake
EOF
10. Create an easy access token (user=root)
    cat << EOF >> /etc/openqa/client.conf
[localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
EOF
11. Restart the openQA webui (user=root)
    systemctl restart openqa-webui
12. Create a new SSH key for Salsa (to be able to push changes to your own fork) (user=user)
    ssh-keygen -t ed25519 -C "VM Debian-openQA"
cat ~/.ssh/id_ed25519.pub
    -> paste in SSH Keys for Salsa (https://salsa.debian.org/-/user_settings/ssh_keys)
13. Prepare the git repository (@@@ no personal branch for now the default and my own fork) (user=geekotest)
    su geekotest -s /bin/bash
cd /var/lib/openqa/share/tests
git clone https://salsa.debian.org/qa/openqa/openqa-tests-debian.git debian
cd debian
## Personal branch skipped for now
##git remote add rclobus git@salsa.debian.org:rclobus/openqa-tests-debian.git
git fetch --all
14. Prepare SSH for password-less login (user=user)
    cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
ssh localhost # Accept the fingerprint with 'yes'
exit # Close the ssh connection
15. Configure the openQA server using ansible (user=user)
    cd /var/lib/openqa/share/tests/debian/configuration
# Ugly, needs improvement: create the groups manually and remove the deleted ids
# At the time of writing this, the highest id was 24
for i in $(seq 1 24); do openqa-cli api -X POST job_groups name=dummy_group$i; done
for i in $(seq 1 9) 15; do openqa-cli api -X DELETE job_groups/$i; done
ansible-playbook configure_openqa.yaml -i server_localhost.yaml
16. Mount the pre-existing ISOs that have been made available in qemu (user=root)
    echo "/external_iso_files /var/lib/openqa/share/factory/iso 9p trans=virtio,ro 0 0" >> /etc/fstab
systemctl daemon-reload
mount /var/lib/openqa/share/factory/iso
    Note: this mount point must be unmounted before updating openqa with apt-get

Prepare the openqa-worker

1. Disable the (currently) not needed slirp VDE service (user=root)
   systemctl disable openqa-slirpvde
2. Use the proxy when running installers under openQA (user=root)
   cat << EOF > /etc/openqa/workers.ini.d/50-http_proxy.ini
# Use the proxy (apt-cacher-ng) from the host
HTTP_PROXY = http://192.168.122.1:3142/
# The value http://localhost:9/ could be used for a 'network-access-denied' setting
EOF
3. Configure worker 2 with more memory (user=root)
   cat << EOF > /etc/openqa/workers.ini.d/55-bigmem-worker.ini
# x86_64 worker for jobs that require more memory
[2]
WORKER_CLASS=qemu_x86_64,qemu_x86_64_bigmem
EOF
4. Configure a worker for arm64 (via emulation, slow!) (user=root)
   cat << EOF > /etc/openqa/workers.ini.d/60-arm-worker.ini
# arm64 worker
[10]
WORKER_CLASS=qemu_aarch64
QEMU_NO_KVM=1
QEMUCPU=max
QEMUMACHINE=virt,usb=off
EOF
5. Configure the secrets (user=root)
   cat << EOF > /etc/openqa/client.conf
[localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
EOF
chown _openqa-worker /etc/openqa/client.conf
chmod 0400 /etc/openqa/client.conf
6. Start the cache service (user=root)
   systemctl restart openqa-worker-cacheservice
systemctl restart openqa-worker-cacheservice-minion
7. Enable the workers (user=root)
   systemctl enable openqa-worker@1
systemctl enable openqa-worker@2
systemctl enable openqa-worker@10
8. Start the workers (user=root)
   systemctl start openqa-worker@1
systemctl start openqa-worker@2
systemctl start openqa-worker@10

Run the first test

1. Download the netinst image (on the host):
   cd /media/fast
wcurl https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-13.1.0-amd64-netinst.iso
2. Run the netinst tests (user=user)
openqa-cli api -X POST isos ISO=debian-13.1.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1310

Creating and maintaining the job groups

Get information about all parent_job_groups and job_groups:
openqa-cli api -X GET job_groups | jq . > job_groups.json
openqa-cli api -X GET parent_groups | jq . > parent_groups.json
Extract identifiers:
cat job_groups.json | jq .[].id | sort -n
Extract existing names:
cat job_groups.json | jq .[].name | sort
Create the parent group for all live versions:
openqa-cli api -X POST parent_groups name="Debian Live" description="Live images for Debian"
#Some magic to fetch the ID of the parent group in $LIVE_PARENT
List the existing groups including sort order (with fancy jq):
openqa-cli api -X GET job_groups | jq '.[] | if .parent_id != null then (.parent_id|tostring) else "0" end + " " + if .sort_order != null then (.sort_order|tostring) else "-" end + " - " + (.id|tostring) + ": " + .name' | sort
Delete some groups:
for i in 2; do openqa-cli api -X DELETE job_groups/${i}; done
Create the groups (note, the group numbers shown here should be the id of the previous POST command):
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/17 name="Debian Live unstable" description="Debian Live images for unstable (sid). This is for early warning, before packages migrate to testing" parent_id=${LIVE_PARENT} sort_order=0
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/18 name="Debian Live testing" description="Debian Live images for testing (trixie). Trixie is not released yet, these are unofficial previews" parent_id=${LIVE_PARENT} sort_order=1
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/19 name="Debian Live stable" description="Debian Live images for stable (bookworm, Debian 12)." parent_id=${LIVE_PARENT} sort_order=2
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/20 name="Debian Live oldstable" description="Debian Live images for oldstable (bullseye, Debian 11)." parent_id=${LIVE_PARENT} sort_order=3

Creating and maintaining the product definitions

The script is: configuration/send_live_products.sh
Input file: live_products.json (beautified by ‘jq .’)
All products (7 ‘DE’s, Pure Blends and text-based images) need to be specified.

YAML definitions of the tests

See also the blog post: https://kalikiana.gitlab.io/post/2021-04-27-working-with-openqa-via-the-command-line/

Instead of this manual work, a Debianised version of tool.py from https://github.com/os-autoinst/opensuse-jobgroups should be used.
Adjust the lines below for the correct job group numbers:
openqa-cli api -X POST job_templates_scheduling/17 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/sid/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/18 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/trixie/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/19 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bookworm/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/20 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bullseye/' live_job_templates.yaml)"

Posting ISOs

openqa-cli api -X POST isos ISO=smallest-build_bullseye_20240210T124037Z.iso DISTRI=debian-live FLAVOR=smallest-build VERSION=bullseye ARCH=x86_64 BUILD=0001



Force feeding many of my ISOs:
for iso in \
cinnamon_bookworm_archive_20241109T101058Z.iso \
cinnamon_bullseye_20240210T124037Z.iso \
cinnamon_sid_git_20241201T081220Z.iso \
debian-junior_sid_20231219T201348Z.iso \
gnome_bookworm_archive_20241109T101058Z.iso \
gnome_bullseye_20240831T110215Z.iso \
gnome_sid_git_20250112T142808Z.iso \
gnome_testing_20240328T081620Z.iso \
gnome_trixie_archive_20250102T141247Z.iso \
junior_sid_20241112T081354Z.iso \
junior_testing_20241112T081355Z.iso \
junior_trixie_20240902T081351Z.iso \
kde_bookworm_archive_20241109T101058Z.iso \
kde_bullseye_20240210T124037Z.iso \
kde_sid_git_20250107T021348Z.iso \
kde_testing_20240331T141705Z_5050.iso \
kde_trixie_archive_20250102T021223Z.iso \
lxde_bookworm_archive_20241109T101058Z.iso \
lxde_bullseye_20240210T124037Z.iso \
lxde_sid_git_20241201T081220Z.iso \
lxqt_bookworm_archive_20241109T101058Z.iso \
lxqt_bullseye_20240210T124037Z.iso \
lxqt_sid_archive_20241216T081316Z.iso \
mate_bookworm_archive_20241109T101058Z.iso \
mate_bullseye_20240210T124037Z.iso \
mate_sid_git_20241201T081220Z.iso \
smallest-build_bookworm_archive_20241109T101058Z.iso \
smallest-build_bullseye_20240210T124037Z.iso \
smallest-build_sid_git_20241201T081220Z.iso \
standard_bookworm_archive_20241109T101058Z.iso \
standard_bullseye_20240210T124037Z.iso \
standard_sid_git_20250122T030756Z.iso \
st_testing_20240302T082324Z.iso \
st_trixie_20231230T081923Z.iso \
st_trixie_20231231T081144Z.iso \
xfce_bookworm_archive_20241109T101058Z.iso \
xfce_bullseye_20240210T124037Z.iso \
xfce_sid_git_20250111T022723Z.iso \
xfce_trixie_20241024T021249Z.iso; do \
FLAVOR=$(echo ${iso} | cut -d_ -f1); VERSION=$(echo ${iso} | cut -d_ -f2); if [ "${FLAVOR}" == "st" ]; then FLAVOR="standard"; fi; BUILD=$(echo ${iso} | awk '{ c=split($0, a, "_"); print substr(a,1, 8); }'); openqa-cli api -X POST isos ISO=${iso} DISTRI=debian-live FLAVOR=${FLAVOR} VERSION=${VERSION} ARCH=x86_64 BUILD=${BUILD}; done

Feeding errors -> because ‘testing’ is not equal to ‘trixie’:
gnome_testing_20240328T081620Z.iso
kde_testing_20240331T141705Z_5050.iso
kde_testing_20240331T141705Z_5050.iso
st_testing_20240302T082324Z.iso

The new looks:

Proposed changes/modifications/improvements to the Debian packaging

• Use the newer openqa-load-templates, i.e. use at least https://github.com/os-autoinst/openQA/commit/97d34ec3b6218c208595db379755fa79f01d2770
• Apply (my) reproducible patch to openqa-dump-templates, https://github.com/os-autoinst/openQA/commit/b135f552f2a477d6f7d98c3ff3e05be2c20e6132, which allows for the storage of the dump in git (to replace generated.json)
• Additional dependency: libgetopt-long-descriptive-perl because openqa-validate-yaml needs it.
• Adjust path in /etc/apparmor.d/usr.share.openqa.worker to the check_qemu_oom script (it is in /usr/lib/os-autoinst/script/check_qemu_oom)
• Enable access to dmesg to be able to detect OOM:
  https://forums.opensuse.org/t/how-to-give-non-root-user-permission-to-run-dmesg-without-su-or-sudo/176611
  sysctl -w kernel.dmesg_restrict=0
  From the docs (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt):
  > When dmesg_restrict is set set to (1), users must have CAP_SYSLOG to use dmesg(8). The kernel config option CONFIG_SECURITY_DMESG_RESTRICT sets the default value of dmesg_restrict.
  Debian restricted access to dmesg for security reasons in 2016 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842226#15)
  When both apparmor is fixed and dmesg is unrestricted, the following line will show on the incomplete job: Reason: backend died: QEMU was killed due to the system being out of memory
• Don’t point to SUSE, the following text line should be replaced:
  If you believe this is a bug you can open an issue. (URL = https://progress.opensuse.org/projects/openqav3/issues/new)

Fun side notes

Importing the configuration from openSUSE

openqa-dump-templates --host openqa.opensuse.org --json > opensuse.openqa.templates.json
openqa-load-templates opensuse.openqa.templates.json

Importing the configuration from Debian

openqa-dump-templates --host openqa.debian.net --json > debian.openqa.templates.json
openqa-load-templates debian.openqa.templates.json

Resetting the openQA database

Warning: this will remove all content from the database

As root:
systemctl stop openqa-webui
systemctl stop openqa-gru
systemctl stop openqa-websockets
systemctl stop openqa-scheduler

As geekotest:
dropdb openqa
There should not be a line like ‘DETAIL: There are N other sessions using the database.’, otherwise a few more openqa-related services need to be stopped

As root:
systemctl start openqa-webui

Clean the job queue

After the worker was turned off and the job queue was filled, remove all entries again, to prepare the openQA instance for production use.
for i in $(openqa-cli api -X GET jobs | jq .[][].id | sort -h); do openqa-cli api -X DELETE jobs/${i}; done

Notes

• Upgrading from bookworm to bookworm-backports is not easy. The ‘best’ solution came around proposal 33 in aptitude, and then I manually re-added git-lfs.
  

Steps to install and configure openQA in my own VM (version 2022-01):

1. Boot from a live image of GNOME unstable 2022-01-21T03:08Z
2. Install to a harddisk with Calamares
3. Install the Debian package openqa
   echo "deb http://deb.debian.org/debian sid main" >> /etc/apt/sources.list
apt-get update
apt-get install openqa
4. Configure openqa
   cd /etc/apache2/sites-enabled
ln -s ../sites-available/openqa.conf.template openqa.conf
# Replace #ServerName with 'ServerName localhost'
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_wstunnel
a2enmod rewrite
a2enmod expires
systemctl restart apache2
   Configure openqa version 2:
   /usr/share/openqa/script/configure-web-proxy
   -> However:
   26: cannot create /etc/apach2/vhosts.d/openqa.conf: Directory nonexistent
5. This might have been required: /usr/share/openqa/script/initdb
6. Configure the login procedure:
   Edit /etc/openqa/openqa.ini
   In the section [auth]: place ‘method = Fake‘
   Edit /etc/openqa/client.conf:
   [localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
   Restart the openQA webui:
   systemctl restart openqa-webui
7. Prepare salsa
   ssh-keygen -t ed25519 -C "VM Debian-openQA"
gedit ~/.ssh/id_ed25519.pub
   -> paste in SSH Keys for Salsa
8. Prepare the git repository:
   cd /var/lib/openqa/tests
git clone git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git debian
9. Initialise the default test settings:
   apt-get install python3-jsonschema
cd /var/lib/openqa/tests/debian
python3 fifloader.py templates.fif.json --update --load
10. Install a local openQA-worker:
    apt-get install openqa-worker
11. Download the netinst image and run it:
    cd /var/lib/openqa/share/factory/iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.2.0-amd64-netinst.iso
openqa-cli api -X POST isos ISO=debian-11.2.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1120
12. Issue: the tooltip with the guided tour did not disappear after being logged in:
    su geekotest
psql openqa
update users set feature_version=0;
\q