Steps to install openQA (a local copy of openqa.debian.net)

1. Boot in qemu from a netinst image (20250125) (8GB memory, 80GB HD, Filesystem: Driver=virtio-9p, Source path=/media/fast Target path=/var/lib/openqa/factory/iso, Export as readonly)
2. Install to a harddisk with GNOME desktop
3. Install preferred packages (proxy, clipboard management)
   apt-get update
apt-get install auto-apt-proxy
apt-get install aptitude spice-vdagent
4. Enable backports
   echo "deb http://deb.debian.org/debian bookworm-backports main" > /etc/apt/sources.list.d/backports.list
apt-get update
5. Activate clipboard management
   shutdown -r now
6. Install the Debian package openqa and friends
   apt-get install openqa openqa-worker os-autoinst
7. Configure openqa (for non-ssl)
   • Manual method (to be replaced)
     cd /etc/apache2/sites-enabled
ln -s ../sites-available/openqa.conf.template openqa.conf
# Replace #ServerName with 'ServerName localhost'
a2enmod headers proxy proxy_http proxy_wstunnel rewrite expires
systemctl restart apache2
   • Automated method (after MR4 gets merged and the updated package is released)
     /usr/share/openqa/script/configure-web-proxy
a2dissite 000-default
systemctl restart apache2
8. Initialise the database -> probably not needed, since it will be started by openqa-webui.service
   systemctl start openqa-setup-db
9. Configure the login procedure:
   Edit /etc/openqa/openqa.ini
   In the section [auth]: place ‘method = Fake‘
   Edit /etc/openqa/client.conf:
   [localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
10. Additional configuration to /etc/openqa/openqa.ini:
    download_domains = reproducible-builds.org debian.org
11. Restart the openQA webui: (aa-enforce should be in the postinst)
    aa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa
systemctl restart openqa-webui
12. Prepare salsa
    ssh-keygen -t ed25519 -C "VM Debian-openQA"
cat ~/.ssh/id_ed25519.pub
    -> paste in SSH Keys for Salsa (https://salsa.debian.org/-/user_settings/ssh_keys)
13. Prepare the git repository:
    cd /var/lib/openqa/tests
git clone https://salsa.debian.org/qa/openqa/openqa-tests-debian.git debian
cd debian
git remote add rclobus git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git
git fetch --all
14. The job groups have been manually constructed. Therefore they must be imported from the openqa.debian.net instance:
    openqa-dump-templates --host openqa.debian.net --json > debian.openqa.templates.json
openqa-load-templates debian.openqa.templates.json
15. Initialise the default test settings (this step needs to be repeated when the job definitions have changed):
    apt-get install python3-jsonschema
cd /var/lib/openqa/tests/debian
python3 fifloader.py templates.fif.json --update --load
16. Mount the pre-existing ISOs as made available in qemu on the host
    echo "/var/lib/openqa/factory/iso /var/lib/openqa/factory/iso 9p trans=virtio 0 0" >> /etc/fstab
mount /var/lib/openqa/factory/iso
    Note: this mount point must be unmounted when updating openqa with apt-get
17. Download the netinst image (on the host):
    cd /media/fast
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.9.0-amd64-netinst.iso
18. Install a local openQA-worker:
    apt-get install openqa-worker
19. Disable the (currently) not needed slirp VDE service
    systemctl disable openqa-slirpvde
20. Configuration in /etc/openqa/workers.ini:
    # Use the proxy (apt-cacher-ng) from the host
    HTTP_PROXY = http://192.168.122.1:3142/
    # The value http://localhost:9/ could be used for a ‘network-access-denied’ setting
    A new section at the end, to add an arm64 worker (via emulation, slow!)
    # arm64 worker
[10]
WORKER_CLASS=qemu_aarch64
QEMU_NO_KVM=1
QEMUCPU=max
QEMUMACHINE=virt,usb=off
21. Start the cache service:
    systemctl restart openqa-worker-cacheservice
22. Start the first worker
    systemctl enable openqa-worker@1
23. Run the netinst tests:
openqa-cli api -X POST isos ISO=debian-12.9.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1290

Migrating away from DISTRI=debian FLAVOR=live-build

Why? The current overview of the live images is a big mixup of oldstable, stable, testing and unstable. It is hard to get a good overview of each build. Using tags, at least the official testing images can be made visible more prominently and the unstable daily images will be on the top of the list. However, rebuilding an (old)stable test is typically impossible, because the ISO files (due to their old timestamp) will have been deleted. Individual quota per distribution can be configured.

The new structure will be:

	old	new	compareD to netinst
DISTRI	debian	debian-live	debian
FLAVOR	live-build	gnome	netinst-iso
VERSION	testing_gnome	testing	testing
BUILD	timestamp	timestamp	timestamp
DESKTOP	gnome	possibly not needed any more, will be a copy of FLAVOR	used to select the DE during installation
group	14 = all live images	a group for oldstable, stable, testing and unstable each

Steps done to perform this migration:

• Git: Adjust lib/Debian/Bootwalker.pm to learn about DISTRI=’debian-live’
• Git: Create symlink in products from debian-live to debian
• Host: Create symlink in /var/lib/openqa/share/tests:
  ln -s debian debian-live
chown geekotest:geekotest -h debian-live
• Manual creation of job_groups and parent_groups
• Run configuration/send_live_products.sh on odn
• Manual application of the job templates YAML files

Todo:

• Adjust Jenkins to start the images differently
• Modify our maintenance structure, to remove ‘templates.fif.json’ in favour of the Debianised variant of https://github.com/os-autoinst/opensuse-jobgroups
  It looks like git submodules (https://git-scm.com/book/en/v2/Git-Tools-Submodules) might be a suitable way to get the original version of SUSEs tool.py and still be able to update whenever a newer version becomes available.

Creating and maintaining the job groups

Get information about all parent_job_groups and job_groups:
openqa-cli api -X GET job_groups | jq . > job_groups.json
openqa-cli api -X GET parent_groups | jq . > parent_groups.json
Extract identifiers:
cat job_groups.json | jq .[].id | sort -n
Extract existing names:
cat job_groups.json | jq .[].name | sort
Create the parent group for all live versions:
openqa-cli api -X POST parent_groups name="Debian Live" description="Live images for Debian"
#Some magic to fetch the ID of the parent group in $LIVE_PARENT
List the existing groups including sort order (with fancy jq):
openqa-cli api -X GET job_groups | jq '.[] | if .parent_id != null then (.parent_id|tostring) else "0" end + " " + if .sort_order != null then (.sort_order|tostring) else "-" end + " - " + (.id|tostring) + ": " + .name' | sort
Delete some groups:
for i in 2; do openqa-cli api -X DELETE job_groups/${i}; done
Create the groups (note, the group numbers shown here should be the id of the previous POST command):
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/17 name="Debian Live unstable" description="Debian Live images for unstable (sid). This is for early warning, before packages migrate to testing" parent_id=${LIVE_PARENT} sort_order=0
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/18 name="Debian Live testing" description="Debian Live images for testing (trixie). Trixie is not released yet, these are unofficial previews" parent_id=${LIVE_PARENT} sort_order=1
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/19 name="Debian Live stable" description="Debian Live images for stable (bookworm, Debian 12)." parent_id=${LIVE_PARENT} sort_order=2
openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT}
openqa-cli api -X PUT job_groups/20 name="Debian Live oldstable" description="Debian Live images for oldstable (bullseye, Debian 11)." parent_id=${LIVE_PARENT} sort_order=3

Creating and maintaining the product definitions

The script is: configuration/send_live_products.sh
Input file: live_products.json (beautified by ‘jq .’)
All products (7 ‘DE’s, Pure Blends and text-based images) need to be specified.

YAML definitions of the tests

See also the blog post: https://kalikiana.gitlab.io/post/2021-04-27-working-with-openqa-via-the-command-line/

Instead of this manual work, a Debianised version of tool.py from https://github.com/os-autoinst/opensuse-jobgroups should be used.
Adjust the lines below for the correct job group numbers:
openqa-cli api -X POST job_templates_scheduling/17 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/sid/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/18 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/trixie/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/19 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bookworm/' live_job_templates.yaml)"
openqa-cli api -X POST job_templates_scheduling/20 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bullseye/' live_job_templates.yaml)"

Posting ISOs

openqa-cli api -X POST isos ISO=smallest-build_bullseye_20240210T124037Z.iso DISTRI=debian-live FLAVOR=smallest-build VERSION=bullseye ARCH=x86_64 BUILD=0001



Force feeding many of my ISOs:
for iso in \
cinnamon_bookworm_archive_20241109T101058Z.iso \
cinnamon_bullseye_20240210T124037Z.iso \
cinnamon_sid_git_20241201T081220Z.iso \
debian-junior_sid_20231219T201348Z.iso \
gnome_bookworm_archive_20241109T101058Z.iso \
gnome_bullseye_20240831T110215Z.iso \
gnome_sid_git_20250112T142808Z.iso \
gnome_testing_20240328T081620Z.iso \
gnome_trixie_archive_20250102T141247Z.iso \
junior_sid_20241112T081354Z.iso \
junior_testing_20241112T081355Z.iso \
junior_trixie_20240902T081351Z.iso \
kde_bookworm_archive_20241109T101058Z.iso \
kde_bullseye_20240210T124037Z.iso \
kde_sid_git_20250107T021348Z.iso \
kde_testing_20240331T141705Z_5050.iso \
kde_trixie_archive_20250102T021223Z.iso \
lxde_bookworm_archive_20241109T101058Z.iso \
lxde_bullseye_20240210T124037Z.iso \
lxde_sid_git_20241201T081220Z.iso \
lxqt_bookworm_archive_20241109T101058Z.iso \
lxqt_bullseye_20240210T124037Z.iso \
lxqt_sid_archive_20241216T081316Z.iso \
mate_bookworm_archive_20241109T101058Z.iso \
mate_bullseye_20240210T124037Z.iso \
mate_sid_git_20241201T081220Z.iso \
smallest-build_bookworm_archive_20241109T101058Z.iso \
smallest-build_bullseye_20240210T124037Z.iso \
smallest-build_sid_git_20241201T081220Z.iso \
standard_bookworm_archive_20241109T101058Z.iso \
standard_bullseye_20240210T124037Z.iso \
standard_sid_git_20250122T030756Z.iso \
st_testing_20240302T082324Z.iso \
st_trixie_20231230T081923Z.iso \
st_trixie_20231231T081144Z.iso \
xfce_bookworm_archive_20241109T101058Z.iso \
xfce_bullseye_20240210T124037Z.iso \
xfce_sid_git_20250111T022723Z.iso \
xfce_trixie_20241024T021249Z.iso; do \
FLAVOR=$(echo ${iso} | cut -d_ -f1); VERSION=$(echo ${iso} | cut -d_ -f2); if [ "${FLAVOR}" == "st" ]; then FLAVOR="standard"; fi; BUILD=$(echo ${iso} | awk '{ c=split($0, a, "_"); print substr(a,1, 8); }'); openqa-cli api -X POST isos ISO=${iso} DISTRI=debian-live FLAVOR=${FLAVOR} VERSION=${VERSION} ARCH=x86_64 BUILD=${BUILD}; done

Feeding errors -> because ‘testing’ is not equal to ‘trixie’:
gnome_testing_20240328T081620Z.iso
kde_testing_20240331T141705Z_5050.iso
kde_testing_20240331T141705Z_5050.iso
st_testing_20240302T082324Z.iso

The new looks:

Proposed changes/modifications/improvements to the Debian packaging

• Use the newer openqa-load-templates, i.e. use at least https://github.com/os-autoinst/openQA/commit/97d34ec3b6218c208595db379755fa79f01d2770
• Apply (my) reproducible patch to openqa-dump-templates, https://github.com/os-autoinst/openQA/commit/b135f552f2a477d6f7d98c3ff3e05be2c20e6132, which allows for the storage of the dump in git (to replace generated.json)
• Additional dependency: libgetopt-long-descriptive-perl because openqa-validate-yaml needs it.
• Adjust path in /etc/apparmor.d/usr.share.openqa.worker to the check_qemu_oom script (it is in /usr/lib/os-autoinst/script/check_qemu_oom)
• Enable access to dmesg to be able to detect OOM:
  https://forums.opensuse.org/t/how-to-give-non-root-user-permission-to-run-dmesg-without-su-or-sudo/176611
  sysctl -w kernel.dmesg_restrict=0
  From the docs (https://www.kernel.org/doc/Documentation/sysctl/kernel.txt):
  > When dmesg_restrict is set set to (1), users must have CAP_SYSLOG to use dmesg(8). The kernel config option CONFIG_SECURITY_DMESG_RESTRICT sets the default value of dmesg_restrict.
  Debian restricted access to dmesg for security reasons in 2016 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842226#15)
  When both apparmor is fixed and dmesg is unrestricted, the following line will show on the incomplete job: Reason: backend died: QEMU was killed due to the system being out of memory
• Don’t point to SUSE, the following text line should be replaced:
  If you believe this is a bug you can open an issue. (URL = https://progress.opensuse.org/projects/openqav3/issues/new)

Fun side notes

Importing the configuration from openSUSE

openqa-dump-templates --host openqa.opensuse.org --json > opensuse.openqa.templates.json
openqa-load-templates opensuse.openqa.templates.json

Importing the configuration from Debian

openqa-dump-templates --host openqa.debian.net --json > debian.openqa.templates.json
openqa-load-templates debian.openqa.templates.json

Resetting the openQA database

Warning: this will remove all content from the database

As root:
systemctl stop openqa-webui
systemctl stop openqa-gru
systemctl stop openqa-websockets
systemctl stop openqa-scheduler

As geekotest:
dropdb openqa
There should not be a line like ‘DETAIL: There are N other sessions using the database.’, otherwise a few more openqa-related services need to be stopped

As root:
systemctl start openqa-webui

Clean the job queue

After the worker was turned off and the job queue was filled, remove all entries again, to prepare the openQA instance for production use.
for i in $(openqa-cli api -X GET jobs | jq .[][].id | sort -h); do openqa-cli api -X DELETE jobs/${i}; done

Notes

• Upgrading from bookworm to bookworm-backports is not easy. The ‘best’ solution came around proposal 33 in aptitude, and then I manually re-added git-lfs.
  

Steps to install and configure openQA in my own VM (version 2022-01):

1. Boot from a live image of GNOME unstable 2022-01-21T03:08Z
2. Install to a harddisk with Calamares
3. Install the Debian package openqa
   echo "deb http://deb.debian.org/debian sid main" >> /etc/apt/sources.list
apt-get update
apt-get install openqa
4. Configure openqa
   cd /etc/apache2/sites-enabled
ln -s ../sites-available/openqa.conf.template openqa.conf
# Replace #ServerName with 'ServerName localhost'
a2enmod headers
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_wstunnel
a2enmod rewrite
a2enmod expires
systemctl restart apache2
   Configure openqa version 2:
   /usr/share/openqa/script/configure-web-proxy
   -> However:
   26: cannot create /etc/apach2/vhosts.d/openqa.conf: Directory nonexistent
5. This might have been required: /usr/share/openqa/script/initdb
6. Configure the login procedure:
   Edit /etc/openqa/openqa.ini
   In the section [auth]: place ‘method = Fake‘
   Edit /etc/openqa/client.conf:
   [localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
   Restart the openQA webui:
   systemctl restart openqa-webui
7. Prepare salsa
   ssh-keygen -t ed25519 -C "VM Debian-openQA"
gedit ~/.ssh/id_ed25519.pub
   -> paste in SSH Keys for Salsa
8. Prepare the git repository:
   cd /var/lib/openqa/tests
git clone git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git debian
9. Initialise the default test settings:
   apt-get install python3-jsonschema
cd /var/lib/openqa/tests/debian
python3 fifloader.py templates.fif.json --update --load
10. Install a local openQA-worker:
    apt-get install openqa-worker
11. Download the netinst image and run it:
    cd /var/lib/openqa/share/factory/iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.2.0-amd64-netinst.iso
openqa-cli api -X POST isos ISO=debian-11.2.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1120
12. Issue: the tooltip with the guided tour did not disappear after being logged in:
    su geekotest
psql openqa
update users set feature_version=0;
\q