It would be nice to have live images for Debian for Bookworm again.
However…
There are two major bottlenecks:
Human resources
The purpose/scope of live images
Human resources
As noted in the discussions, the live images cannot be maintained by a single person. It must be maintained by a group. So far nobody has really raised their hand and offered to push forward the support for live images.
I’ve dug into the archives, live-build has some painful history. However, it is a tool that is available at the moment and it is used for several other live-CDs as well.
At release-day the amount of testing live images is huge. It is a lot of manual work to verify the new images. Automating will reduce the work, but it cannot replace running on real hardware.
Two tools will help a lot. They have been added in the past years:
Jenkins. Ongoing. This generates the images twice and tests whether they are reproducible.
openQA. In progress. This will test many features of the live image. It simulates a computer from the moment it boots. It does not require special configuration/packages inside the image that is under test.
The purpose/scope of live images
If the live images are to be generated for Bookworm again, it needs to be discussed first what the expectations are for live images.
Some features:
Installer, with full off-line installation support:
Debian-installer, to install from the boot menu
Calamares installer, to install from the live environment
Internationalization/Localization support: choose your locale/language at boot
Accessibility support: speech and high-contrast modes
All major desktops are provided (currently Bullseye has 8)
Security support
Will work properly both on-line and off-line
Easy support for persistence
The installed software represents a workable system
The installed software should have as few as possible pre-configured options or special tweaks
The image must be generated reproducibly
Unofficial support for non-free firmware
Support for new blends like Debian Jr.
Maintenance should be relatively low
Which tool to use? live-build, live-wrapper, FAI, …
Hardware: at least amd64. Bullseye also has i386
Support several installation options:
Explicit support for ISO chainloading (a single UBS boot stick could hold several ISO image files), similar to rufus, YUMI, Ventoy, …
dd if=image.iso of=/dev/sdMyUSBStick
mount the ISO image, copy all files to the USB stick
Support BIOS and UEFI boot (secure and non-secure)
Other features, …
Documentation of the the features mentioned above
Some desired features might contradict other features.
Personally I think that the following features are out-of-scope:
A small installer -> use FAI instead
A rescue image -> other solutions exist already
A small image with only ‘C’ and no i18n/l10n/a18y support (but I think this might be a good addition to the list of current images)
Some functionality tests in openQA (not published yet, but running in a local, private openQA instance)
Extracting the rebuild script from Jenkins and including a freshly-built installer (using their daily-build script)
Gentle prodding
Personally I’m quite hesitant to volunteer, because I don’t know how much time I’ll have in the near future. However, I’m willing to help to get things started again.
Activate all GRUB/syslinux menu entries to verify that they perform correctly
Other openQA tests are planned/under review:
Verify the correct functionality of some applications in the live environment
Run the Debian installer
Run the Calamares installer
Verify the installed image (both from the boot menu and the checksum)
I’ve reviewed only amd64. (The snapshot mirror that I use carries only amd64)
I propose the following testing chain, which enforces reproducibility and functionality:
A) Use Jenkins to generate an ISO image based on the latest timestamp from the snapshot.debian.org/snapshot.notset.fr archive
B) Use Jenkins to generate the image again with the same timestamp
PASS/FAIL: If checksum is identical continue, otherwise abort
C) Run openQA tests
- Test all boot menu entries
- The live environment should start and the main applications should work
- The Debian installer from the boot menu should function properly
- The Calamares installer from the live environment should function properly
- After the installer is finished, the newly installed Debian should boot and the main applications should work
- etc...
PASS/FAIL: If all openQA tests pass (or have softfail) continue, otherwise abort
D) Publish the image on https://www.debian.org/CD/
This means that the live images will be published after some delay. (A snapshot needs to exist and the tests also take some amount of time). However, it can be fully automated, and many trivial issues can be caught early, during daily tests.
Cloning the fork repo for the very first time – it does not have any LFS files copied yet, so you must clone it without the LFS files (otherwise cloning would fail)
export GIT_LFS_SKIP_SMUDGE=y
git clone <address of your fork>
Synchronizing upstream -> fork (copying LFS files to your fork) – initially and each time you git fetch the upstream
Install the Debian package openqa and friends apt-get install openqa openqa-worker os-autoinst
Configure openqa (for non-ssl)
Manual method (to be replaced) cd /etc/apache2/sites-enabled ln -s ../sites-available/openqa.conf.template openqa.conf # Replace #ServerName with 'ServerName localhost' a2enmod headers proxy proxy_http proxy_wstunnel rewrite expires systemctl restart apache2
Automated method (after MR4 gets merged and the updated package is released) /usr/share/openqa/script/configure-web-proxy a2dissite 000-default systemctl restart apache2
Initialise the database -> probably not needed, since it will be started by openqa-webui.service systemctl start openqa-setup-db
Configure the login procedure: Edit /etc/openqa/openqa.ini In the section [auth]: place ‘method = Fake‘ Edit /etc/openqa/client.conf: [localhost] key = 1234567890ABCDEF secret = 1234567890ABCDEF
Additional configuration to /etc/openqa/openqa.ini: download_domains = reproducible-builds.org debian.org
Restart the openQA webui: (aa-enforce should be in the postinst) aa-enforce /etc/apparmor.d/usr.share.openqa.script.openqa systemctl restart openqa-webui
Prepare the git repository: cd /var/lib/openqa/tests git clone https://salsa.debian.org/qa/openqa/openqa-tests-debian.git debian cd debian git remote add rclobus git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git git fetch --all
The job groups have been manually constructed. Therefore they must be imported from the openqa.debian.net instance: openqa-dump-templates --host openqa.debian.net --json > debian.openqa.templates.json openqa-load-templates debian.openqa.templates.json
Initialise the default test settings (this step needs to be repeated when the job definitions have changed): apt-get install python3-jsonschema cd /var/lib/openqa/tests/debian python3 fifloader.py templates.fif.json --update --load
Mount the pre-existing ISOs as made available in qemu on the host echo "/var/lib/openqa/factory/iso /var/lib/openqa/factory/iso 9p trans=virtio 0 0" >> /etc/fstab mount /var/lib/openqa/factory/iso Note: this mount point must be unmounted when updating openqa with apt-get
Download the netinst image (on the host): cd /media/fast wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.9.0-amd64-netinst.iso
Install a local openQA-worker: apt-get install openqa-worker
Disable the (currently) not needed slirp VDE service systemctl disable openqa-slirpvde
Configuration in /etc/openqa/workers.ini: # Use the proxy (apt-cacher-ng) from the host HTTP_PROXY = http://192.168.122.1:3142/ # The value http://localhost:9/ could be used for a ‘network-access-denied’ setting A new section at the end, to add an arm64 worker (via emulation, slow!) # arm64 worker [10] WORKER_CLASS=qemu_aarch64 QEMU_NO_KVM=1 QEMUCPU=max QEMUMACHINE=virt,usb=off
Start the cache service: systemctl restart openqa-worker-cacheservice
Start the first worker systemctl enable openqa-worker@1
Run the netinst tests: openqa-cli api -X POST isos ISO=debian-12.9.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1290
Migrating away from DISTRI=debian FLAVOR=live-build
Why? The current overview of the live images is a big mixup of oldstable, stable, testing and unstable. It is hard to get a good overview of each build. Using tags, at least the official testing images can be made visible more prominently and the unstable daily images will be on the top of the list. However, rebuilding an (old)stable test is typically impossible, because the ISO files (due to their old timestamp) will have been deleted. Individual quota per distribution can be configured.
The new structure will be:
old
new
compareD to netinst
DISTRI
debian
debian-live
debian
FLAVOR
live-build
gnome
netinst-iso
VERSION
testing_gnome
testing
testing
BUILD
timestamp
timestamp
timestamp
DESKTOP
gnome
possibly not needed any more, will be a copy of FLAVOR
used to select the DE during installation
group
14 = all live images
a group for oldstable, stable, testing and unstable each
Steps done to perform this migration:
Git: Adjust lib/Debian/Bootwalker.pm to learn about DISTRI=’debian-live’
Git: Create symlink in products from debian-live to debian
Get information about all parent_job_groups and job_groups: openqa-cli api -X GET job_groups | jq . > job_groups.json openqa-cli api -X GET parent_groups | jq . > parent_groups.json Extract identifiers: cat job_groups.json | jq .[].id | sort -n Extract existing names: cat job_groups.json | jq .[].name | sort Create the parent group for all live versions: openqa-cli api -X POST parent_groups name="Debian Live" description="Live images for Debian" #Some magic to fetch the ID of the parent group in $LIVE_PARENT List the existing groups including sort order (with fancy jq): openqa-cli api -X GET job_groups | jq '.[] | if .parent_id != null then (.parent_id|tostring) else "0" end + " " + if .sort_order != null then (.sort_order|tostring) else "-" end + " - " + (.id|tostring) + ": " + .name' | sort Delete some groups: for i in 2; do openqa-cli api -X DELETE job_groups/${i}; done Create the groups (note, the group numbers shown here should be the id of the previous POST command): openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT} openqa-cli api -X PUT job_groups/17 name="Debian Live unstable" description="Debian Live images for unstable (sid). This is for early warning, before packages migrate to testing" parent_id=${LIVE_PARENT} sort_order=0 openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT} openqa-cli api -X PUT job_groups/18 name="Debian Live testing" description="Debian Live images for testing (trixie). Trixie is not released yet, these are unofficial previews" parent_id=${LIVE_PARENT} sort_order=1 openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT} openqa-cli api -X PUT job_groups/19 name="Debian Live stable" description="Debian Live images for stable (bookworm, Debian 12)." parent_id=${LIVE_PARENT} sort_order=2 openqa-cli api -X POST job_groups name="temp" parent_id=${LIVE_PARENT} openqa-cli api -X PUT job_groups/20 name="Debian Live oldstable" description="Debian Live images for oldstable (bullseye, Debian 11)." parent_id=${LIVE_PARENT} sort_order=3
Creating and maintaining the product definitions
The script is: configuration/send_live_products.sh Input file: live_products.json (beautified by ‘jq .’) All products (7 ‘DE’s, Pure Blends and text-based images) need to be specified.
Instead of this manual work, a Debianised version of tool.py from https://github.com/os-autoinst/opensuse-jobgroups should be used. Adjust the lines below for the correct job group numbers: openqa-cli api -X POST job_templates_scheduling/17 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/sid/' live_job_templates.yaml)" openqa-cli api -X POST job_templates_scheduling/18 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/trixie/' live_job_templates.yaml)" openqa-cli api -X POST job_templates_scheduling/19 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bookworm/' live_job_templates.yaml)" openqa-cli api -X POST job_templates_scheduling/20 schema=JobTemplates-01.yaml template="$(sed -e 's/distribution/bullseye/' live_job_templates.yaml)"
Posting ISOs
openqa-cli api -X POST isos ISO=smallest-build_bullseye_20240210T124037Z.iso DISTRI=debian-live FLAVOR=smallest-build VERSION=bullseye ARCH=x86_64 BUILD=0001
Feeding errors -> because ‘testing’ is not equal to ‘trixie’: gnome_testing_20240328T081620Z.iso kde_testing_20240331T141705Z_5050.iso kde_testing_20240331T141705Z_5050.iso st_testing_20240302T082324Z.iso
The new looks:
Proposed changes/modifications/improvements to the Debian packaging
As geekotest: dropdb openqa There should not be a line like ‘DETAIL: There are N other sessions using the database.’, otherwise a few more openqa-related services need to be stopped
As root: systemctl start openqa-webui
Clean the job queue
After the worker was turned off and the job queue was filled, remove all entries again, to prepare the openQA instance for production use. for i in $(openqa-cli api -X GET jobs | jq .[][].id | sort -h); do openqa-cli api -X DELETE jobs/${i}; done
Notes
Upgrading from bookworm to bookworm-backports is not easy. The ‘best’ solution came around proposal 33 in aptitude, and then I manually re-added git-lfs.
Steps to install and configure openQA in my own VM (version 2022-01):
Boot from a live image of GNOME unstable 2022-01-21T03:08Z
This might have been required: /usr/share/openqa/script/initdb
Configure the login procedure: Edit /etc/openqa/openqa.ini In the section [auth]: place ‘method = Fake‘ Edit /etc/openqa/client.conf: [localhost] key = 1234567890ABCDEF secret = 1234567890ABCDEF Restart the openQA webui: systemctl restart openqa-webui
Prepare salsa ssh-keygen -t ed25519 -C "VM Debian-openQA" gedit ~/.ssh/id_ed25519.pub -> paste in SSH Keys for Salsa
Prepare the git repository: cd /var/lib/openqa/tests git clone git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git debian
Initialise the default test settings: apt-get install python3-jsonschema cd /var/lib/openqa/tests/debian python3 fifloader.py templates.fif.json --update --load
Install a local openQA-worker: apt-get install openqa-worker
Download the netinst image and run it: cd /var/lib/openqa/share/factory/iso wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.2.0-amd64-netinst.iso openqa-cli api -X POST isos ISO=debian-11.2.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1120
Issue: the tooltip with the guided tour did not disappear after being logged in: su geekotest psql openqa update users set feature_version=0; \q
