Tag Archives: openQA

Live images for Debian 12 (bookworm)

Leave a reply

Deja-vu

This is a continuation of previous discussions. See Documenting the generation of the live images in Debian, Porting the standard image from live-wrapper to live-build and its follow-ups and Replacing live-wrapper for live images by live-build?. Some testing was discussed here: Some Debian Live testing.

Current state

The generation of live images for bookworm and sid have been turned off, see Status of weekly live builds. Sometimes questions are raised about the live images: cdimage weekly-live-builds not regenerating and #1006122

Desired state

It would be nice to have live images for Debian for Bookworm again.

However…

There are two major bottlenecks:

  1. Human resources
  2. The purpose/scope of live images

Human resources

As noted in the discussions, the live images cannot be maintained by a single person. It must be maintained by a group. So far nobody has really raised their hand and offered to push forward the support for live images.

I’ve dug into the archives, live-build has some painful history. However, it is a tool that is available at the moment and it is used for several other live-CDs as well.

At release-day the amount of testing live images is huge. It is a lot of manual work to verify the new images. Automating will reduce the work, but it cannot replace running on real hardware.

Two tools will help a lot. They have been added in the past years:

  1. Jenkins. Ongoing. This generates the images twice and tests whether they are reproducible.
  2. openQA. In progress. This will test many features of the live image. It simulates a computer from the moment it boots. It does not require special configuration/packages inside the image that is under test.

The purpose/scope of live images

If the live images are to be generated for Bookworm again, it needs to be discussed first what the expectations are for live images.

Some features:

  • Installer, with full off-line installation support:
    • Debian-installer, to install from the boot menu
    • Calamares installer, to install from the live environment
  • Internationalization/Localization support: choose your locale/language at boot
  • Accessibility support: speech and high-contrast modes
  • All major desktops are provided (currently Bullseye has 8)
  • Security support
  • Will work properly both on-line and off-line
  • Easy support for persistence
  • The installed software represents a workable system
  • The installed software should have as few as possible pre-configured options or special tweaks
  • The image must be generated reproducibly
  • Unofficial support for non-free firmware
  • Support for new blends like Debian Jr.
  • Maintenance should be relatively low
  • Which tool to use? live-build, live-wrapper, FAI, …
  • Hardware: at least amd64. Bullseye also has i386
  • Support several installation options:
    • Explicit support for ISO chainloading (a single UBS boot stick could hold several ISO image files), similar to rufus, YUMI, Ventoy, …
    • dd if=image.iso of=/dev/sdMyUSBStick
    • mount the ISO image, copy all files to the USB stick
  • Support BIOS and UEFI boot (secure and non-secure)
  • Other features, …
  • Documentation of the the features mentioned above

Some desired features might contradict other features.

Personally I think that the following features are out-of-scope:

  • A small installer -> use FAI instead
  • A rescue image -> other solutions exist already
  • A small image with only ‘C’ and no i18n/l10n/a18y support (but I think this might be a good addition to the list of current images)

My involvement

So far I’ve been working on:

  • Documentation improvements of live-manual
  • Reproducibility fixes for live-build (Wiki)
  • Some functionality tests in openQA (not published yet, but running in a local, private openQA instance)
  • Extracting the rebuild script from Jenkins and including a freshly-built installer (using their daily-build script)
  • Gentle prodding

Personally I’m quite hesitant to volunteer, because I don’t know how much time I’ll have in the near future. However, I’m willing to help to get things started again.

Some background (kindly provided by pabs)

A list of derivatives that use live-build: https://wiki.debian.org/Derivatives/CensusFull

Some other tools that can generate live images: https://wiki.debian.org/SystemBuildTools

Note: This information is somewhat outdated.

Where are we now (2022-05-19)?

  • live-wrapper is not available in Bookworm any more (#1009282), the request for fixes or alternatives dates from 2018-09.
  • live-build can be used to generate reproducible live images.
  • FAI has been proposed to generate live images.
  • All major desktops have reproducible images (as tested by Jenkins)
  • openQA testing is active:
    • Activate all GRUB/syslinux menu entries to verify that they perform correctly
  • Other openQA tests are planned/under review:
    • Verify the correct functionality of some applications in the live environment
    • Run the Debian installer
    • Run the Calamares installer
    • Verify the installed image (both from the boot menu and the checksum)
  • I’ve reviewed only amd64. (The snapshot mirror that I use carries only amd64)
  • I propose the following testing chain, which enforces reproducibility and functionality:
A) Use Jenkins to generate an ISO image based on the latest timestamp from the snapshot.debian.org/snapshot.notset.fr archive
B) Use Jenkins to generate the image again with the same timestamp
PASS/FAIL: If checksum is identical continue, otherwise abort
C) Run openQA tests
   - Test all boot menu entries
   - The live environment should start and the main applications should work
   - The Debian installer from the boot menu should function properly
   - The Calamares installer from the live environment should function properly
   - After the installer is finished, the newly installed Debian should boot and the main applications should work
   - etc...
PASS/FAIL: If all openQA tests pass (or have softfail) continue, otherwise abort
D) Publish the image on https://www.debian.org/CD/

This means that the live images will be published after some delay. (A snapshot needs to exist and the tests also take some amount of time). However, it can be fully automated, and many trivial issues can be caught early, during daily tests.

Git-related commands

Initialisation

git remote add upstream git@salsa.debian.org….
git config –global –add user.name “Roland Clobus”
git config –global –add user.email “rclobus@rclobus.nl”

Protected branch (as e.g. for openQA)

git checkout debian
git pull upstream debian
# do stuff
git commit
git push upstream debian

Git-lfs and forks

After forking openqa-tests-debian, the LFS part was not cloned. Stackoverflow found a solution:

<Begin quote>

https://stackoverflow.com/questions/55359067/what-is-the-workflow-for-git-lfs-with-forks

git clone <address of your fork>
  1. Cloning the fork repo for the very first time – it does not have any LFS files copied yet, so you must clone it without the LFS files (otherwise cloning would fail)
export GIT_LFS_SKIP_SMUDGE=y
git clone <address of your fork>
  1. Synchronizing upstream -> fork (copying LFS files to your fork) – initially and each time you git fetch the upstream
git lfs fetch --all <upstream>
git lfs push --all <fork>
  1. Synchronizing fork -> upstream (only the current branch) – each time after your PR got merged
git lfs fetch <fork>
git lfs push <upstream>

<End quote>

Beautification of local history

git commit –amend –no-edit –author “Roland Clobus <rclobus@rclobus.nl>”

git commit –amend –no-edit –date “$(date)”

Cleanup

Branch cleanup: https://stackoverflow.com/questions/6127328/how-do-i-delete-all-git-branches-which-have-been-merged

git branch --merged

git branch -d merged_branch_name

Time travel

Go back in time to checkout a specific timestamp: https://stackoverflow.com/questions/6990484/how-to-checkout-in-git-by-date

git checkout `git rev-list -n 1 --first-parent --before="2022-06-01 00:00Z" upstream/debian`

openQA

Steps to install and configure openQA in my own VM:

  1. Boot from a live image of GNOME unstable 2022-01-21T03:08Z
  2. Install to a harddisk with Calamares
  3. Install the Debian package openqa
    echo "deb http://deb.debian.org/debian sid main" >> /etc/apt/sources.list
    apt-get update
    apt-get install openqa
  4. Configure openqa
    cd /etc/apache2/sites-enabled
    ln -s ../sites-available/openqa.conf.template openqa.conf
    # Replace #ServerName with 'ServerName localhost'
    a2enmod headers
    a2enmod proxy
    a2enmod proxy_http
    a2enmod proxy_wstunnel
    a2enmod rewrite
    a2enmod expires
    systemctl restart apache2
    Configure openqa version 2:
    /usr/share/openqa/script/configure-web-proxy
    -> However:
    26: cannot create /etc/apach2/vhosts.d/openqa.conf: Directory nonexistent
  5. This might have been required: /usr/share/openqa/script/initdb
  6. Configure the login procedure:
    Edit /etc/openqa/openqa.ini
    In the section [auth]: place ‘method = Fake‘
    Edit /etc/openqa/client.conf:
    [localhost]
    key = 1234567890ABCDEF
    secret = 1234567890ABCDEF
    Restart the openQA webui:
    systemctl restart openqa-webui
  7. Prepare salsa
    ssh-keygen -t ed25519 -C "VM Debian-openQA"
    gedit ~/.ssh/id_ed25519.pub
    -> paste in SSH Keys for Salsa
  8. Prepare the git repository:
    cd /var/lib/openqa/tests
    git clone git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git debian
  9. Initialise the default test settings:
    apt-get install python3-jsonschema
    cd /var/lib/openqa/tests/debian
    python3 fifloader.py templates.fif.json --update --load
  10. Install a local openQA-worker:
    apt-get install openqa-worker
  11. Download the netinst image and run it:
    cd /var/lib/openqa/share/factory/iso
    wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.2.0-amd64-netinst.iso
    openqa-cli api -X POST isos ISO=debian-11.2.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1120
  12. Issue: the tooltip with the guided tour did not disappear after being logged in:
    su geekotest
    psql openqa
    update users set feature_version=0;
    \q