Live images for Debian 12 (bookworm)

Deja-vu

This is a continuation of previous discussions. See Documenting the generation of the live images in Debian, Porting the standard image from live-wrapper to live-build and its follow-ups and Replacing live-wrapper for live images by live-build?. Some testing was discussed here: Some Debian Live testing.

Current state

The generation of live images for bookworm and sid have been turned off, see Status of weekly live builds. Sometimes questions are raised about the live images: cdimage weekly-live-builds not regenerating and #1006122

Desired state

It would be nice to have live images for Debian for Bookworm again.

However…

There are two major bottlenecks:

Human resources
The purpose/scope of live images

Human resources

As noted in the discussions, the live images cannot be maintained by a single person. It must be maintained by a group. So far nobody has really raised their hand and offered to push forward the support for live images.

I’ve dug into the archives, live-build has some painful history. However, it is a tool that is available at the moment and it is used for several other live-CDs as well.

At release-day the amount of testing live images is huge. It is a lot of manual work to verify the new images. Automating will reduce the work, but it cannot replace running on real hardware.

Two tools will help a lot. They have been added in the past years:

Jenkins. Ongoing. This generates the images twice and tests whether they are reproducible.
openQA. In progress. This will test many features of the live image. It simulates a computer from the moment it boots. It does not require special configuration/packages inside the image that is under test.

The purpose/scope of live images

If the live images are to be generated for Bookworm again, it needs to be discussed first what the expectations are for live images.

Some features:

Installer, with full off-line installation support:
- Debian-installer, to install from the boot menu
- Calamares installer, to install from the live environment
Internationalization/Localization support: choose your locale/language at boot
Accessibility support: speech and high-contrast modes
All major desktops are provided (currently Bullseye has 8)
Security support
Will work properly both on-line and off-line
Easy support for persistence
The installed software represents a workable system
The installed software should have as few as possible pre-configured options or special tweaks
The image must be generated reproducibly
Unofficial support for non-free firmware
Support for new blends like Debian Jr.
Maintenance should be relatively low
Which tool to use? live-build, live-wrapper, FAI, …
Hardware: at least amd64. Bullseye also has i386
Support several installation options:
- Explicit support for ISO chainloading (a single UBS boot stick could hold several ISO image files), similar to rufus, YUMI, Ventoy, …
- dd if=image.iso of=/dev/sdMyUSBStick
- mount the ISO image, copy all files to the USB stick
Support BIOS and UEFI boot (secure and non-secure)
Other features, …
Documentation of the the features mentioned above

Some desired features might contradict other features.

Personally I think that the following features are out-of-scope:

A small installer -> use FAI instead
A rescue image -> other solutions exist already
A small image with only ‘C’ and no i18n/l10n/a18y support (but I think this might be a good addition to the list of current images)

My involvement

So far I’ve been working on:

Documentation improvements of live-manual
Reproducibility fixes for live-build (Wiki)
Some functionality tests in openQA (not published yet, but running in a local, private openQA instance)
Extracting the rebuild script from Jenkins and including a freshly-built installer (using their daily-build script)
Gentle prodding

Personally I’m quite hesitant to volunteer, because I don’t know how much time I’ll have in the near future. However, I’m willing to help to get things started again.

Some background (kindly provided by pabs)

A list of derivatives that use live-build: https://wiki.debian.org/Derivatives/CensusFull

Some other tools that can generate live images: https://wiki.debian.org/SystemBuildTools

Note: This information is somewhat outdated.

Where are we now (2022-05-19)?

live-wrapper is not available in Bookworm any more (#1009282), the request for fixes or alternatives dates from 2018-09.
live-build can be used to generate reproducible live images.
FAI has been proposed to generate live images.
All major desktops have reproducible images (as tested by Jenkins)
openQA testing is active:
- Activate all GRUB/syslinux menu entries to verify that they perform correctly
Other openQA tests are planned/under review:
- Verify the correct functionality of some applications in the live environment
- Run the Debian installer
- Run the Calamares installer
- Verify the installed image (both from the boot menu and the checksum)
I’ve reviewed only amd64. (The snapshot mirror that I use carries only amd64)
I propose the following testing chain, which enforces reproducibility and functionality:

A) Use Jenkins to generate an ISO image based on the latest timestamp from the snapshot.debian.org/snapshot.notset.fr archive
B) Use Jenkins to generate the image again with the same timestamp
PASS/FAIL: If checksum is identical continue, otherwise abort
C) Run openQA tests
   - Test all boot menu entries
   - The live environment should start and the main applications should work
   - The Debian installer from the boot menu should function properly
   - The Calamares installer from the live environment should function properly
   - After the installer is finished, the newly installed Debian should boot and the main applications should work
   - etc...
PASS/FAIL: If all openQA tests pass (or have softfail) continue, otherwise abort
D) Publish the image on https://www.debian.org/CD/

This means that the live images will be published after some delay. (A snapshot needs to exist and the tests also take some amount of time). However, it can be fully automated, and many trivial issues can be caught early, during daily tests.

Git-related commands

Initialisation

git remote add upstream git@salsa.debian.org….
git config –global –add user.name “Roland Clobus”
git config –global –add user.email “rclobus@rclobus.nl”

Protected branch (as e.g. for openQA)

git checkout debian
git pull upstream debian
# do stuff
git commit
git push upstream debian

Git-lfs and forks

After forking openqa-tests-debian, the LFS part was not cloned. Stackoverflow found a solution:

https://stackoverflow.com/questions/55359067/what-is-the-workflow-for-git-lfs-with-forks

git clone <address of your fork>

Cloning the fork repo for the very first time – it does not have any LFS files copied yet, so you must clone it without the LFS files (otherwise cloning would fail)

export GIT_LFS_SKIP_SMUDGE=y
git clone <address of your fork>

Synchronizing upstream -> fork (copying LFS files to your fork) – initially and each time you git fetch the upstream

git lfs fetch --all <upstream>
git lfs push --all <fork>

Synchronizing fork -> upstream (only the current branch) – each time after your PR got merged

git lfs fetch <fork>
git lfs push <upstream>

Beautification of local history

git commit –amend –no-edit –author “Roland Clobus <rclobus@rclobus.nl>”

git commit –amend –no-edit –date “$(date)”

Cleanup

Branch cleanup: https://stackoverflow.com/questions/6127328/how-do-i-delete-all-git-branches-which-have-been-merged

git branch --merged

git branch -d merged_branch_name

Time travel

Go back in time to checkout a specific timestamp: https://stackoverflow.com/questions/6990484/how-to-checkout-in-git-by-date

git checkout `git rev-list -n 1 --first-parent --before="2022-06-01 00:00Z" upstream/debian`

openQA

Steps to install and configure openQA in my own VM:

Boot from a live image of GNOME unstable 2022-01-21T03:08Z
Install to a harddisk with Calamares
Install the Debian package openqa
echo "deb http://deb.debian.org/debian sid main" >> /etc/apt/sources.list apt-get update apt-get install openqa
Configure openqa
cd /etc/apache2/sites-enabled ln -s ../sites-available/openqa.conf.template openqa.conf # Replace #ServerName with 'ServerName localhost' a2enmod headers a2enmod proxy a2enmod proxy_http a2enmod proxy_wstunnel a2enmod rewrite a2enmod expires systemctl restart apache2
Configure openqa version 2:
/usr/share/openqa/script/configure-web-proxy
-> However:
26: cannot create /etc/apach2/vhosts.d/openqa.conf: Directory nonexistent
This might have been required: /usr/share/openqa/script/initdb
Configure the login procedure:
Edit /etc/openqa/openqa.ini
In the section [auth]: place ‘method = Fake‘
Edit /etc/openqa/client.conf:
[localhost]
key = 1234567890ABCDEF
secret = 1234567890ABCDEF
Restart the openQA webui:
systemctl restart openqa-webui
Prepare salsa
ssh-keygen -t ed25519 -C "VM Debian-openQA" gedit ~/.ssh/id_ed25519.pub
-> paste in SSH Keys for Salsa
Prepare the git repository:
cd /var/lib/openqa/tests git clone git@salsa.debian.org:rclobus-guest/openqa-tests-debian.git debian
Initialise the default test settings:
apt-get install python3-jsonschema cd /var/lib/openqa/tests/debian python3 fifloader.py templates.fif.json --update --load
Install a local openQA-worker:
apt-get install openqa-worker
Download the netinst image and run it:
cd /var/lib/openqa/share/factory/iso wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.2.0-amd64-netinst.iso openqa-cli api -X POST isos ISO=debian-11.2.0-amd64-netinst.iso DISTRI=debian VERSION=stable FLAVOR=netinst-iso ARCH=x86_64 BUILD=1120
Issue: the tooltip with the guided tour did not disappear after being logged in:
su geekotest psql openqa update users set feature_version=0; \q

GPG expiration

The expiration of my GPG keys happens on January, 1st.

In ‘Passwords and Keys’ it is easy to update the expiration date. In Preferences, add ‘hkp://keys.openpgp.net’.

gpg --list-keys rclobus@rclobus.nl

gpg --send-key 50554B33905D8FB1922446FE62C57C6AA61495BD
The default is hkp://hkps.pool.sks-keyservers.net

gpg --keyserver keys.openpgp.org --send-key 50554B33905D8FB1922446FE62C57C6AA61495BD

As instructed by openpgp.org:

gpg --export rclobus@rclobus.nl | curl -T - https://keys.openpgp.org

As instructed by https://anweshadas.in/how-to-use-yubikey-or-any-gpg-smartcard-in-thunderbird-78/

Preferences | Config Editor… | I Accept the risk

mail.openpgp.allow_external_gnupg -> true

Restart Thunderbird

Account Settings | End-to-end encryption | Use GnuPG | 50554B33905D8FB1922446FE62C57C6AA61495BD

New 2023:

Thunderbird requires the key to be uploaded here: https://keys.openpgp.org

Debian information: https://wiki.debian.org/Keysigning

Reproducible Debian-live images

This is a description of my work in progress in making Debian-live images reproducible.

Update 2021-02-10

This page will not be updated any more. The content has moved to the Debian Wiki.

Update 2020-12-06

I’ve finished MR 216 https://salsa.debian.org/live-team/live-build/-/merge_requests/216. Next up: MR 218, the reproducibility fixes.

Update 2020-11-18

Thanks for the responses so far. I’m preparing merge requests which can be applied to the repository piece-by-piece.

Detailed HOWTO

The basic script/commands:

# Running from ramdisk
# Running as root
export LIVE_BUILD=/home/roland/git/live-build
export SOURCE_DATE_EPOCH=918014706
cd /dev/shm
mkdir live
cd live
mount /dev/shm -odev,exec,remount

# A very basic configuration
lb clean
lb config --apt-http-proxy http://localhost:3142
lb build

# The comparison is made with diffoscope
mv live-image-amd64.hybrid.iso /somewhere_else/live-918014706-vXX.iso
diffoscope --html-dir /softwhere_else/html /somewhere_else/live-918014706-vXX-1.iso /somewhere_else/live-918014706-vXX.iso

SOURCE_DATE_EPOCH must be after 1980-01-01 for the EFI images
SOURCE_DATE_EPOCH is generated by date +%s --date 1999-02-03T04:05:06Z, which is a recognizable timestamp
Running from a ramdisk is an enormous speedup
Current focus: building the image, not the content or runnability of the image

Development machine setup

Debian/sid
apt-cacher-ng running on http://localhost:3142

Ideas

Turn off the Internet access while building an image, to ensure that a minimum amount of network traffic is generated
Use a snapshot instead of Debian Stable for the image, to make it easier to compare older images -> done
Inside the image: use auto-apt-proxy (or squid-deb-proxy-client)
Does the image need the file /var/cache/ldconfig/aux-cache, as it is changed between builds? It is generated by ldconfig. If the file is removed and ldconfig is executed, the contents can be regenerated and is identical even after some time.

List of files in scripts/build

With git commit:
binary_checksums
binary_disk -> needs review for values of LB_DEBIAN_INSTALLER
binary_grub_cfg
binary_grub-efi
binary_iso
binary_loadlin
binary_manifest
binary_rootfs
binary_syslinux
binary_win32-loader
chroot
efi-image
installer_debian-installer

No change required:
binary
binary_chroot

Under Review:

Local modifications:
binary_grub-pc -> Not in use for the default config
binary_memtest -> Not in use for the default config --memtest memtest86+
chroot_hacks -> Remove files from the chroot

Not investigated yet:
binary_grub-legacy
binary_hdd
binary_hooks
binary_includes
binary_linux-image
binary_netboot
binary_onie
binary_package-lists
binary_tar
binary_zsync
bootstrap
bootstrap_archives
bootstrap_cache
bootstrap_debootstrap
build
chroot_apt
chroot_archives
chroot_cache
chroot_debianchroot
chroot_devpts
chroot_dpkg
chroot_firmware
chroot_hooks
chroot_hostname
chroot_hosts
chroot_includes
chroot_install-packages
chroot_interactive
chroot_linux-image
chroot_package-lists
chroot_prep
chroot_preseed
chroot_proc
chroot_resolv
chroot_selinuxfs
chroot_sysfs
chroot_sysv-rc
chroot_tmpfs
config
grub-cpmodules
installer
installer_preseed
source
source_checksums
source_debian
source_disk
source_hdd
source_hooks
source_iso
source_live
source_tar

Earlier work:

Pass along SOURCE_DATE_EPOCH to the chroot: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=832998 -> will be extended with the merge request

A summary of the current state of the Debian ISO images: https://lists.reproducible-builds.org/pipermail/rb-general/2020-August/002018.html

This blog is mentioned in the September 2020 Reproducible Builds blog: https://reproducible-builds.org/reports/2020-09/

Merge requests

The foundation: https://salsa.debian.org/live-team/live-build/-/merge_requests/209

Merge request 209 is superseded by https://salsa.debian.org/live-team/live-build/-/merge_requests/210 which also contains the changes from 209.
I just noticed the comment in 209, I’ll take a look how I can avoid a lot of ‘Touch’ commands.

Using lb build I am now able to have all files and folders on the image with the timestamp.

The files that are different (their content) are: .disk/archive_trace, var/cache/apt/pkgcache.bin, var/cache/apt/srcpkgcache.bin, var/cache/ldconfig/aux-cache, sha256sum.txt

Issues found

Reported

binary_syslinux: The sed -e ‘d’ commands with ‘#’ will not work, a slash is needed, fix for 7ffd2288d944840937f556bd56703ba381f4edcc (2015-01-15) and 578dbee516a370935e1b2e49205c524370e1f8d0 (2015-01-29) -> https://salsa.debian.org/live-team/live-build/-/merge_requests/211

Not reported yet

Package apt-cacher-ng: the file /etc/apt-cacher-ng/acnf.conf needs the following line to allow the file .disk/archive_trace to be filled properly:

VfilePatternEx: /project/trace/ftp-master\.debian\.org$

When preserving the timestamps for loadlin the question arises: are these install.bat files correct? The path seems incorrect to me. When running loadlin.exe on Windows10 2004, 64-bit, the 16-bit executable was rejected by Windows.

When running the installer in expert mode in Dutch on a GB-Windows 10, in text mode, I get the graphical installer, not the text installer.

2020-09-30 Using the snapshot repository

lb config --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --parent-mirror-binary http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --security false --updates false

This needs the modified acnf.conf file (as mentioned above). The URLs need the localhost:3142 part, because debootstrap does not use the apt-http-proxy value. Security and updates have been disabled to have the minimal amount of external influences for this run.

Open point: Inside the image, the direct link to snapshot.debian.org should be used

2020-09-30 Using –interactive

When using --interactive, a shell is created before the binary image is constructed

rm /var/cache/apt/*.bin
rm /var/cache/ldconfig/aux-cache

Results:

live/filesystem.packages does not have the correct timestamp any more.
root/.bash_history gets created in the squashfs image
Several folders get a new ‘size’
/var/cache/apt/pkgcache.bin is still in the squashfs image (probably due to the installation of mkisofs)

Conclusion:

--interactive helps, but is not the final solution.

2020-09-30 Using chroot_hacks

chroot_hacks is the last step in the chroot structure. An additional step between binary_checksums and binary_iso might be needed to delete the files from /var/cache.

2020-10-05 Basic ‘lb build’ with snapshot is building reproducible

Success. The minimal lb build (with a local cache) results in a reproducible image.
The image contains the regular snapshot repository URL.

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20200919T085932Z --security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/20200919T085932Z --parent-mirror-binary-security http:///snapshot.debian.org/archive/debian-security/20200919T085932Z --updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster

Info from 2020-10-06

The next step: finding the date of the 10.6 live image

There are many timestamps in the live image (created by live-wrapper).

cat .disk/info
Debian Live 10.6 official: 2020-09-26T10:36

TZ="UTC" unsquashfs -lls filesystem.squashfs | sort -k 4,5
-rw-r--r-- root/root 228576 2020-09-26 09:14 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_contrib_binary-amd64_Packages
-rw-r--r-- root/root 29374021 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_i18n_Translation-en
-rw-r--r-- root/root 40370452 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_source_Sources
-rw-r--r-- root/root 44651743 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_binary-amd64_Packages
-rw-r--r-- root/root 121480 2020-09-26 10:06 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_InRelease

unsquashfs -mkfs-time filesystem.squashfs
1601116542

root@silent:/dev/shm/x/squashfs-root/var/lib/apt/lists# grep "Date: " *
local-mirror.cdbuilder.debian.org_debian_dists_buster_InRelease:Date: Sat, 26 Sep 2020 09:54:48 UTC

It turns out that the local-mirror was slightly faster than the snapshot mirror. Kernel 4.19.0-11 got added in snapshot.debian.org on 20200926T104248Z

date +%s --date 2020-09-26T10:42:48Z
1601116968

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200926T104248Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20200926T104248Z --security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/20200926T104248Z --parent-mirror-binary-security http://snapshot.debian.org/archive/debian-security/20200926T104248Z --updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster
echo "live-task-standard" > config/package-lists/desktop.list.chroot

Diffoscope is having a hard time comparing the live-wrapper official live image and my local live image, which is generated by live-build:
diffoscope debian-live-10.6.0-amd64-standard.iso live-st02.iso --html-dir htmlout --max-diff-block-lines-saved 1000 --max-diff-input-lines 1000 --max-diff-block-lines 300 --max-page-size 1000000

2020-10-08

The basic image is reproducible, contains some hacks. Next step: working on --debian-installer live

lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200926T104248Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20200926T104248Z --security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/20200926T104248Z --parent-mirror-binary-security http://snapshot.debian.org/archive/debian-security/20200926T104248Z --updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster --debian-installer live

git push: https://salsa.debian.org/rclobus-guest/live-build/-/merge_requests/new?merge_request%5Bsource_branch%5D=rclobus%2Freproducible-foundation

Some major differences between the official Debian Live Standard image and the lb build image:

udeb location: pool vs pool-udeb
installer location: d-i vs installer
/etc/machine-id in official image
/etc/default/console-setup: iso-8859-15 vs utf-8
/etc/apt/sources.list: only deb.debian.org vs deb-src + updates + security
isolinux/menu.cfg: many languages vs none
efi vs EFI
_DISK vs .disk
No Joliet vs Joliet

2020-10-31

/etc/shadow: The gid are ‘days since 1970-01-01’. This should be adjusted.

installer_debian-installer: I’ve found the reason why the files /dev/lock and /dev/lock-frontend got added to the squashfs image. A MR is pending.

I’m getting close to publishing the result to the mailing lists. Cleanup of some local hacks is required.

2020-11-11

The mail is finally sent. The standard image looks as close a possible to the current official live image.

Non-reproducible: /var/lib/systemd/catalog/database.

Testing with Windows 10 1909 (in a VM): The image is called ‘Debian buster 20‘ -> 16 characters. /firmware is an empty directory. /debian does not exist, /dists only contains buster. All these issues are explained by the lack of support for RockRidge on Windows.

The image contains deb.debian.org for regular updates, so no references to snapshot.debian.org are inside the image.

The installer doesn’t work: win32-loader.ini refers to the folder install, which got renamed to d-i to match the live-wrapper image.

2020-11-18

Many thanks to the people who have responded so far. As per request, I’ve prepared a few merge requests, which each contain an isolated feature/bugfix.

Merge Request 1: https://salsa.debian.org/live-team/live-build/-/merge_requests/215
Bugfix for LB_DERIVATIVE

Merge Request 2: https://salsa.debian.org/live-team/live-build/-/merge_requests/216
Bugfix for the files lock and lock-frontend

Merge Request 3: https://salsa.debian.org/live-team/live-build/-/merge_requests/217
Bugfix: Live installer can run without LB_CACHE_PACKAGES

Merge Request 4: https://salsa.debian.org/live-team/live-build/-/merge_requests/218
Reproducible framework

2020-12-06

MR215 and 217 have been added to upstream/master. MR216 was finished 2020-12-03 and is pending a merge. Next: MR218

MR218 (reproducibility)

The following scenarios need to be investigated, each with 2 builds:

Set EPOCH_SOURCE_DATE, run lb config, run lb build
Run lb config, set EPOCH_SOURCE_DATE, run lb build
Run lb config, run lb build

For each scenario, the effect of the timestamp must be explainable.

This is the scenario for a reproducible build
The configuration file and the ISO file use the same timestamp
The LB_ISO_VOLUME variable will be fixed to the timestamp of the moment the configuration is created -> not good. The timestamp of lb build should have been used. Fixed in c23105b1e3c8d11fc8fc92bd84f0156d280fc54b
Next issue: run lb config twice -> date: invalid date ‘@’. Instead of placing a timestamp in the config file, I would use a similar strategy as with @LB_VERSION@.

New merge requests:

https://salsa.debian.org/live-team/live-build/-/merge_requests/220: Bugfix: use minutes instead of month in the time of the modification date field

Myst: Masterpiece Edition on Windows 7

Steps to install and run Myst: Masterpiece Edition on Windows 7

Create an image with WinCDEmu
Install Myst
Install QT4 from the CD/DVD
It did not work correctly (initial movies played OK, black screen with howling wind, final credits work)
Uninstalled QT7 (the most recent version)
I have tried all kinds of compatibility options, at the end, I’ve used none
Uninstall QT7
Install QT4 from the CD/DVD
It works, but the Cyan movie sounds bad
Start QuickTime, Edit | Preferences | Connection Speed
In the dialog, drop down to Sound Out
Device for Playback: waveOut: Windows’ preferred device, at 48kHz, 16bit stereo
Now the Cyan movies sounds OK
The game can be played properly now

Cleanup of cruft in Debian

Cleanup of unused, left-over old configuration files

aptitude search '~c'

In aptitude: filter ~c

Cleanup of transitional packages

deborphan

Find manually installed packages

Filter: !~M~i

Find non-maintained files

cruft

Find modified configuration files

https://serverfault.com/questions/90400/how-to-check-for-modified-config-files-on-a-debian-system#90401

dpkg-query -W -f='${Conffiles}\n' '*' | awk 'OFS=" "{print $2,$1}' | LANG=C md5sum -c 2>/dev/null | awk -F': ' '$2 !~ /OK$/{print $1}' | sort | less

debsums -ce

Android backup

Investigation of an Android backup file:

Create a backup file:
adb backup -f `date +%Y%m%d`_backup.ab -apt -obb -all -system

Extract its contents:
dd if=~/android/20161101_backup.ab bs=1 skip=24 | zlib-flate -uncompress > x.tar

Find resource hogs (as root):
find -exec file '{}' ';' | grep "ISO Media" | awk '{ split($0, a, ":"); print a[1] }'

Android apk files are located in /data/app, they can be (temporarily) removed when there is not sufficient space for installing new apps.

Upgrading Debian on the Sheevaplug

Starting point: Sheevaplug running Debian 7.9 on an ubifs filesystem.

See the installation notes for Jessie.

Step 1: Create a backup.
I want to create the backup when the partitions to backup are not mounted.
Step 1.1: Prepare a boot environment
Get uImage and uInitrd as described on the Sheevaplug installation page:
wget http://ftp.debian.org/debian/dists/stable/main/installer-armel/current/images/kirkwood/netboot/marvell/sheevaplug/uImage wget http://ftp.debian.org/debian/dists/stable/main/installer-armel/current/images/kirkwood/netboot/marvell/sheevaplug/uInitrd md5sum u* f05abb9a8cc619fff9bf2d585dc4b231 uImage da2240f621404712a908dbc9593c3bda uInitrd

Step 1.2: Reboot into u-boot
Within u-boot:
usb start fatload usb 0:1 0x00800000 /uImage fatload usb 0:1 0x01100000 /uInitrd setenv bootargs console=ttyS0,115200n8 base-installer/initramfs-tools/driver-policy=most bootm 0x00800000 0x01100000
Run the installer, for locale “C”. Use the manual network configuration. Pick a suitable Debian mirror. Give usernames and passwords. When the partition disks appears, go to ‘Go Back’
** mount -t ubifs rootfs /x does not work.
** Mount the usb to /x: mkdir /x;mount -t /dev/sda1 /x
** dd if=/dev/mtd2ro of=/x/mtd2ro
Copy the file to the PC and verify whether it can be read (based on UBIFS documentation:
modprobe nandsim first_id_byte=0x20 second_id_byte=0xaa third_id_byte=0x00 fourth_id_byte=0x15 overridesize=12 modprobe ubi mtd=0 ubidetach /dev/ubi_ctrl -m 0 ubiformat /dev/mtd0 -f mtd2ro ubiattach /dev/ubi_ctrl -m 0 mkdir x mount -t ubifs ubi0 x

During @apt-get upgrade@, the disk was full.
mount –bind /y/usr/usr /usr
mount –bind /y/lib/lib /lib
Then remove /lib with @rm -fr /lib@
ARGS! All command depend on /lib… Restore it with:
/y/lib/lib/ld-linux.so.3 –library-path /y/lib/lib:/y/lib/lib/arm-linux-gnueabi /bin/mkdir /lib
/y/lib/lib/ld-linux.so.3 –library-path /y/lib/lib:/y/lib/lib/arm-linux-gnueabi /bin/cp /y/lib/lib/ld-linux.so.3 /lib
/y/lib/lib/ld-linux.so.3 –library-path /y/lib/lib:/y/lib/lib/arm-linux-gnueabi /bin/cp /y/lib/lib/arm-linux-gnueabi /lib/ -ax

Accessing ubifs from within the Sheevaplug

mkdir /xx
cd /xx
wget http://ftp.debian.org/debian/pool/main/l/linux/linux-image-3.2.0-4-ixp4xx_3.2.78-1_armel.deb
ar x linux-image-3.2.0-4-ixp4xx_3.2.78-1_armel.deb
xzcat data.tar.xz | tar x

wget http://ftp.debian.org/debian/pool/main/l/linux/linux-image-4.13.0-1-marvell_4.13.13-1_armel.deb
ar x linux-image-4.13.0-1-marvell_4.13.13-1_armel.deb
xzcat data.tar.xz | tar x
depmod -b /xx
modprobe -d /xx ubi

wget http://ftp.de.debian.org/debian/pool/main/m/mtd-utils/mtd-utils_1.5.1-1_armel.deb
mkdir mtd-utils
cd mtd-utils/
ar x ../mtd-utils_1.5.1-1_armel.deb
xzcat data.tar.xz | tar x

modprobe -d /xx ubifs

-> fails: UBIFS error (pid 3918): cannot initialize compressor zlib, error -2

wget http://security.debian.org/debian-security/pool/updates/main/l/linux/linux-image-3.16.0-4-ixp4xx_3.16.7-ckt20-1+deb8u4_armel.deb
ar x linux-image-3.16.0-4-ixp4xx_3.16.7-ckt20-1+deb8u4_armel.deb
xzcat data.tar.xz > data.tar
mkdir data
cd data

8 tar xf ../data.tar ./lib/modules/3.16.0-4-ixp4xx/kernel/fs/ubifs/ubifs.ko
10 tar xf ../data.tar ./lib/modules/3.16.0-4-ixp4xx/kernel/drivers/mtd/ubi/ubi.ko
12 modprobe ./lib/modules/3.16.0-4-ixp4xx/kernel/fs/ubifs/ubifs.ko
13 modprobe ./lib/modules/3.16.0-4-ixp4xx/kernel/drivers/mtd/ubi/ubi.ko
—
wget http://ftp.de.debian.org/debian/pool/main/m/mtd-utils/mtd-utils_1.5.1-1_armel.deb
mkdir mtd-utils
cd mtd-utils/
ar x ../mtd-utils_1.5.1-1_armel.deb
xzcat data.tar.xz > data.tar
cd usr/sbin
./ubiattach -m 2
mount -t ubifs ubi0 /y
—
Restoring a backup:
./ubidetach -m 2
./ubiformat /dev/mtd2 -f /media/mtd2ro
./ubiattach -m 2
dmesg | tail -20
mount -t ubifs ubi0 /y
ls /y

Sheevaplug and power loss

After a power loss, the Sheevaplug did not start up any more.
After some fiddling, I got it to work (ubifs needed to be mounted)
Then…
I attempted to upgrade U-Boot.
I wasn’t prepared to do a lot of migrating and attempted to downgrade and flashed U-Boot with garbage.

With Debian 6 I was able to run openocd and reflash U-Boot, as described here:

This will reflash the plug with a new U-Boot called uboot.bin (this is the .bin file and not ELF and the name must be uboot.bin).

> cd
> openocd -f /usr/local/lib/openocd/board/sheevaplug.cfg -c init -c sheevaplug_reflash_uboot -c exit

Wait 1.5 minutes.

It needs to be an older version of Debian, because the newest version rejects the JTAG part of the USB connection.