This is a description of my work in progress in making Debian-live images reproducible.
This page will not be updated any more. The content has moved to the Debian Wiki.
I’ve finished MR 216 https://salsa.debian.org/live-team/live-build/-/merge_requests/216. Next up: MR 218, the reproducibility fixes.
Thanks for the responses so far. I’m preparing merge requests which can be applied to the repository piece-by-piece.
The basic script/commands:
# Running from ramdisk # Running as root export LIVE_BUILD=/home/roland/git/live-build export SOURCE_DATE_EPOCH=918014706 cd /dev/shm mkdir live cd live mount /dev/shm -odev,exec,remount # A very basic configuration lb clean lb config --apt-http-proxy http://localhost:3142 lb build # The comparison is made with diffoscope mv live-image-amd64.hybrid.iso /somewhere_else/live-918014706-vXX.iso diffoscope --html-dir /softwhere_else/html /somewhere_else/live-918014706-vXX-1.iso /somewhere_else/live-918014706-vXX.iso
- SOURCE_DATE_EPOCH must be after 1980-01-01 for the EFI images
- SOURCE_DATE_EPOCH is generated by
date +%s --date 1999-02-03T04:05:06Z, which is a recognizable timestamp
- Running from a ramdisk is an enormous speedup
- Current focus: building the image, not the content or runnability of the image
Development machine setup
- apt-cacher-ng running on http://localhost:3142
- Turn off the Internet access while building an image, to ensure that a minimum amount of network traffic is generated
- Use a snapshot instead of Debian Stable for the image, to make it easier to compare older images -> done
- Inside the image: use
- Does the image need the file
/var/cache/ldconfig/aux-cache, as it is changed between builds? It is generated by
ldconfig. If the file is removed and
ldconfigis executed, the contents can be regenerated and is identical even after some time.
List of files in scripts/build
With git commit: binary_checksums binary_disk -> needs review for values of LB_DEBIAN_INSTALLER binary_grub_cfg binary_grub-efi binary_iso binary_loadlin binary_manifest binary_rootfs binary_syslinux binary_win32-loader chroot efi-image installer_debian-installer No change required: binary binary_chroot Under Review: Local modifications: binary_grub-pc -> Not in use for the default config binary_memtest -> Not in use for the default config --memtest memtest86+ chroot_hacks -> Remove files from the chroot Not investigated yet: binary_grub-legacy binary_hdd binary_hooks binary_includes binary_linux-image binary_netboot binary_onie binary_package-lists binary_tar binary_zsync bootstrap bootstrap_archives bootstrap_cache bootstrap_debootstrap build chroot_apt chroot_archives chroot_cache chroot_debianchroot chroot_devpts chroot_dpkg chroot_firmware chroot_hooks chroot_hostname chroot_hosts chroot_includes chroot_install-packages chroot_interactive chroot_linux-image chroot_package-lists chroot_prep chroot_preseed chroot_proc chroot_resolv chroot_selinuxfs chroot_sysfs chroot_sysv-rc chroot_tmpfs config grub-cpmodules installer installer_preseed source source_checksums source_debian source_disk source_hdd source_hooks source_iso source_live source_tar
SOURCE_DATE_EPOCH to the chroot: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=832998 -> will be extended with the merge request
A summary of the current state of the Debian ISO images: https://lists.reproducible-builds.org/pipermail/rb-general/2020-August/002018.html
This blog is mentioned in the September 2020 Reproducible Builds blog: https://reproducible-builds.org/reports/2020-09/
Merge request 209 is superseded by https://salsa.debian.org/live-team/live-build/-/merge_requests/210 which also contains the changes from 209.
I just noticed the comment in 209, I’ll take a look how I can avoid a lot of ‘Touch’ commands.
lb build I am now able to have all files and folders on the image with the timestamp.
The files that are different (their content) are:
.disk/archive_trace, var/cache/apt/pkgcache.bin, var/cache/apt/srcpkgcache.bin, var/cache/ldconfig/aux-cache, sha256sum.txt
binary_syslinux: The sed -e ‘d’ commands with ‘#’ will not work, a slash is needed, fix for 7ffd2288d944840937f556bd56703ba381f4edcc (2015-01-15) and 578dbee516a370935e1b2e49205c524370e1f8d0 (2015-01-29) -> https://salsa.debian.org/live-team/live-build/-/merge_requests/211
Not reported yet
apt-cacher-ng: the file
/etc/apt-cacher-ng/acnf.conf needs the following line to allow the file
.disk/archive_trace to be filled properly:
When preserving the timestamps for
loadlin the question arises: are these
install.bat files correct? The path seems incorrect to me. When running
loadlin.exe on Windows10 2004, 64-bit, the 16-bit executable was rejected by Windows.
When running the installer in expert mode in Dutch on a GB-Windows 10, in text mode, I get the graphical installer, not the text installer.
2020-09-30 Using the snapshot repository
lb config --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --parent-mirror-binary http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --security false --updates false
This needs the modified
acnf.conf file (as mentioned above). The URLs need the
localhost:3142 part, because
debootstrap does not use the
apt-http-proxy value. Security and updates have been disabled to have the minimal amount of external influences for this run.
Open point: Inside the image, the direct link to
snapshot.debian.org should be used
2020-09-30 Using –interactive
--interactive, a shell is created before the binary image is constructed
rm /var/cache/apt/*.bin rm /var/cache/ldconfig/aux-cache
live/filesystem.packagesdoes not have the correct timestamp any more.
root/.bash_historygets created in the squashfs image
- Several folders get a new ‘size’
/var/cache/apt/pkgcache.binis still in the squashfs image (probably due to the installation of
--interactive helps, but is not the final solution.
2020-09-30 Using chroot_hacks
chroot_hacks is the last step in the chroot structure. An additional step between
binary_iso might be needed to delete the files from
2020-10-05 Basic ‘lb build’ with snapshot is building reproducible
Success. The minimal
lb build (with a local cache) results in a reproducible image.
The image contains the regular snapshot repository URL.
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200919T085932Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20200919T085932Z --security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/20200919T085932Z --parent-mirror-binary-security http:///snapshot.debian.org/archive/debian-security/20200919T085932Z --updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster
Info from 2020-10-06
The next step: finding the date of the 10.6 live image
There are many timestamps in the live image (created by live-wrapper).
Debian Live 10.6 official: 2020-09-26T10:36
TZ="UTC" unsquashfs -lls filesystem.squashfs | sort -k 4,5
-rw-r--r-- root/root 228576 2020-09-26 09:14 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_contrib_binary-amd64_Packages
-rw-r--r-- root/root 29374021 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_i18n_Translation-en
-rw-r--r-- root/root 40370452 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_source_Sources
-rw-r--r-- root/root 44651743 2020-09-26 09:15 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_main_binary-amd64_Packages
-rw-r--r-- root/root 121480 2020-09-26 10:06 squashfs-root/var/lib/apt/lists/local-mirror.cdbuilder.debian.org_debian_dists_buster_InRelease
unsquashfs -mkfs-time filesystem.squashfs
root@silent:/dev/shm/x/squashfs-root/var/lib/apt/lists# grep "Date: " *
local-mirror.cdbuilder.debian.org_debian_dists_buster_InRelease:Date: Sat, 26 Sep 2020 09:54:48 UTC
It turns out that the local-mirror was slightly faster than the snapshot mirror. Kernel 4.19.0-11 got added in
date +%s --date 2020-09-26T10:42:48Z 1601116968
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/
20200926T104248Z--security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/
20200926T104248Z--updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster echo "live-task-standard" > config/package-lists/desktop.list.chroot
Diffoscope is having a hard time comparing the live-wrapper official live image and my local live image, which is generated by live-build:
diffoscope debian-live-10.6.0-amd64-standard.iso live-st02.iso --html-dir htmlout --max-diff-block-lines-saved 1000 --max-diff-input-lines 1000 --max-diff-block-lines 300 --max-page-size 1000000
The basic image is reproducible, contains some hacks. Next step: working on
lb config --apt-http-proxy http://localhost:3142 --parent-mirror-bootstrap http://localhost:3142/snapshot.debian.org/archive/debian/20200926T104248Z --parent-mirror-binary http://snapshot.debian.org/archive/debian/20200926T104248Z --security true --parent-mirror-chroot-security http://localhost:3142/snapshot.debian.org/archive/debian-security/20200926T104248Z --parent-mirror-binary-security http://snapshot.debian.org/archive/debian-security/20200926T104248Z --updates true --apt-options "-y -o Acquire::Check-Valid-Until=false" --distribution buster --debian-installer live
Some major differences between the official Debian Live Standard image and the lb build image:
- udeb location: pool vs pool-udeb
- installer location: d-i vs installer
- /etc/machine-id in official image
- /etc/default/console-setup: iso-8859-15 vs utf-8
- /etc/apt/sources.list: only deb.debian.org vs deb-src + updates + security
- isolinux/menu.cfg: many languages vs none
- efi vs EFI
- _DISK vs .disk
- No Joliet vs Joliet
/etc/shadow: The gid are ‘days since 1970-01-01’. This should be adjusted.
installer_debian-installer: I’ve found the reason why the files
/dev/lock-frontend got added to the squashfs image. A MR is pending.
I’m getting close to publishing the result to the mailing lists. Cleanup of some local hacks is required.
The mail is finally sent. The standard image looks as close a possible to the current official live image.
Testing with Windows 10 1909 (in a VM): The image is called ‘
Debian buster 20‘ -> 16 characters.
/firmware is an empty directory.
/debian does not exist,
/dists only contains
buster. All these issues are explained by the lack of support for RockRidge on Windows.
The image contains
deb.debian.org for regular updates, so no references to
snapshot.debian.org are inside the image.
The installer doesn’t work:
win32-loader.ini refers to the folder
install, which got renamed to
d-i to match the live-wrapper image.
Many thanks to the people who have responded so far. As per request, I’ve prepared a few merge requests, which each contain an isolated feature/bugfix.
Merge Request 1: https://salsa.debian.org/live-team/live-build/-/merge_requests/215
Bugfix for LB_DERIVATIVE
Merge Request 2: https://salsa.debian.org/live-team/live-build/-/merge_requests/216
Bugfix for the files
Merge Request 3: https://salsa.debian.org/live-team/live-build/-/merge_requests/217
Bugfix: Live installer can run without LB_CACHE_PACKAGES
Merge Request 4: https://salsa.debian.org/live-team/live-build/-/merge_requests/218
MR215 and 217 have been added to upstream/master. MR216 was finished 2020-12-03 and is pending a merge. Next: MR218
The following scenarios need to be investigated, each with 2 builds:
lb config, run
lb config, set
lb config, run
For each scenario, the effect of the timestamp must be explainable.
- This is the scenario for a reproducible build
The configuration file and the ISO file use the same timestamp
LB_ISO_VOLUMEvariable will be fixed to the timestamp of the moment the configuration is created -> not good. The timestamp of
lb buildshould have been used. Fixed in c23105b1e3c8d11fc8fc92bd84f0156d280fc54b
Next issue: run
lb configtwice ->
date: invalid date ‘@’. Instead of placing a timestamp in the config file, I would use a similar strategy as with
New merge requests:
https://salsa.debian.org/live-team/live-build/-/merge_requests/220: Bugfix: use minutes instead of month in the time of the modification date field