UKI Cleanup by Johan-Liebert1 · Pull Request #2200 · bootc-dev/bootc

Johan-Liebert1 · 2026-05-15T07:22:34Z

ukify: Allow passing custom kernel, initramfs

While building a sealed UKI image we'd want to remove the original
kernel + initramfs from the final image and have only the final UKI
present. This was not possible before as bootc container ukify
expected kernel + initramfs to be present in usr/lib/modules of
container root

Fixes: #2185

dockerfile/uki: Rework to remove kernel + initrd

Now that we can pass kernel and initrd paths to bootc ukify, rework
our UKI Dockerfile to remove kernel + initrd from the final layer
and only keep the UKI

This still will not remove the kernel + initrd from the tarball but
have whiteout instead

See #2027 (comment)

test/integration: Test vmlinuz non-existence with UKI

vmlinuz and intrd should not be present in UKI images; add test for the
same

gemini-code-assist

Code Review

This pull request refactors the UKI (Unified Kernel Image) build process to support passing explicit kernel and initramfs paths via CLI arguments, reducing reliance on auto-discovery within the rootfs. Key changes include updating the seal-uki and finalize-uki scripts to use named arguments, modifying Dockerfile stages to extract and clean up kernel components, and extending the Rust library and CLI to handle the new parameters. Review feedback identified a potential path resolution bug in the Rust file existence checks, a filename mismatch in the upgrade test Dockerfile, and suggested improvements for error handling and validation in the seal-uki script.

travier · 2026-05-20T07:48:38Z

I think we should have the kernel and initrd as required arguments and thus only support the case where they are not in the rootfs anymore as I don't see a use case where someone would want a sealed image with both a UKI and split out kernel and initrd.

Edit: That would break the current bootc container ukify signature but I don't think we've declared it stable yet?

cgwalters · 2026-05-20T21:55:58Z

Hmm. I kind of lean towards not breaking it at least right away, it seems really easy to continue to support what we have now too. We could mark it deprecated though.

travier · 2026-05-21T14:21:29Z

Failures are transient & volatile related. Maybe from #2201?

Johan-Liebert1 · 2026-05-28T10:30:54Z

Hmm. I kind of lean towards not breaking it at least right away, it seems really easy to continue to support what we have now too. We could mark it deprecated though.

Don't know if clap has a way to do this, if it does I couldn't find it. We just print our custom warning if --kernel and --initramfs are not passed in

Johan-Liebert1 · 2026-06-30T10:44:28Z

Flake

content: error: Installing to disk: Creating ostree deployment: Pulling image into bootc containers-storage: Pulling image via podman API: quay.io/centos-bootc/centos-bootc:stream10: Pull error from podman: unable to copy from source docker://quay.io/centos-bootc/centos-bootc:stream10: copying system image from manifest list: writing blob: storing blob to file "/var/tmp/container_images_storage3750836339/14": happened during read: unexpected EOF (while reconnecting: Get "https://cdn01.quay.io/quayio-production-s3/sha256/be/be0ff3fbfc0a914d2c6291a6436f322ec64a7328f27acfd122e4ecace059d0b7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20260630%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260630T082011Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=8eb711c1321e233fc0c3493dee60f3e4e20d4edbc45b23d8ec4fd33823ca0136&region=us-east-1&namespace=centos-bootc&repo_name=centos-bootc&akamai_signature=exp=1782808511~hmac=266629fe9ab7c037820ab797749318a3806cfb3558e899ac9e7d714d729549e4": EOF)

reran

Now that we can pass kernel and initrd paths to `bootc ukify`, rework our UKI Dockerfile to remove kernel + initrd from the final layer and only keep the UKI This still will not *remove* the kernel + initrd from the tarball but have whiteout instead See bootc-dev#2027 (comment) Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

vmlinuz and intrd should not be present in UKI images; add test for the same Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

So we can just use bootc to extract the `.linux` and `.initrd` sections from the UKI and not have to use objcopy Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

This command is equivalent to `mv /target-root/usr/lib/modules/$kver/{vmlinuz,initramfs.img} /out/$kver` We could just use `mv`, but having an actual bootc cmd is cleaner Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Since we do not want kernel + initrd in the final UKI dockerfile, we now build the initrd inside the `target-rootfs` generated by `bootc-base-imagectl`. Instead of rebuilding the initrd ourselves, now we add `/var/usr` directory to the target-rootfs with `--add-dir` cli option to bootc-base-imagectl. This directory contains our rpms and other configs required for building the initramfs After that's done we split the rootfs and vmlinuz + initrd into /target-rootfs and /kernel/$kver respectively Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Now since we need to build our initramfs before the `fetch` build stage, we need packages built first as we need `bootc` and `bootc-initramfs-setup` binaries Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

We do not want to use `--nodeps` for installing our built rpms in the target-rootfs. So, we create a new dnf repo in `/tmp/localrepo`, copy our rpms in there and set it's priority as highest. We pass `--install bootc` to bootc-base-imagectl which will install from our newly created repo ensuring we have the latest binaries in the final image Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

The three-devices-partial-ESP scenario fails with "No device found for /dev/loop1p1" because the dual-ESP test's cleanup leaves stale kernel partition nodes and LVM device registrations. The prior cleanup called `pvremove -f $loop` on the loop device itself (e.g. /dev/loop0) instead of on the partitions where PVs actually live (e.g. /dev/loop0p2), making it a no-op. After `losetup -d`, the kernel retained busy partition nodes. When the next test reused the loop device with a different partition layout, `partx -u` tried to atomically reconcile old and new partitions — but could not remove the busy stale partition bootc-dev#2, so it aborted entirely without adding the new partition #1. Fix both cleanup and setup: Cleanup: - pvremove on actual partition devices (p1/p2/p3), not the loop device - wipefs -a on each partition to clear signatures holding references - udevadm settle to let udev finish processing - partx -d to remove kernel partition entries before losetup -d - Remove /etc/lvm/devices/system.devices to clear stale device IDs Setup (setup_disk_with_partitions, setup_disk_with_root): - Replace `partx -u` with `partx -d` (ignore errors) then `partx -a`, so a stuck stale partition does not block adding new ones - Replace `sleep 1sec` with `udevadm settle` for reliable synchronization Co-Authored-By: Claude Opus 4.6 Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Johan-Liebert1 · 2026-07-03T08:50:45Z

[centos-9,ostree] failed with

Error: 
   0: Failed to find or create base disk
   1: Failed to install bootc to base disk: "/home/runner/.local/share/libvirt/images/bootc-base-fda3be867822565c.djfT7X.tmp.qcow2"
   2: Timeout waiting for readiness after 240s (202 attempts)

and [centos-10,ostree] failed with

----------------------------------------
Plan: /tmt/plans/integration/plan-42-loader-entries-source
----------------------------------------
Run ID: bootc-tmt-8ki58tkb-tmt-plans-integration-plan-42-loader-entries-source

/var/tmp/tmt/bootc-tmt-8ki58tkb-tmt-plans-integration-plan-42-loader-entries-source

/tmt/plans/integration/plan-42-loader-entries-source
    report
        how: display
        order: 50
        prepare
        execute
            00:00:00 fail /tmt/tests/tests/test-42-loader-entries-source
                Note: check 'avc' is informational
                logs (with content):
                    output.txt (/var/tmp/tmt/bootc-tmt-8ki58tkb-tmt-plans-integration-plan-42-loader-entries-source/tmt/plans/integration/plan-42-loader-entries-source/execute/data/guest/default-0/tmt/tests/tests/test-42-loader-entries-source-1/output.txt)
                        content: Warning: Permanently added '[localhost]:2996' (ED25519) to the list of known hosts.
                        content: TAP version 14
                        content: loader-entries set-options-for-source
                        content: ok: validation and initial staging
                        content: Connection to localhost closed.
                        content: Warning: Permanently added '[localhost]:2996' (ED25519) to the list of known hosts.
                        content: ok: multiple sources staged before reboot both survived
                        content: ok: system kargs preserved
                        content: ok: kargs and source keys survived reboot
                        content: Connection to localhost closed.
                        content: Warning: Permanently added '[localhost]:2996' (ED25519) to the list of known hosts.
                        content: ok: source replacement persisted, system kargs preserved
                        content: Connection to localhost closed.
                        content: Warning: Permanently added '[localhost]:2996' (ED25519) to the list of known hosts.
                        content: ok: multiple sources coexist
                        content: ok: --options '' clears kargs
                        content: Connection to localhost closed.
                        content: Warning: Permanently added '[localhost]:2996' (ED25519) to the list of known hosts.
                        content: Error:   x dracut karg should be gone
                        content:      ,-[/var/tmp/tmt/bootc-tmt-8ki58tkb-tmt-plans-integration-plan-42-loader-entries-source/tmt/plans/integration/plan-42-loader-entries-source/discover/default-0/tests/tmt/tests/booted/test-loader-entries-source.nu:193:12]
                        content:  192 |     let cmdline = parse_cmdline
                        content:  193 |     assert ("rd.driver.pre=vfio-pci" not-in $cmdline) "dracut karg should be gone"
                        content:      :            ^^^^^^^^^^^^^^^^^^^^^|^^^^^^^^^^^^^^^^^^^^
                        content:      :                                 `-- It is not true.
                        content:  194 |     assert ("nohz=on" in $cmdline) "tuned nohz=on should still be present"
                        content:      `----
                        content: 
                        content: Connection to localhost closed.
                    failures.yaml (/var/tmp/tmt/bootc-tmt-8ki58tkb-tmt-plans-integration-plan-42-loader-entries-source/tmt/plans/integration/plan-42-loader-entries-source/execute/data/guest/default-0/tmt/tests/tests/test-42-loader-entries-source-1/failures.yaml)
                00:00:05 pass avc (after-test check)
        summary: 1 test failed

total: 1 test failed

I have reran, but not sure if the second one is a flake or not...

Johan-Liebert1 · 2026-07-03T12:41:35Z

I can't view the merge queue results until another merge is performed. Is this a new github bug?

Johan-Liebert1 · 2026-07-03T14:16:24Z

So this test passes for ostree v2026.1 but fails for v2026.2. I am not sure where the actual issue is. I see this that has ostreedev/ostree#3583 some kargs related changes... @jmarrero any ideas?

Johan-Liebert1 · 2026-07-03T14:24:28Z

The issue is something else ig, these should be the final kargs

kargs_refs: ["root=UUID=2d5a2f0d-6647-463e-b080-b3c0500909de", "rw", "localtestkarg=somevalue", "otherlocalkarg=42", "kargsd-test=1", "kargsd-othertest=2", "console=ttyS0,115200n8", "testing-
kargsd=3", "console=hvc0","ostree=/ostree/boot.1/default/733c008a844763a44d0056779d575fc37a97997f8f6f60af38c73094ccb1f526/0"]

but I still have my karg in the staged deployment file

[...] extrax-options-source-johanbruh=hurb a{ss}ή(

Johan-Liebert1 requested a review from cgwalters May 15, 2026 07:22

Johan-Liebert1 added the ci/merge Run full CI suite (all OSes) — equivalent to merge queue label May 15, 2026

gemini-code-assist Bot reviewed May 15, 2026

View reviewed changes

Comment thread crates/lib/src/ukify.rs Outdated

Comment thread tmt/tests/Dockerfile.upgrade Outdated

Comment thread contrib/packaging/seal-uki Outdated

Johan-Liebert1 force-pushed the uki-cleanup branch from cb42d78 to 8a3ed64 Compare May 15, 2026 08:09

github-actions Bot added the area/documentation Updates to the documentation label May 15, 2026

Johan-Liebert1 force-pushed the uki-cleanup branch 7 times, most recently from 66dc0e3 to 3a9dc2b Compare May 18, 2026 10:09

cgwalters reviewed May 18, 2026

View reviewed changes

Comment thread tmt/tests/booted/test-install-to-filesystem-var-mount.sh Outdated

Comment thread tmt/plans/integration.fmf

Johan-Liebert1 force-pushed the uki-cleanup branch from 3a9dc2b to 8e581d4 Compare May 19, 2026 09:35

Johan-Liebert1 requested a review from cgwalters May 19, 2026 10:13

Johan-Liebert1 force-pushed the uki-cleanup branch from 8e581d4 to 503b07a Compare May 19, 2026 13:00

cgwalters reviewed May 19, 2026

View reviewed changes

Comment thread crates/lib/src/bootc_composefs/status.rs Outdated

Comment thread crates/lib/src/bootc_composefs/status.rs Outdated

Comment thread tmt/tests/booted/test-install-to-filesystem-var-mount.sh

Comment thread Dockerfile Outdated

travier reviewed May 21, 2026

View reviewed changes

Comment thread tmt/tests/booted/test-install-to-filesystem-var-mount.sh Outdated

Johan-Liebert1 force-pushed the uki-cleanup branch from 503b07a to 8a5c7ae Compare May 21, 2026 10:10

Johan-Liebert1 force-pushed the uki-cleanup branch from 8a5c7ae to cc8a2d8 Compare May 28, 2026 06:48

github-actions Bot added area/install Issues related to `bootc install` area/ostree Issues related to ostree labels May 28, 2026

Johan-Liebert1 force-pushed the uki-cleanup branch from f2c3424 to 667a0fd Compare May 28, 2026 10:30

Johan-Liebert1 force-pushed the uki-cleanup branch 2 times, most recently from 4d6bed6 to 20e2fed Compare May 28, 2026 13:11

cgwalters requested changes May 28, 2026

View reviewed changes

Comment thread crates/lib/src/cli.rs Outdated

Comment thread crates/lib/src/cli.rs Outdated

Johan-Liebert1 force-pushed the uki-cleanup branch 6 times, most recently from aa321cd to c392676 Compare June 30, 2026 06:26

Johan-Liebert1 requested a review from cgwalters June 30, 2026 09:16

cgwalters previously approved these changes Jun 30, 2026

View reviewed changes

cgwalters added this pull request to the merge queue Jun 30, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 30, 2026

Johan-Liebert1 added this pull request to the merge queue Jul 1, 2026

Johan-Liebert1 added 4 commits July 1, 2026 11:22

test/integration: Test vmlinuz non-existence with UKI

bef390c

vmlinuz and intrd should not be present in UKI images; add test for the same Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

cli: Add cmd bootc internals uki-extract

df0ae03

So we can just use bootc to extract the `.linux` and `.initrd` sections from the UKI and not have to use objcopy Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

cli: Add cmd bootc container split-kernel-and-rootfs

ece22d8

This command is equivalent to `mv /target-root/usr/lib/modules/$kver/{vmlinuz,initramfs.img} /out/$kver` We could just use `mv`, but having an actual bootc cmd is cleaner Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Johan-Liebert1 removed this pull request from the merge queue due to a manual request Jul 1, 2026

Johan-Liebert1 dismissed cgwalters’s stale review via 707b64e July 1, 2026 08:09

Johan-Liebert1 force-pushed the uki-cleanup branch from c392676 to 707b64e Compare July 1, 2026 08:09

Johan-Liebert1 added 4 commits July 1, 2026 16:54

justfile: Build packages as deps for stage fetch

c7edd25

Now since we need to build our initramfs before the `fetch` build stage, we need packages built first as we need `bootc` and `bootc-initramfs-setup` binaries Signed-off-by: Pragyan Poudyal <pragyanpoudyal41999@gmail.com>

Johan-Liebert1 force-pushed the uki-cleanup branch from 707b64e to eecbb11 Compare July 1, 2026 11:24

Johan-Liebert1 mentioned this pull request Jul 2, 2026

ci: Don't let heavy jobs block merge-queue entry in PRs #2274

Open

cgwalters approved these changes Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

Johan-Liebert1 commented May 15, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

travier commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented May 20, 2026

Uh oh!

Uh oh!

travier commented May 21, 2026

Uh oh!

Johan-Liebert1 commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Johan-Liebert1 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Johan-Liebert1 commented Jul 3, 2026

Uh oh!

Johan-Liebert1 commented Jul 3, 2026

Uh oh!

Johan-Liebert1 commented Jul 3, 2026

Uh oh!

Johan-Liebert1 commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

travier commented May 20, 2026 •

edited

Loading

Johan-Liebert1 commented May 28, 2026 •

edited

Loading

Johan-Liebert1 commented Jun 30, 2026 •

edited

Loading