bklibcvenv: hermetic glibc helper (analogue to bkpyvenv)

[tannevaled]

Summary

Propose a brewkit helper libexec/bklibcvenv that lets a recipe produce a binary bottle whose runtime libc is fully hermetic — i.e. the binaries always use the bundled gnu.org/glibc@X.Y regardless of what the host has. Same shape as the existing libexec/bkpyvenv (two-phase stage / seal), and aligned with the pattern @codex described in #342:

Hermetic binary: use a wrapper:

exec "$prefix/lib/ld-linux-x86-64.so.2" \
  --library-path "$prefix/lib:$other_libs" \
  "$prefix/libexec/foo" "\$@"

The inner ELF can even have a dummy/system PT_INTERP; the wrapper explicitly invokes the loader.

Context

Triggered by:

• RFC: pkgx.sh/bootstrap-tools — seed bottle to close pantry's bootstrap problem pantry#12972 — host-independent / older-glibc work; @jhheider proposes brewkit default to gnu.org/glibc@2.17 for Linux builds, override per-package via build.dependencies.
• fix-elf: patchelf PT_INTERP to strip +brewing post-install #342 (withdrawn) — initial attempt to fix PT_INTERP +brewing leak via fix-elf. Wrong layer; @codex's review converged on the wrapper-script approach.
• feat(gnu.org/glibc): host-independent recipe, 9 versions × 2 arches incl. HPC 2.17 / 2.24 pantry#12968 — the glibc recipe's test already demonstrates the wrapper pattern works: \"\$LIBDIR/\$LDSO\" --library-path \"\$LIBDIR\" ./test-bottle produces a verifiably-hermetic execution.

Proposed API

Mirror bkpyvenv:

# In a recipe's build.script, after make install:
- bklibcvenv stage {{prefix}}           # sets CC/CXX/CPP/SYSROOT for hermetic build
- ../configure --prefix={{prefix}}  # (already wired through CC etc.)
- make && make install
- bklibcvenv seal  {{prefix}}           # bin/* → libexec/*; bin/* becomes shell wrappers

Phase 1: `bklibcvenv stage `

• Resolve the glibc bottle prefix (from `build.dependencies.gnu.org/glibc` resolution, env-injected by brewkit)
• Determine arch's LDSO (`ld-linux-x86-64.so.2` / `ld-linux-aarch64.so.1`) and LIBDIR (`/lib/glibc-X.Y/`)
• Export:
  • `SYSROOT=`
  • `CC="$CC -nostdinc -isystem /include -L -Wl,--rpath= -Wl,--dynamic-linker=/"`
  • `CXX="$CXX …"` similarly
  • `CPP="$CPP -nostdinc -isystem …"`

Phase 2: `bklibcvenv seal `

For every ELF executable in `$prefix/bin/`:

1. `mv $prefix/bin/foo → $prefix/libexec/foo`

2. Write `$prefix/bin/foo`:

   #!/bin/sh
   here=\"\$(cd \"\$(dirname \"\$0\")\" && pwd)\"
   exec \"\$here/../lib/glibc-X.Y/<LDSO>\" \
     --library-path \"\$here/../lib/glibc-X.Y:\$here/../lib\" \
     \"\$here/../libexec/\$(basename \"\$0\")\" \"\$@\"

3. `chmod +x $prefix/bin/foo`

Skip symlinks, skip non-ELFs (shell scripts already handled by `fix-shebangs.ts`).

Relation to @jhheider's default-glibc plan

These compose orthogonally:

Layer	Mechanism	Who needs it
Build defaults to glibc 2.17 (forward-compat baseline)	brewkit `useShellEnv` injection (no recipe change)	~95% of recipes
Override glibc version	`build.dependencies.gnu.org/glibc: '>=2.34'`	A few recipes needing newer symbols
Opt out entirely	`build.skip: linux-glibc` (like `skip: fix-machos`)	Rare — libc-less Go binaries, kernel modules
Hermetic binary (host glibc never touched at runtime)	`bklibcvenv`	Numerical / scientific packages, packages that must run on Alpine + Debian + RHEL identically

`bklibcvenv` is purely opt-in — recipes that don't need runtime hermeticity simply don't call it.

Pilot recipes

Candidates that would benefit and could validate the helper:

• Something with C extensions and tight glibc-symbol coupling (numpy.org?)
• A scientific computing lib that has to work on HPC clusters (manylinux2014 era)
• llm.cpp or similar ML inference binary that ships a self-contained tarball today

Open questions

1. Wrapper language: pure POSIX `sh` (least dependency, slowest startup) vs a tiny compiled wrapper (faster, but bootstrap problem). Lean toward `sh` like `fix-shebangs.ts` output, but worth discussing.
2. `--library-path` content: only the bundled glibc's LIBDIR, or also `$prefix/lib` for the bottle's own non-libc shared libs? Probably both — but order matters (loader resolves left-to-right).
3. Detecting "this prefix has bklibcvenv-sealed wrappers": brewkit's auditor should know to follow `bin/foo` → `libexec/foo` and audit the inner ELF, not the wrapper.
4. Interaction with PT_INTERP: the libexec inner ELF can keep whatever PT_INTERP it has (including `+brewing` — the wrapper ignores it). So this completely sidesteps the brewkit#342 problem.

Next steps

Happy to draft a PR for `libexec/bklibcvenv` if there's appetite. Would also need:

• Pantry-side: at least one pilot recipe using `bklibcvenv stage` / `seal`
• brewkit auditor: skip the wrapper's "shebang" check, follow into libexec for the real audit
• Docs in brewkit AGENTS.md ("hermetic glibc helper, mirrors bkpyvenv")

cc @jhheider

[tannevaled]

Renamed: bkcenv → bkcvenv for naming symmetry with bkpyvenv (bk + lang + venv). The "v" reads as "virtual"/"vendored" — same intuition as Python's venv: an isolated, self-contained runtime environment bundled with the binary.

[jhheider]

how do we feel about bklibcvenv for better clarity.

1. POSIX sh is correct
2. good question. does it need the rest of LD_LIBRARY_PATH to get dependencies?
3. this is a good idea; we don't currently do this for bkpyvenv, but we do detect broken symlinks.
4. 👍

how do we feel about taking a glibc version to stage and runtime checking the system's glibc version? is it, as my brain thinks, just going to be way too much overhead? maybe check and write a state file in a pkgx state directory for quick reference? just trying to think through. I guess if we use this script, we specifically want the glibc we depended on.

[tannevaled]

Rename → bklibcvenv

Accepted. bk + libc + venv is clearer than bkcvenv — leaves room for a future bkc*env for non-libc C things if that ever makes sense. Updating the issue title + body, and the cross-reference in pkgxdev/pantry#12972.

Replies

1. POSIX sh — ✅ same shape as bkpyvenv and fix-shebangs.ts output.

2. --library-path content — yes, the wrapper needs to inherit transitive lib paths from build.dependencies, the same way useShellEnv already does today. The wrapper template would be:

#!/bin/sh
here="$(cd "$(dirname "$0")" && pwd)"
exec "$here/../lib/glibc-X.Y/ld-linux-*.so.*" \
  --library-path "$here/../lib/glibc-X.Y:$here/../lib:DEP1/lib:DEP2/lib:..." \
  "$here/../libexec/$(basename "$0")" "$@"

DEP1/lib:DEP2/lib:... is the colon-joined list of <dep>/lib for every entry in the recipe's build.dependencies. bklibcvenv stage reads them from brewkit's env (already populated) and bakes them into the wrapper at seal time. Self-contained — no env var needed at runtime.

3. Auditor follow-through — agreed; track as a follow-up brewkit PR. Same shape as the broken-symlink detection that bkpyvenv triggers today; the auditor would just need to also follow bin/foo → libexec/foo when the wrapper matches our generated template.

4. PT_INTERP sidestep — 👍. Inner ELF in libexec/ keeps whatever PT_INTERP it has (could be the build-time +brewing path, or a system path, doesn't matter). Wrapper invokes the loader explicitly so PT_INTERP is never read.

New question: runtime glibc-version check

I like the state-file idea. Three modes for stage:

• bklibcvenv stage <prefix> — default, always-hermetic
• bklibcvenv stage --min-glibc=2.34 <prefix> — bake the minimum version into the wrapper
• bklibcvenv stage --prefer-system <prefix> — runtime check, fall back to bundled

For the runtime-check mode the wrapper would be:

#!/bin/sh
# Generated by bklibcvenv seal --prefer-system --min-glibc=2.34
STATE="${PKGX_STATE:-$HOME/.cache/pkgx}/libcvenv/<pkg-id>"
here="$(cd "$(dirname "$0")" && pwd)"

if [ -f "$STATE/system-ok" ]; then
  exec "$here/../libexec/$(basename "$0")" "$@"   # system glibc, no wrapper
fi

sys_v=$(getconf GNU_LIBC_VERSION 2>/dev/null | awk '{print $2}')
if [ -n "$sys_v" ] && \
   [ "$(printf '%s\n2.34\n' "$sys_v" | sort -V | head -1)" = "2.34" ]; then
  mkdir -p "$STATE" && touch "$STATE/system-ok"
  exec "$here/../libexec/$(basename "$0")" "$@"
fi

exec "$here/../lib/glibc-X.Y/ld-linux-*.so.*" \
  --library-path "$here/../lib/glibc-X.Y:..." \
  "$here/../libexec/$(basename "$0")" "$@"

• Cold start: getconf + sort + awk (~5ms) + a stat
• Warm start: one stat
• Cache invalidation: when pkgx upgrades the system glibc-compat baseline, bump <pkg-id> (or just rm the state dir)

WDYT — want me to draft a PR around this shape?

[jhheider]

Love it. Instead of baking in the deps, appending or LD_LIBRARY_PATH at runtime should work, right?

[tannevaled]

Validation: pkgxdev/pantry commit 5354c73f is bklibcvenv implemented inline in the glibc recipe.

@jhheider just pushed this on the merged glibc PR. The recipe now:

• Moves bin/* + sbin/* ELFs into libexec/glibc-bin/ and libexec/glibc-sbin/
• Replaces each bin/foo with a POSIX sh wrapper that:
  1. Computes its own prefix via cd "$(dirname $0)/.." (no absolute paths)
  2. Invokes ld.so --library-path "$libdir" "$prefix/libexec/.../foo" explicitly
• Linker scripts (libc.so etc.) use basenames — bottle is now FULLY relocatable

This is exactly the shape proposed in this RFC, just written inline in YAML instead of as a libexec/bklibcvenv helper. So the design is empirically validated.

Refactoring path to make it a brewkit helper:

# libexec/bklibcvenv  (new)
case "$1" in
  seal)
    PREFIX=$2
    case "$(uname -m)" in
      x86_64) LDSO=ld-linux-x86-64.so.2 ;;
      aarch64) LDSO=ld-linux-aarch64.so.1 ;;
    esac
    LIBC_VERSION="${LIBC_VERSION:-detected-from-libc.so.6-soname}"
    for dir in bin sbin; do
      mkdir -p "$PREFIX/libexec/glibc-$dir"
      for f in "$PREFIX/$dir"/*; do
        [ -f "$f" ] && [ ! -L "$f" ] || continue
        patchelf --print-interpreter "$f" >/dev/null 2>&1 || continue
        mv "$f" "$PREFIX/libexec/glibc-$dir/"
        cat > "$f" <<EOT
#!/bin/sh
bindir=$(...)  # canonical resolution
prefix=$(...)
libdir="\$prefix/lib/glibc-$LIBC_VERSION"
exec "\$libdir/$LDSO" --library-path "\$libdir" "\$prefix/libexec/glibc-$dir/\$(basename \"\$0\")" "\$@"
EOT
        chmod 775 "$f"
      done
    done
    ;;
esac

Recipe becomes:

- bklibcvenv seal {{prefix}}

Two recipes worth of usage (current glibc + any future older-glibc versions) is enough to justify the extraction. Want me to draft a PR that extracts this into libexec/bklibcvenv + share/brewkit template, with the glibc recipe migrated as the first consumer?

(And @jhheider's commit also serves as validation for #347's bkwinvenv sketch — exact same pattern, just .cmd wrappers + LoadLibraryEx-equivalent layout via DLL search order.)

[tannevaled]

Drafted the extraction PR: #348 (closes this issue).

Shape mirrors bkpyvenv:

• libexec/bklibcvenv — recipe-side helper
• share/brewkit/libcvenv-wrapper.sh — POSIX sh wrapper template

Recipe call:

- bklibcvenv seal {{prefix}} glibc-{{version.marketing}}

…replaces the ~20-line inline for dir in bin sbin; ... sed -e ... block currently in pkgxdev/pantry@5354c73f.

@jhheider — happy to follow up with the glibc-recipe migration as a separate pantry PR if you want to see the full delta before greenlighting this. Or close this if you'd rather keep the pattern inline (lower abstraction cost, but no reuse path for future libc bottles).