[ci] Make "Linux > Build" reruns recover from immutable-artifact collision

[simonrozsival]

Summary

Make the Linux > Build CI job recover on a rerun instead of getting permanently stuck once it has published its SDK artifact. Today, if the first attempt builds and publishes nuget-linux-unsigned but then fails in a later (often post-job) step, every "rerun failed jobs" attempt fails at the publish step with:

##[error]Artifact nuget-linux-unsigned already exists for build <id>.

…because pipeline artifact names are immutable per build. The only recovery is to queue an entirely new build. This PR detects the already-published artifact up front and skips the rebuild + republish so the rerun goes green and reuses what the earlier attempt produced.

Background — what actually happens

Observed on public build 1489305 (dotnet-android, dnceng-public):

Attempt 1 (original run) — the build itself succeeded:

• make jenkins ✓ → make create-nupkgs ✓ → copy linux sdk ✓
• upload linux sdk ✓ and upload linux sdk symbols ✓ — nuget-linux-unsigned was successfully published
• Then the auto-injected post-job Cache@2 save ("cache Android toolchain archives") failed because the agent disk was full:

  ##[warning]Free disk space on / is lower than 5%; Currently used: 96.90%
  tar: 42e30a39...archive.tar: Wrote only 4096 of 10240 bytes
  tar: Error is not recoverable: exiting now
  ##[error]Process returned non-zero exit code: 2
  

  A failing post-job step marks the whole job failed — even though the real product (the SDK artifact) was already built and uploaded.

Attempt 2 (rerun failed jobs):

• The job re-runs from scratch on a fresh, clean agent (workspace: clean: all, ephemeral 1ES pool), rebuilds everything (~20+ min), re-uploads the content, then fails at the final associate step:

  Content upload is done!   ← bytes uploaded fine
  ##[error]Artifact nuget-linux-unsigned already exists for build 1489305.
  

Why the build becomes un-rerunnable

Pipeline artifacts are keyed by (buildId + artifactName) and are immutable. The artifact name only registers after its content is fully committed — that is literally the step that fails with "already exists". So once attempt 1 registers nuget-linux-unsigned on the build, every rerun of that job re-executes the publish and collides. No rerun can ever go green; only a brand-new build (new buildId) will pass.

The underlying trigger here is a full agent disk during the post-job toolchain-cache save — infrastructure we do not control on the public pool. Rather than chase the disk-full root cause, this PR makes the job robust on reruns regardless of why the earlier attempt failed after publishing.

What this PR changes

build-tools/automation/yaml-templates/build-linux-steps.yaml:

1. New early guard step — "check for existing linux sdk artifacts". Queries the build's artifacts REST API (_apis/build/builds/$(Build.BuildId)/artifacts) using $(System.AccessToken) and probes each artifact independently, setting a per-artifact job variable: AlreadyBuiltLinuxSdk for nuget-linux-unsigned and AlreadyBuiltLinuxSdkSymbols for nuget-linux-unsigned-symbols. Any API/network error falls back to false → build normally (safe default).
2. Skip the expensive work only when both artifacts already exist. make jenkins, make create-nupkgs, the symbols CopyFiles/DeleteFiles, and copy linux sdk gain condition: and(succeeded(), or(ne(variables['AlreadyBuiltLinuxSdk'], 'true'), ne(variables['AlreadyBuiltLinuxSdkSymbols'], 'true'))) — so a partial prior publish still rebuilds the files needed for the missing artifact.
3. Skip each colliding publish independently. The SDK publish is gated on AlreadyBuiltLinuxSdk, the -symbols publish on AlreadyBuiltLinuxSdkSymbols. If a prior attempt published the SDK but died before -symbols, the rerun rebuilds, skips the SDK publish (no collision), and publishes only the missing -symbols artifact.

build-tools/automation/yaml-templates/build-linux.yaml:

4. Moved the 1ES SDK publish out of templateContext.outputs (which always runs at job end and cannot be conditioned) into inline 1ES.PublishPipelineArtifact@1 steps in the steps template, guarded by the skip condition. This mirrors the already-approved pattern in publish-artifact.yaml (used for the "Build Results - Linux" artifact in this same job), so it stays within 1ES governance.

Behavior after this change

On a rerun where both artifacts already exist, the job: checks the API → sets both AlreadyBuiltLinuxSdk=true and AlreadyBuiltLinuxSdkSymbols=true → skips the rebuild and both publishes → finishes green in ~1–2 minutes, reusing the valid artifacts from the earlier attempt. If only one artifact was published before the earlier attempt failed, the rerun rebuilds and republishes just the missing one. Downstream stages already consume nuget-linux-unsigned from the build via DownloadPipelineArtifact, so they are unaffected. Because nothing heavy runs, the disk never re-fills, so the post-job cache save no longer fails either.

First runs are unchanged: the artifact doesn't exist yet, so the guard returns false and the full build runs as before.

Why not other approaches

• Attempt-unique artifact name (like the (Attempt N) suffix used for "Build Results") — can't be used for nuget-linux-unsigned, because downstream stages reference the fixed name.
• Fix the disk-full root cause — the disk pressure is on the public 1ES pool image and not something we can reliably control; and even a well-behaved build can fail after publishing for other reasons, so rerun-safety is valuable on its own.
• Condition on templateContext.outputs entries — output entries always run at job end and there's no in-repo precedent for conditioning them; using inline 1ES.PublishPipelineArtifact@1 (already used elsewhere in the repo) is the more certain, precedented mechanism.

Testing / validation

• Both templates parse as valid YAML.
• The artifact existence check was validated against build 1489305: the REST API cleanly reports each artifact present via the exact "name":"<name>" fixed-string match, so probing nuget-linux-unsigned does not false-match nuget-linux-unsigned-symbols and each variable is set independently.
• Full validation happens in this PR's own dotnet-android run (first run exercises the normal build path; the skip path is exercised on any rerun).

Notes

Since this only affects CI pipeline YAML, there is no product/runtime impact.

[Copilot]

Pull request overview

This PR updates the Linux build pipeline templates to make reruns resilient when the Linux SDK pipeline artifact has already been published in a previous attempt (artifact names are immutable per buildId + artifactName, so republishing on rerun can permanently fail).

Changes:

• Added an early step in build-linux-steps.yaml to query the Azure DevOps build artifacts API and set a variable used to skip rebuild/republish on rerun.
• Added condition: guards to skip the expensive build/packaging steps when the artifact already exists.
• Moved Linux SDK artifact publishing out of templateContext.outputs (unconditional) and into explicit (conditionally guarded) publish steps.

Show a summary per file

File	Description
build-tools/automation/yaml-templates/build-linux.yaml	Removes templateContext.outputs artifact publishing so publishing can be conditioned within steps.
build-tools/automation/yaml-templates/build-linux-steps.yaml	Adds artifact-existence probing and conditions to skip rebuild/republish work on reruns.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 3

[jonathanpeppers]

Did these just move? We might want to queue a main/Xamarin.Android/DevDiv pipeline build to make sure nothing broke there.

[simonrozsival]

Yes — they moved out of templateContext.outputs (which always runs at job end and can't be conditioned) into inline 1ES.PublishPipelineArtifact@1 steps so they can be skipped on reruns. The public dotnet-android run here exercises them, but agreed it's worth queueing an internal Xamarin.Android/DevDiv build to confirm the 1ES production template path still publishes correctly. I can kick one off — or if you'd prefer to queue it, happy to hold for that before merge.

[jonathanpeppers]

I guess this one is from a fork, so we can't queue the internal pipeline...

[jonathanpeppers]

Ok, I pushed your branch here to origin, testing at:

• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=14557800&view=results