Skip to content

Validate the page before re-inserting an index pointer in fixIndexHole (bug 1017957 class race)#263

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/bug1017957-fixindexhole-race
Jul 11, 2026
Merged

Validate the page before re-inserting an index pointer in fixIndexHole (bug 1017957 class race)#263
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/bug1017957-fixindexhole-race

Conversation

@vharseko

Copy link
Copy Markdown
Member

Problem

Bug1017957Test.induceCorruptionByStress fails intermittently on CI with transient CorruptVolumeExceptions (invalid page type 30: should be 2, up to 47 in one 10-second run on macos-latest, 26), while the final IntegrityCheck reports zero faults. Review of #257 argued this is a real race rather than a benign observation artifact — that turned out to be correct.

Root cause

The CI stack traces show a descent reading a pointer under a claim on a live index page and landing on a page typed PAGE_TYPE_GARBAGE (30) — i.e. a live index entry pointed at a garbage page. Working back from that:

  1. rebalanceSplit() (the RebalanceException path inside a spanning removeKeyRange) creates a page reachable only through its sibling chain — an index hole — and defers the parent index entry to a background CleanupManager.CleanupIndexHole action.
  2. Before the background queue runs (poll interval up to ~1 s), a covering removeKeyRange joins that range: the hole page is unlinked onto a garbage chain. Its content and type survive (or it is retyped PAGE_TYPE_GARBAGE if it becomes a chain root), and it may be reused by an unrelated allocation.
  3. Exchange.fixIndexHole() performed no validation: it read the "first key" of whatever the page now holds and inserted a key-pointer pair into the live index — planting a dangling entry to a garbage or reused page. This is exactly the bug 1017957 mechanism-1 failure mode; all three signatures from the 2012 report (invalid page type, walked right more than 50 pages, LONG_RECORD chain is invalid) fall out of it.
  4. Subsequent covering removes delete the dangling entry, so IntegrityCheck at test end sees a clean volume — which is why the corruption looked "transient". Faster JVMs iterate more, producing more rebalance splits racing the background queue — hence the failures appearing on modern runners.

Fix

Guard the insertion in fixIndexHole(). Structure deletes require the tree writer claim, so under the reader claim the method already holds, a page's reachability cannot change; validation is therefore race-free:

  • drop the action if the page no longer has this level's page type, or is empty (retyped garbage root / reused elsewhere);
  • verify the page is still reachable in this tree: a search for its current first key at that level must land on the page itself (via the B-link right-walk — that is how a hole page is reached), and drop the stale action otherwise.

No claim is held across the verification descent, preserving the parent-then-child claim order (no deadlock).

Verification

  • Full persistit/core suite: 565 tests, 0 failures, 0 errors (5 pre-existing skips).
  • IntegrityCheckTest.testIndexFixHoles: valid index holes are still repaired (the guard does not drop legitimate actions).
  • RecoveryTest.testIndexHoles, Bug1022567Test, TreeTransactionalLifetimeTest: green.
  • Bug1017957Test.induceCorruptionByStress with its original strict zero-exception assertion: 6/6 repeated runs green.
  • The race itself reproduces only on CI runners (0/21 local attempts even before the fix), so CI is where this fix is ultimately validated. The strict assertion in Bug1017957Test — restored in Fix bug-1017957 class race in fixIndexHole; stabilize TreeTransactionalLifetimeTest and WarmupTest #257 — is the regression detector.

…e (bug 1017957 class race)

rebalanceSplit() links a new page only through its sibling chain and defers the
parent index entry to a background CleanupIndexHole action. By the time the
CleanupManager runs it, the action may be arbitrarily stale: a covering
removeKeyRange can have unlinked the page onto a garbage chain (content and
type intact, or retyped PAGE_TYPE_GARBAGE if it became a chain root), or the
page may have been reused. fixIndexHole() performed no validation and inserted
a key-pointer pair for whatever the page now holds, planting a dangling index
entry to a garbage or reused page -- the bug 1017957 failure mode. Concurrent
descents following such an entry throw transient CorruptVolumeExceptions
("invalid page type 30: should be 2" observed on CI under
Bug1017957Test.induceCorruptionByStress; "walked right more than 50 pages" and
"LONG_RECORD chain is invalid" are the same race's other signatures). The
volume looks consistent afterwards because later covering removes delete the
dangling entry before IntegrityCheck runs, which is why the corruption appeared
transient. Faster JVMs widen the window: more iterations produce more
rebalance splits racing the background queue.

Guard the insertion. Under the tree reader claim (structure deletes require
the writer claim, so reachability cannot change while it is held): (a) drop
the action if the page is no longer this level's type or is empty; (b) verify
the page is still reachable in this tree -- a search for its current first key
at that level must land on the page itself via the B-link right-walk -- and
drop the stale action otherwise. No claim is held across the verification
descent, preserving the parent-then-child claim order.

Verified: full persistit/core suite 565/565 green; IntegrityCheckTest
testIndexFixHoles confirms valid holes are still repaired;
Bug1017957Test.induceCorruptionByStress with its original strict zero-exception
assertion green across repeated runs. The race itself reproduces only on CI
runners, which is where this fix is ultimately validated.
@vharseko vharseko merged commit fe7a7b9 into OpenIdentityPlatform:master Jul 11, 2026
14 checks passed
@vharseko vharseko deleted the fix/bug1017957-fixindexhole-race branch July 11, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants