Skip to content

persistit: keep JournalManager consistent when rollover fails near the block end#267

Open
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/persistit-journal-rollover-interrupt
Open

persistit: keep JournalManager consistent when rollover fails near the block end#267
vharseko wants to merge 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:fix/persistit-journal-rollover-interrupt

Conversation

@vharseko

Copy link
Copy Markdown
Member

Summary

Fixes #265.

Under sustained Thread.interrupt() stress the JournalManager write-buffer bookkeeping could be torn around a journal rollover, after which every journal writer tripped the invariant assert (writeBufferAddress=2,999,999,998 position=0 currentAddress=3,000,000,038). The failure is two-phase:

  1. Poisoning. rollover() is not exception-safe: writeJournalEnd() advances _currentAddress by JE.OVERHEAD (40 bytes) to within 40 bytes of the block end, and only then does the I/O (flush/truncate/force). An InterruptedIOException there (surfaced by MediatedFileChannel on an interrupted thread) abandons the rollover before _currentAddress jumps to the next block boundary. The invariant still holds at this point, but the reserve prepareWriteBuffer maintains for the JE record is gone: fewer than JE.OVERHEAD bytes remain to the block end, and the write-buffer limit is clamped to that remainder.
  2. Tearing. The next journal write retries the rollover. The JE.put* helpers write into the backing array directly, bypassing the ByteBuffer limit, so nothing fails until advance(JE.OVERHEAD) updates _currentAddress first and then throws from _writeBuffer.position(...) (newPosition > limit), leaving _currentAddress exactly 40 bytes ahead of _writeBufferAddress + position - the observed values. Every subsequent journal write then fails the assert (or hits the same IllegalArgumentException with assertions disabled).

Changes

  • JournalManager.writeJournalEnd(): skip the JE record when it no longer fits before the block end, so a retried rollover completes and heals the poisoned state; recovery treats the missing JE as a dirty shutdown.
  • JournalManager.advance(): update the buffer position (which validates against the limit and may throw) before _currentAddress, so a failure cannot tear the invariant.
  • JournalRecord: fail fast (IndexOutOfBoundsException) when a record write would exceed the buffer limit instead of silently writing past it into the backing array.
  • IOFailureTest#testRolloverRecoversAfterFailureNearBlockEnd: deterministic reproduction - positions _currentAddress exactly JE.OVERHEAD + 2 bytes before the block end, aborts the rollover with an injected truncate() failure, verifies the poisoned geometry, then verifies the retried rollover heals and journal writes keep working.
  • TransactionTest2: workers catch Throwable so an internal Error is reported and counted instead of silently killing the thread (the cascade that polluted transactionsConcurrentWithPersistitClose in the same CI run).

Testing

  • The new test fails on the unfixed code with IllegalArgumentException: newPosition > limit: (40 > 2) (and the same error from tearDown, showing the state never heals) and passes with the fix.
  • Full persistit/core suite: 566 tests, 0 failures, 0 errors, 5 pre-existing skips.

…e block end

An IOException thrown inside JournalManager.rollover() - e.g. an
InterruptedIOException when the rolling thread is interrupted during
flush, truncate or force - after writeJournalEnd() had already consumed
the JE reserve left _currentAddress closer than JE.OVERHEAD to the block
end. The retried rollover then wrote the JE record past the buffer limit
(JournalRecord putters bypass the ByteBuffer bounds checks) and
advance() updated _currentAddress before the position update threw,
tearing the invariant _writeBufferAddress + _writeBuffer.position() ==
_currentAddress. Every subsequent journal write then failed, seen as the
AssertionError storm in TransactionTest2.transactionsWithInterrupts.

- writeJournalEnd(): skip the JE record when it no longer fits before
  the block end so a retried rollover completes and heals the state;
  recovery treats the missing JE as a dirty shutdown.
- advance(): update the buffer position, which validates against the
  limit and may throw, before _currentAddress so a failure cannot tear
  the bookkeeping.
- JournalRecord: fail fast when a record write would exceed the buffer
  limit instead of silently writing past it into the backing array.
- IOFailureTest: deterministically reproduce the abandoned-rollover
  geometry with an injected truncate failure.
- TransactionTest2: catch Throwable in workers so internal Errors are
  reported and counted instead of silently killing the thread.

Fixes OpenIdentityPlatform#265
@vharseko vharseko added bug tests Test code changes labels Jul 10, 2026
@vharseko vharseko requested a review from maximthomas July 10, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug tests Test code changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

persistit: JournalManager write-buffer bookkeeping torn by interrupt during journal rollover

1 participant