Skip to content

Make AciList reads lock-free with true copy-on-write#674

Open
vharseko wants to merge 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:features/compare-1
Open

Make AciList reads lock-free with true copy-on-write#674
vharseko wants to merge 3 commits into
OpenIdentityPlatform:masterfrom
vharseko:features/compare-1

Conversation

@vharseko

@vharseko vharseko commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

First result of profiling the COMPARE hot path with the same methodology
as the BIND series (#660, #667#670, #672). The one COMPARE-specific
serialization point vs BIND is access control: AciList.getCandidateAcis()
acquires a ReentrantReadWriteLock read lock on every access-controlled
operation
(compare, search, modify, …) just to walk the ACI map. The field
comment promises "copy-on-write technique to avoid locking when reading",
but the mutators actually modify the volatile DITCacheMap (and its value
lists) in place under the write lock — so the read lock could never be
dropped. This PR makes the copy-on-write real and removes the read lock from
the hot path — the same pattern as the CompressedSchema fix (#670).

Problem

private volatile DITCacheMap<List<Aci>> aciList = new DITCacheMap<>();
// "We use the copy-on-write technique to avoid locking when reading."
...
public List<Aci> getCandidateAcis(DN baseDN) {
    ...
    lock.readLock().lock();      // on EVERY compare/search/modify
    ...
}
public void addAci(DN dn, SortedSet<Aci> acis) {
    lock.writeLock().lock();
    aciList.put(dn, ...);        // in-place mutation — CoW promise broken
}

Every acquire/release CAS-es the shared RRWL sync word — a cross-core
cache-coherency hotspot on the handler singleton, growing with core count.

Change

  • All seven mutators (addAci ×3, modAciOldNewEntry, removeAci ×2,
    renameAci) now build a copy of the map with copied value lists
    (`copyAciList()") under the write lock, modify the copy, and publish it
    through the volatile reference. A published map is never modified in
    place. ACI mutations are rare (startup, backend initialization, ACI
    changes), so the copy cost is negligible.
  • getCandidateAcis() reads a snapshot lock-free.
  • A plain ReentrantLock serializes mutators (the read/write lock lost
    its last read-lock user — see the review follow-up below); public
    behaviour is unchanged.

Benchmark

COMPARE scenario, packaged server with the bind-series fixes applied, je
backend, 5,000 users, JDK 11, 8-core host, access loggers disabled. Load
driver: 200 persistent connections bound as regular users (real ACI
evaluation, no root bypass), each issuing COMPARE uid=user.N / uid / user.N for random users; latency via HdrHistogram; 40 s warm-up run after
every server restart, then 15 s + 75 s measured run under caffeinate.
Interleaved order old→new→new→old→old→new after a cool-down:

slot variant ops/s mean p99 p99.9
1 old 33,582 5.95 ms 19.7 ms 35.3 ms
2 new 36,311 5.50 ms 15.8 ms 26.6 ms
3 new 37,648 5.31 ms 16.7 ms 30.5 ms
4 old 36,584 5.46 ms 15.7 ms 27.1 ms
5 old 37,625 5.31 ms 15.0 ms 25.8 ms
6 new (discarded: end-of-series thermal degradation of the shared host, 0 errors)

Adjacent pairs favor the lock-free version (+8.1%, +2.9%); as with #670 the
local win on 8 cores is modest and the structural value is removing a
serialization ceiling that grows with core count. Errors: 0 in all runs.

Review follow-up (b712f6e)

All three optional review suggestions applied:

  • ReentrantReadWriteLockReentrantLock: the read lock has no users
    since reads went lock-free.
  • Documented in the aciList field javadoc that Aci instances are
    treated as immutable once published — the map and its value lists are
    the only mutable containers the COW contract covers.
  • Added AciListTests, a concurrency regression test that locks in the
    COW contract: three reader threads snapshot getCandidateAcis()
    continuously while a writer appends to the value list under the
    queried key (Entry-based addAci), resets it and renames a
    neighbouring key (50k iterations); readers assert every snapshot is
    consistent and exception-free. The detector was validated in both
    directions: with copyAciList() stubbed to return the live map
    (simulated in-place-mutation regression) the test fails, with real COW
    it passes. Put-only and rename-only churn variants were found NOT to
    catch the regression (same-key put is structure-neutral and rename
    churns unrelated hash buckets), which is why the append path is the
    primary channel.

Testing

  • Behavioural equivalence on the full ACI suite: AciTests (515 tests,
    TestNG group slow, excluded from the default run) produces identical
    results before and after the change
    — 513 pass, plus the same 2
    pre-existing failures (testCompare, testValidAcis) that also fail on
    unmodified master (worth a separate issue; this suite is never exercised
    in CI).
  • CompareOperationTestCase (37), GetEffectiveRightsTestCase (7) — pass.
  • After the follow-up: AciListTests + AciBodyTest +
    GetEffectiveRightsTestCase — 15 tests, 0 failures.

Files

  • opendj-server-legacy/src/main/java/org/opends/server/authorization/dseecompat/AciList.java
  • opendj-server-legacy/src/test/java/org/opends/server/authorization/dseecompat/AciListTests.java (new)

AciList.getCandidateAcis() acquired a ReentrantReadWriteLock read lock
on every access-controlled operation (compare, search, modify, ...) to
walk the ACI map. The field comment claims "copy-on-write technique to
avoid locking when reading", but mutators actually modified the volatile
DITCacheMap (and its value lists) in place under the write lock, so the
read lock could not be dropped.

Make the copy-on-write real: mutators (add/remove/rename/modify) build a
copy of the map with copied value lists under the write lock and publish
it through the volatile reference; a published map is never modified in
place. getCandidateAcis() now reads a snapshot lock-free. ACI mutations
are rare (startup, backend initialization, ACI modifications), so the
copy cost is negligible.
Replace the ReentrantReadWriteLock with a ReentrantLock — the read lock
has no users since reads went lock-free — and document that Aci
instances are treated as immutable once published. Add AciListTests,
which hammers getCandidateAcis() snapshots from several threads while a
writer appends to the list under the queried key, resets it and renames
a neighbouring key; validated to fail against an in-place-mutation
regression and pass with copy-on-write.
@vharseko vharseko added performance Performance / concurrency / lock-contention work ACI Access Control Instructions subsystem labels Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ACI Access Control Instructions subsystem benchmark enhancement performance Performance / concurrency / lock-contention work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants