[#728] Reject TCP self-connects in replication connect paths#729
Open
vharseko wants to merge 1 commit into
Open
[#728] Reject TCP self-connects in replication connect paths#729vharseko wants to merge 1 commit into
vharseko wants to merge 1 commit into
Conversation
Test listen ports for replication servers are drawn from the OS ephemeral range (TestCaseUtils.findFreePorts binds port 0). When an RS is stopped, its former peers keep reconnecting to the vacated port in a retry loop; on Linux the kernel eventually assigns the destination port itself as the local port of such a connect and TCP simultaneous open "establishes" the connection to itself. The self-connected socket then occupies the very port the next RS is about to bind: ReplicationServer.initialize() logs "could not bind to the listen port ... Address already in use", swallows the error, and the RS runs without a listener - the intermittent AssuredReplicationServerTest.testSafeDataLevelOne setup failure on CI (isConnected() expected [true] but found [false]). Detect self-connected sockets right after connect() in both outgoing paths (ReplicationBroker.performPhaseOneHandshake, ReplicationServer.connect) and fail the attempt with ConnectException: the existing error handling closes the socket - releasing the port - and retries or blacklists as usual. Same fix as ZOOKEEPER-2101 / KAFKA-1836 for the identical bug. A legitimate connection can never have identical local and remote endpoints, so no false positives are possible. TIME_WAIT was ruled out: the JDK enables SO_REUSEADDR on ServerSocket by default on POSIX platforms, so rebinding over TIME_WAIT already works.
maximthomas
approved these changes
Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #728.
Problem
AssuredReplicationServerTest.testSafeDataLevelOnefails intermittently onCI during setup (e.g. ubuntu-latest/25 on PR #727's run, unrelated to it):
createFakeReplicationDomain:321 expected [true] but found [false], with thereal cause right above in the server log:
Root cause
TCP self-connect (simultaneous open), the same bug class as
ZOOKEEPER-2101 / KAFKA-1836 / JDK-8067207:
TestCaseUtils.findFreePorts()=bind(0),i.e. they live inside the OS ephemeral port range (the OS assigned
65531 in the first place).
(
ReplicationBrokers of fake domains, other RSs) keep reconnecting to thevacated port in a retry loop.
connect()draws a local ephemeral port; on Linux, when the kernelpicks the destination port itself, the connect succeeds via TCP
simultaneous open — the socket is connected to itself and now occupies
the very port the next RS is about to bind.
ReplicationServer.initialize()fails to bind, only logs the error, andthe RS runs without a listener — the fake domain then cannot connect.
TIME_WAIT was ruled out: the JDK enables
SO_REUSEADDRonServerSocketbydefault on POSIX platforms (verified empirically), so rebinding over
TIME_WAIT remnants already works. macOS/BSD refuse self-connects (
EINVAL),consistent with the flake being ubuntu-only.
Fix
Detect self-connected sockets right after
connect()in both outgoingreplication connect paths and fail the attempt with
ConnectException:StaticUtils.isSelfConnection(Socket)— local endpoint == remote endpoint.A legitimate established connection can never satisfy this, so false
positives are impossible.
ReplicationBroker.performPhaseOneHandshake()— the existing errorhandling closes the socket (releasing the stolen port) and retries.
ReplicationServer.connect()— the existing failure path closes the socketand blacklists the address for a few connect iterations.
Tests
TestStaticUtils.testIsSelfConnectionFalseForNormalConnection— both endsof a normal loopback connection are not self-connections.
TestStaticUtils.testIsSelfConnectionTrueForSelfConnectedSocket— forces adeterministic self-connect (binds the local end to the very port being
connected) and asserts detection; skipped on BSD-based stacks (macOS
refuses explicit self-connects with
EINVAL), runs on the Linux CI legs.GenerationIdTest(4/4) andTestStaticUtils(1755, 1 skip on macOS) greenlocally with the guard in place — normal broker/RS connects unaffected.
Follow-ups tracked in #728:
initialize()swallowing the bind failure, andthe
isAddressInUse()dummy-connect probe being self-connect-prone itself.