Egress tunnel capture + agent identity by npolshakova · Pull Request #338 · agent-substrate/substrate

Nina Polshakova (npolshakova) · 2026-06-26T21:46:12Z

POC for #126

Based on design discussed in https://docs.google.com/document/d/1KmpIFu2gnqy9gp95wASgIo_vkJ_dA1DZckV8upET6bs/edit?usp=sharing

Tests pass
Appropriate changes to documentation are included in the PR

Summary:
This is a proof of concept egress capture path for actors. It introduces a reusable internal/egresscapture package that:

starts local capture listeners
drives the CONNECT authority from HTTP Host or TLS SNI, and opens a CONNECT-style tunnel to a configured PEP address.

The gVisor and microvm runtimes wire this into actor network setup by redirecting actor HTTP/HTTPS egress traffic to local capture ports. Agentgateway is used as the receiving proxy to prove that captured actor traffic reaches the tunnel endpoint.

Notes:

Tested the gvisor setup and microvm setups in kind. Can split these up into separate chunks to make it easier to review, but this shows the shared code.
Agent identity is currently passed as unsigned metadata headers:
- x-ate-actor-id
- x-ate-actor-template
- x-ate-actor-template-namespace
- x-ate-original-destination
- x-ate-connect-authority
Signed agent/actor identity is out of scope for this PR and should replace the current metadata headers approach once agent identity goes in [Feature] Actor Identity #124
PEP control plane is out of scope of this PR (just using agentgateway to prove out the proxy gets traffic from the tunnel)

Shashank (shashankram) · 2026-06-26T21:57:07Z

+	egressCaptureHTTPPort   = uint16(15001)
+	egressCaptureHTTPSPort  = uint16(15002)


Why do you need to capture HTTP vs HTTPS separately? A single listener could handle both and since SO_ORIGINAL_DST is used anyway to lookup the original port in deriveConnectAuthority.

Yeah I think that's a good clean up- originally I was testing http first and had one redirect per protocol.

Eitan Yarmush (EItanya) · 2026-06-29T22:03:55Z

+const (
+	egressCapturePort       = uint16(15001)
+	egressOriginalHTTPPort  = uint16(80)
+	egressOriginalHTTPSPort = uint16(443)


Is there any specific reason we're only capturing these ports, and not just redirecting everything?

So right now ateom derives the CONNECT authority from HTTP Host header for port 80 OR TLS SNI for port 443. For other ports, the current capture code just falls back to raw original destination IP:port.

We could support trying to figure out the authority with other ports, but it would be a little more complicated because we'd need some sort of classifier:

try TLS SNI on any port

try HTTP Host on any port

fall back to recent per-actor DNS correlation

if still can't figure it out, use IP:port

Eitan Yarmush (EItanya) · 2026-06-29T22:04:40Z

+	egressOriginalHTTPPort  = uint16(80)
+	egressOriginalHTTPSPort = uint16(443)


Same question about only supporting these 2 ports

Eitan Yarmush (EItanya) · 2026-06-29T22:06:09Z

 		cleanupErr = errors.Join(cleanupErr, fmt.Errorf("while removing actor nftables rules: %w", err))
 		slog.WarnContext(ctx, "Failed to remove actor nftables rules; continuing actor netns cleanup", slog.Any("err", err))
 	}
+	if s.egressCapture != nil {


Is there a time we would want no egress capture?

That's a good question- I think in general, egress capture should always be enabled but you can imagine scenarios where that doesn't make sense:

No PEP egress is setup. In this case I think traffic should still be captured by the ateom proxy, but not forwarded to any PEP

Flexibility for workloads using traffic we do not capture yet (UDP/QUIC, non-80/443 TCP if we don't want to do the classifier, etc.)

Mike Morris (@mikemorris) This is kind of related to the discussion we had in slack. Like you brought up, if we're expecting all traffic to always egress through a PEP, then the PEP registration can be a first-class API. Are there cases though where we don't want the traffic to get captured/egress through a PEP?

Eitan Yarmush (EItanya) · 2026-06-29T22:07:31Z

 	return counter
 }

+const defaultEgressURL = "https://httpbin.org/get"


Why was this added to the counter, seems like we should have a different demo potentially

It seemed more lightweight to just have a new endpoint under the counter demo for future testing, but we can make it a separate demo!

Eitan Yarmush (EItanya) · 2026-06-29T22:08:37Z

Should we name this package egress, or maybe tunnel?

Eitan Yarmush (EItanya) · 2026-06-29T22:09:31Z

+	// Keep tunnel protocol support behind factories so additional transports
+	// such as HBONE can plug in without changing capture/listener logic.


I know we discussed keeping the protocol generic, but do we need to start there, it makes the first merge more confusing to read, and I'm not clear if it's actually necessary

Yeah I was hoping to show how it could be extended in the future, but let me clean it up.

Signed-off-by: npolshakova <nina.polshakova@solo.io>

Shashank (shashankram) reviewed Jun 26, 2026

View reviewed changes

Nina Polshakova (npolshakova) force-pushed the egress-tunnel branch 2 times, most recently from 9b63f65 to e678eeb Compare June 29, 2026 16:50

Nina Polshakova (npolshakova) marked this pull request as ready for review June 29, 2026 17:24

Eitan Yarmush (EItanya) reviewed Jun 29, 2026

View reviewed changes

Nina Polshakova (npolshakova) added 3 commits July 1, 2026 15:55

initial tunnel

6f21366

Signed-off-by: npolshakova <nina.polshakova@solo.io>

fix flag forwarding microvm

0747816

Signed-off-by: npolshakova <nina.polshakova@solo.io>

address feedback

2e910e3

Signed-off-by: npolshakova <nina.polshakova@solo.io>

Nina Polshakova (npolshakova) force-pushed the egress-tunnel branch from 0995370 to 2e910e3 Compare July 1, 2026 22:56

		egressCaptureHTTPPort = uint16(15001)
		egressCaptureHTTPSPort = uint16(15002)

		egressOriginalHTTPPort = uint16(80)
		egressOriginalHTTPSPort = uint16(443)

		// Keep tunnel protocol support behind factories so additional transports
		// such as HBONE can plug in without changing capture/listener logic.

Uh oh!

Conversation

Nina Polshakova (npolshakova) commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Nina Polshakova (npolshakova) commented Jun 26, 2026 •

edited

Loading