Skip to content

fix(tracing): Fix unsampled/deferred trace propagation in span streaming#6757

Draft
sentrivana wants to merge 14 commits into
masterfrom
ivana/traceparent-test
Draft

fix(tracing): Fix unsampled/deferred trace propagation in span streaming#6757
sentrivana wants to merge 14 commits into
masterfrom
ivana/traceparent-test

Conversation

@sentrivana

@sentrivana sentrivana commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Description

The SDK didn't behave correctly in some trace propagation scenarios in stream mode.

The current SDK design was emitting headers with a deferred sampling decision if the sampling decision was negative (whether inherited or not), because it was falling back to TWP behavior too eagerly.

Here's how it should be working. In the following, tracing enabled means either a traces_sample_rate or a traces_sampler is defined. Tracing disabled means both are None and tracing without performance is active.

There are three kinds of sampling decisions: sampled (1), unsampled (0), and deferred (``), meaning no decision.

  1. No incoming trace headers
    • tracing enabled: make a sampling decision and propagate it
    • tracing disabled: do not make a decision, propagate a deferred decision (no sampling decision)
  2. Incoming trace headers with a sampled/unsampled decision
    • tracing enabled: respect and propagate incoming decision
    • tracing disabled: respect and propagate incoming decision
  3. Incoming trace headers with a deferred sampling decision (essentially same as 1)
    • tracing enabled: make a sampling decision and propagate it
    • tracing disabled: propagate deferred sampling decision

The fix: NoOpStreamedSpans now carry their own trace and span ID information as well as the current propagation data (a snapshot of the propagation context, like StreamedSpans) and use it to emit the correct headers.

Issues

Closes #6755

Reminders

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Codecov Results 📊

90184 passed | ⏭️ 6302 skipped | Total: 96486 | Pass Rate: 93.47% | Execution Time: 313m 36s

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +150
Passed Tests 📈 +150
Failed Tests
Skipped Tests

All tests are passing successfully.

✅ Patch coverage is 96.67%. Project has 2443 uncovered lines.
❌ Project coverage is 89.75%. Comparing base (base) to head (head).

Files with missing lines (1)
File Patch % Lines
sentry_sdk/traces.py 91.67% ⚠️ 1 Missing
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
- Coverage    89.76%    89.75%    -0.01%
==========================================
  Files          192       192         —
  Lines        23823     23833       +10
  Branches      8226      8232        +6
==========================================
+ Hits         21384     21390        +6
- Misses        2439      2443        +4
- Partials      1351      1348        -3

Generated by Codecov Action

Comment thread sentry_sdk/scope.py
Comment thread sentry_sdk/tracing_utils.py Outdated
Comment thread sentry_sdk/tracing_utils.py Outdated
Comment thread tests/tracing/test_span_streaming.py
@sentrivana sentrivana marked this pull request as ready for review July 6, 2026 12:01
@sentrivana sentrivana requested a review from a team as a code owner July 6, 2026 12:01
Comment thread sentry_sdk/traces.py
parent_span_id: "Optional[str]" = None,
parent_sampled: "Optional[bool]" = None,
baggage: "Optional[Baggage]" = None,
sampled: "Optional[bool]" = False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default sampled=False produces contradictory trace propagation headers for ignored child spans of sampled parents

When an ignored child span (is_ignored_span=True) is created inside a sampled real StreamedSpan, sampled is not passed and defaults to False. The outgoing sentry-trace header then carries -0 (explicitly unsampled) while the baggage header, derived from _segment._get_baggage() on the real sampled segment, carries sentry-sampled=true. Downstream services receiving these contradictory headers will typically honor the traceparent's -0 and drop the trace, breaking continuity of a sampled trace.

Evidence
  • scope.py creates NoOpStreamedSpan(segment=parent_span._segment, ..., unsampled_reason="ignored") without passing sampled when is_ignored_span is true for a child of a real StreamedSpan (scope.py ~line 1350).
  • The default sampled=False is stored as self._sampled (line 634 of traces.py).
  • stdlib.py passes the ignored HTTP span explicitly to iter_trace_propagation_headers(span=span) (stdlib.py ~line 160).
  • iter_trace_propagation_headers calls span._iter_headers() because NoOpStreamedSpan is not an instance of NoOpSpan and has_tracing_enabled is True (scope.py ~line 703).
  • Inside _iter_headers(), self._to_traceparent() uses self.sampled = False → appends -0, but self._segment._get_baggage() uses the real sampled StreamedSpan segment whose sampled is True → baggage includes sentry-sampled=true.

Identified by Warden code-review · RXZ-PUR

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will verify but we might need to take the segment's sampled flag into account here, instead of the child. It used to not make a difference, but now it does.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 46b410a. Configure here.

Comment thread sentry_sdk/scope.py
trace_id=propagation_context.trace_id,
parent_span_id=propagation_context.parent_span_id,
parent_sampled=propagation_context.parent_sampled,
baggage=propagation_context.baggage,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignored spans wrong trace propagation

Medium Severity

Segment-level ignored spans create an active NoOpStreamedSpan without a propagation sampled value, so it defaults to False. Outgoing headers now come from that noop via _to_traceparent, which uses sampled, not parent_sampled. Downstream traces can be marked unsampled even when the incoming trace was sampled or should stay deferred.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 46b410a. Configure here.

@sentrivana sentrivana marked this pull request as draft July 6, 2026 12:09
@sentrivana

Copy link
Copy Markdown
Contributor Author

Converting to draft to address bot comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sampling decision is deferred when it should be unsampled

1 participant