Skip to content

fix: bound profiler memory growth by trimming interned call stacks - #5503

Merged
jamescrosswell merged 20 commits into
mainfrom
fix/5469-trim-live-session-state
Aug 31, 2026
Merged

fix: bound profiler memory growth by trimming interned call stacks#5503
jamescrosswell merged 20 commits into
mainfrom
fix/5469-trim-live-session-state

Conversation

@jamescrosswell

@jamescrosswell jamescrosswell commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #5502 to keep the mechanical changes for the submodule bump independent of the fix itself. This PR targets deps/update-perfview and is just the fix.

Fixes #5469

What was leaking

TraceCallStacks.InternCallStackIndex interns every distinct call stack it observes and never
releases any of them. That is correct for a trace file, which is finite, but the profiler's
EventPipe session runs for the life of the process — so the tables grow forever. Two properties made
it worse than it sounds: growth tracks the diversity of stack shapes rather than sample volume,
and it happens whether or not a profile is running, which is why lowering ProfilesSampleRate
never helped and only setting it to 0 did.

The fix

TraceLog.TrimLiveSessionState() (microsoft/perfview#2452, contributed for this) discards those tables. This wires it up from an always-on ThreadSample handler on the factory.

A trim reissues CallStackIndex from zero, and SampleProfileBuilder caches against it, so a stale entry would resolve a sample to an unrelated stack — wrong frames, no exception.

So the cache must be invalidated. SampleProfilerSession carries a generation that increments on every trim; SampleProfileBuilder records what it saw and drops _stackIndexes when the generation increments. Affected stacks get walked again, at worst appearing twice in Profile.Stacks, which samples reference by the builder's own ids. Potentially we spend a bit more CPU then - but we avoid the memory leak.

Both run on the dispatch thread, so ordering can't bite: trim-then-AddSample sees the bump and
clears, AddSample-then-trim resolved against tables that were valid at that instant and sees the
bump next time.

The builder's other caches need no handling: _frameIndexesByCodeAddressIndex and
_frameIndexesByMethodIndex key off tables a trim doesn't touch, and _threadIndexes keys off
TraceLog.Threads, which is a different table from the per-thread roots inside TraceCallStacks.

Trim Frequency

MaxCallStackCount is 100,000 (~10 MB of tables) - roughly every 7 minutes in the reporting customer's scenarion, and a trim measured at ~0.012 MiB of unreturned RSS. It is an internal static rather than a SentryOption - we could expose this if there was a compelling reason but I think best to leave internal initially.

Why not just restart the session

That was the obvious alternative and it was measured and rejected: it bounds managed memory but costs
3.1–3.3 MiB of RSS per recycle that is never returned, making RSS grow roughly 3x faster than the
leak. Trimming in place costs ~0.012 MiB, because no new EventPipe session is created.

Effect

Same workload, 600 s, real time session on .NET 9 in a Linux container:

without trim with trim
interned call stacks at 600 s 1,231,862, still climbing bounded, cycling 919 – 7,121
managed heap 3.4 → 119.2 MiB 3.5 → ~7 MiB, flat
RSS 65.2 → 200.7 MiB 65.6 → 81.4 MiB
samples processed 307,418 340,987

Throughput went slightly up, since the baseline spends real time growing and collecting those
tables. That workload is deliberately far more stack-diverse than a real service so the effect is
measurable in minutes — the shape matches the field report, the rate is exaggerated.

Test

Profiler_WhenNoProfileRunning_TrimsInternedCallStacks covers the idle path — no profile is ever
started, so the trim has to come from ordinary samples.

Profiler_WhileProfileRunning_StillTrimsInternedCallStacks covers the starvation case: it holds a
profile open for the whole test, asserts trimming still happens, then collects and validates the
profile to show a mid-profile trim doesn't corrupt its output. Both were mutation-tested — restoring
the old _inProgress guard makes the second fail with "No trim occurred while a profile was running".

Sentry.Profiling.Tests on net10.0: 19 passed, 0 failed, 3 skipped. net8.0/net9.0 hosts can't be
launched locally (no arm64 runtimes for those on this machine), so CI covers those.

Changelog Entry

fix: Memory leak in Sentry.Profiling due to EventLog interning tables growing indefinitely

jamescrosswell and others added 2 commits August 24, 2026 10:30
Bumps the perfview submodule from e343a0cf (v3.1.15-5) to 9c4f637c (v3.2.6-3),
227 commits. That includes microsoft/perfview#2452, which adds
TraceLog.TrimLiveSessionState() - the API needed to bound the call stack
interning growth reported in #5469. Wiring it up is a separate change.

Also removes the committed sample.etlx test fixture. Its ETLX format version
is tied to the TraceEvent build (the new FastSerialization accepts >= 78, the
committed copy was 74), so it goes stale on every submodule bump.
TraceLogProcessorTests regenerates it from sample.nettrace when absent, so it
is gitignored instead of committed.

The verified snapshots change accordingly: module names lose a spurious ".il"
suffix (System.Private.CoreLib.il -> System.Private.CoreLib). That is the only
difference across both snapshots apart from a trailing newline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TraceLog interns every distinct call stack it observes and never releases them,
so the continuous EventPipe session behind transaction profiling grew without
bound - roughly 0.6 GB/day to OOM under production traffic on Linux.

Uses TraceLog.TrimLiveSessionState() (microsoft/perfview#2452) to discard those
tables once they pass a budget. The trim runs from an always-on ThreadSample
handler on the factory, which matters for two reasons: it is the session's event
processing thread, which is the only thread TrimLiveSessionState permits, and
the tables grow whether or not a profile is running, so trimming only at profile
boundaries would leave the growth unchecked whenever traffic is sparse.

Trimming is gated on no profile being in progress AND the previous profile
having stopped consuming samples. The second condition is not redundant:
_inProgress is cleared when a transaction finishes, but that profiler stays
subscribed and keeps resolving samples up to its end timestamp - roughly two
seconds later, because TraceLog dispatches on a delay. Trimming inside that
window would leave SampleProfileBuilder's CallStackIndex-keyed cache resolving
to unrelated stacks. Passing the end timestamp to OnFinish lets the handler tell
when that drain has finished.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 7.69231% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.67%. Comparing base (4e0e0de) to head (908f26f).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
...ry.Profiling/SamplingTransactionProfilerFactory.cs 14.28% 12 Missing ⚠️
src/Sentry.Profiling/SampleProfileBuilder.cs 0.00% 6 Missing and 1 partial ⚠️
src/Sentry.Profiling/SampleProfilerSession.cs 0.00% 4 Missing ⚠️
...rc/Sentry.Profiling/SamplingTransactionProfiler.cs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5503      +/-   ##
==========================================
- Coverage   74.74%   74.67%   -0.07%     
==========================================
  Files         513      515       +2     
  Lines       18829    18909      +80     
  Branches     3682     3691       +9     
==========================================
+ Hits        14074    14121      +47     
- Misses       3875     3904      +29     
- Partials      880      884       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/Sentry.Profiling/SamplingTransactionProfiler.cs Outdated
Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
jamescrosswell and others added 10 commits August 24, 2026 16:55
Co-authored-by: James Crosswell <jamescrosswell@users.noreply.github.com>
Removing the committed sample.etlx puts this reflection path on the critical
path for every clean checkout, where before it never ran. CreateFromEventPipeEventSources
is non-public API in the perfview submodule, so a bump can move it; the
null-conditional call would then no-op and surface as a confusing file-not-found
on the TraceLog constructor instead of naming the real cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CreateFromEventPipeEventSources does not take ownership of the source - upstream's
own TraceLog.CreateFromEventPipeDataFile wraps it in a using for the same reason.
Removing the committed sample.etlx means this path now runs on every clean checkout,
so the handle was actually being leaked rather than sitting in dead code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Generating it on demand raced across target frameworks: _resourcesPath resolves to
the shared source Resources directory for every TFM, and dotnet test runs one host
per TFM, so they collided on both sample.etlx and the sample.etlx.new temp that
TraceEvent writes alongside it. That already failed on .NET (win-arm64) with
"the process cannot access the file ... because it is being used by another process".

Committing it regenerated at the current format version keeps the tests deterministic
and takes that whole class of failure off the table. README documents how to
regenerate after a future submodule bump, which was the original reason for not
committing it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
Gating the trim on "no profile running, and the previous one has drained" could
starve indefinitely. Profiles run back to back under sustained load - a new one
starts as soon as the previous clears _inProgress - so the window where trimming
was permitted could simply never coincide with a dispatched sample. The tables
then grow unbounded, in exactly the workload where that matters most.

The guards were conservative because a trim reissues CallStackIndex from zero
and SampleProfileBuilder caches against it. Rather than avoid that, invalidate
it: SampleProfilerSession carries a generation that increments on every trim,
and the builder drops _stackIndexes when it sees the generation move. Its other
caches are unaffected, since a trim touches neither the code address and method
tables nor TraceLog.Threads.

That makes trimming safe at any moment, so both guards, _lastProfileEndTimeMs
and the timestamp on OnFinish all go away.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/Sentry.Profiling/SampleProfileBuilder.cs Outdated
Comment thread src/Sentry.Profiling/SampleProfilerSession.cs Outdated
Comment thread src/Sentry.Profiling/SampleProfilerSession.cs Outdated
Co-authored-by: James Crosswell <jamescrosswell@users.noreply.github.com>
@jamescrosswell
jamescrosswell marked this pull request as ready for review August 25, 2026 05:12
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Aug 25, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 21be80c. Configure here.

Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs
jamescrosswell and others added 3 commits August 25, 2026 20:27
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both trim tests exited their work loop at the first trim, which made them
timing-dependent in opposite ways.

The idle test then compared a high-water mark sampled on the test thread against
a count read later, while the dispatch thread kept interning - so with a tiny
budget the "after" reading could exceed the "peak". It now runs for a fixed
duration and asserts the property that actually matters: the table stays bounded.

The in-profile test finished the profile milliseconds after starting it, so no
samples had been dispatched yet and ValidateProfile found none. It now runs for a
fixed duration too, which is also what makes collecting and validating the profile
meaningful as a check that a mid-profile trim doesn't corrupt output.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Base automatically changed from deps/update-perfview to main August 25, 2026 22:56
@jamescrosswell
jamescrosswell requested a review from vaind August 26, 2026 00:55
#5502 was squash merged, so the bump content now arrives from main rather than
from this branch's commits. #5470 also landed, which reworked the factory - it
introduced its own lock-guarded _session, so the volatile field added here is
dropped in favour of it, and the trim handler captures the session from the
startup closure rather than reading that field off the dispatch thread.

@vaind vaind left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach LGTM, some minor things below.

Comment thread src/Sentry.Profiling/SamplingTransactionProfilerFactory.cs Outdated
// The interning tables grow whether or not a profile is running, so this deliberately never
// starts one - the trim has to happen off the back of ordinary samples. See #5469.
var originalMax = SamplingTransactionProfilerFactory.MaxCallStackCount;
SamplingTransactionProfilerFactory.MaxCallStackCount = 100;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 says

SamplingTransactionProfilerTests carries [CollectionDefinition(..., DisableParallelization = true)] on itself, which defines a collection but doesn't join it — xUnit needs [Collection(name)] on the class. Tests inside one class serialize regardless, so the two new tests can't collide with each other. But ProfilingSentryOptionsExtensionsTests runs in a separate collection, and HubDispose_* starts real EventPipe sessions — those will run with a budget of 100 during the 3-second windows. Harmless today (they only assert IsDisposed), but it's a trap for the next test that asserts on profile content. An instance field on the factory instead of a static removes the global mutation and both try/finally blocks.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch... fixed (partially) in 908f26f

Also created:

That might explain why we've had issues with flaky profiling tests (locally and in CI).

If TrimLiveSessionState throws, the interning tables stay over budget, so every
subsequent sample re-enters the try, throws again and logs again. There is no
state change to break the cycle - it is a permanent hot loop on the event
processing thread, not a transient burst. Measured without the latch: 32,846
attempts in 3 seconds, each one an exception throw and a log write.

Logged at Error rather than Warning: this is not a condition we anticipate, and
it leaves profiling memory growth unbounded for the rest of the session.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tests were mutating SamplingTransactionProfilerFactory.MaxCallStackCount globally
and restoring it in a finally. That leaks across test classes: the
[CollectionDefinition] on SamplingTransactionProfilerTests only defines a
collection, it never joins one, so the class keeps xUnit's default of its own
implicit collection and runs in parallel with ProfilingSentryOptionsExtensionsTests
- whose HubDispose_* tests start real EventPipe sessions and would see a budget of
100 during these tests' three second windows.

Harmless today, since those tests only assert on disposal, but a trap for the next
test that asserts on profile content. An instance field removes the shared state
and two of the three try/finally blocks; the remaining one still resets the
OnTrimForTests hook, which is genuinely static.

The parallelization problem itself is filed separately as #5524.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/Sentry.Profiling/SampleProfileBuilder.cs
@jamescrosswell
jamescrosswell merged commit 6e31ca3 into main Aug 31, 2026
42 checks passed
@jamescrosswell
jamescrosswell deleted the fix/5469-trim-live-session-state branch August 31, 2026 11:14
plz12345 added a commit to Whisparr/Whisparr-Eros that referenced this pull request Sep 4, 2026
Updated
[Selenium.WebDriver.ChromeDriver](https://github.com/jsakamoto/nupkg-selenium-webdriver-chromedriver/)
from 152.0.7977.7500 to 152.0.7977.8200.

<details>
<summary>Release notes</summary>

_Sourced from [Selenium.WebDriver.ChromeDriver's
releases](https://github.com/jsakamoto/nupkg-selenium-webdriver-chromedriver//releases)._

No release notes found for this version range.

Commits viewable in [compare
view](https://github.com/jsakamoto/nupkg-selenium-webdriver-chromedriver//commits).
</details>

Updated [Sentry](https://github.com/getsentry/sentry-dotnet) from 6.9.0
to 6.10.0.

<details>
<summary>Release notes</summary>

_Sourced from [Sentry's
releases](https://github.com/getsentry/sentry-dotnet/releases)._

## 6.10.0

### Features ✨

- feat: Logs sent via `SentrySdk.Logger` no longer require `EnableLogs`
by @​jamescrosswell in
[#​5512](getsentry/sentry-dotnet#5512)
- feat: `SentryOptions.EnableMetrics` is obsolete and ignored by
@​jamescrosswell in
[#​5509](getsentry/sentry-dotnet#5509)

### Fixes 🐛

- fix: Prevent managed exceptions from leaking as NSExceptions,
resulting in duplicate exception capture on iOS by @​jpnurmi in
[#​5525](getsentry/sentry-dotnet#5525)
- fix(profiling): release the EventPipe session when the SDK shuts down
by @​jamescrosswell in
[#​5470](getsentry/sentry-dotnet#5470)
- fix: Memory leak in Sentry.Profiling due to EventLog interning tables
growing indefinitely by @​jamescrosswell in
[#​5503](getsentry/sentry-dotnet#5503)
- fix: Attachments not being sent properly when Spotlight is enabled by
@​XAN9xXx in
[#​5511](getsentry/sentry-dotnet#5511)
- fix: Heap dump files are now deleted from disk once they have been
sent to Sentry by @​XAN9xXx in
[#​5481](getsentry/sentry-dotnet#5481)
- fix: populate sentry.sdk.name and sentry.sdk.version for console apps
by @​zkasuran in
[#​5483](getsentry/sentry-dotnet#5483)

### Dependencies ⬆️

#### Deps

- chore(deps): update Java SDK to v8.54.0 by @​github-actions in
[#​5517](getsentry/sentry-dotnet#5517)
- chore(deps): update Cocoa SDK to v9.26.1 by @​github-actions in
[#​5516](getsentry/sentry-dotnet#5516)
- chore(deps): update CLI to v3.7.0 by @​github-actions in
[#​5520](getsentry/sentry-dotnet#5520)
- chore(deps): update Native SDK to v0.16.4 by @​github-actions in
[#​5508](getsentry/sentry-dotnet#5508)
- chore(deps): update Java SDK to v8.53.0 by @​github-actions in
[#​5484](getsentry/sentry-dotnet#5484)

### Other

- deps: update perfview (removes the .il suffix from profile module
names) by @​jamescrosswell in
[#​5502](getsentry/sentry-dotnet#5502)

Commits viewable in [compare
view](getsentry/sentry-dotnet@6.9.0...6.10.0).
</details>

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak with Sentry.Profiling 6.6.0 on Linux: TraceEvent call-stack/method index tables accumulate unboundedly under traffic (~0.6 GB/day to OOM)

2 participants