Detect scope and continuation leaks in instrumentation tests

[amarziali]

What Does This Do

Adds default-on scope and continuation lifecycle diagnostics to instrumentation tests.

The diagnostic records continuation and scope events during each test, waits briefly for asynchronous cleanup, and fails with an actionable timeline when it detects problems such as:

• A continuation that was never released
• A scope that was never closed
• A continuation resolved more than once
• Activation after resolution
• Cleanup performed from an unexpected thread
• Cleanup occurring after the test boundary

The timeline includes elapsed time, thread transitions, ownership state, and relevant call sites so the missing lifecycle edge can be identified without reconstructing it from a trace timeout.

  flowchart LR
      A[Capture continuation] --> B[Resume context]
      B --> C[Run asynchronous work]
      C --> D[Close scope]
      D --> E[Release continuation]
      E --> F[Write complete trace]

      C -. missing cleanup .-> G[Test ends]
      G --> H[Diagnostic failure]
      H --> I[Timeline with thread and callsite]

      style E fill:#d5f5e3,stroke:#239b56
      style H fill:#fadbd8,stroke:#c0392b

Loading

How is it enabled?

The diagnostic runs automatically for both instrumentation-test harnesses:

• Spock tests extending InstrumentationSpecification
• JUnit tests extending AbstractInstrumentationTest

A narrow opt-out is available only for tests that are incompatible with the diagnostic itself:

@TrackScopeContinuations(
enabled = false,
reason = "Explain the incompatibility and removal condition")

The reason is mandatory. The opt-out is not intended for unresolved continuation leaks.

Additional behavior

• Allows a bounded quiescence period for legitimate asynchronous cleanup.
• Recognizes iteration scopes registered for deferred cleanup without ignoring unrelated iteration-scope leaks.
• Keeps recording isolated to the test harness.
• Preserves an original test failure while attaching diagnostic failures as additional evidence.
• Includes focused tests for lifecycle classification, reporting, configuration, stack filtering, and concurrent snapshots.
• Adds a troubleshooting skill that guides an LLM from the emitted timeline to the broken ownership edge and a compact human-readable diagram.

Motivation

Strict trace writing can reveal that asynchronous work is still pending, but it does not explain where ownership was lost.

This diagnostic connects capture, activation, scope closure, and continuation resolution into one timeline, making continuation failures reproducible and actionable for both humans and automated integration work.

Fixed issues leveraging that:

• Fix continuation cleanup when TimerTask is cancelled #12464
• Prevent duplicate Kotlin coroutine continuation release #12466
• Prevent OkHttp TaskRunner from retaining request contexts #12468
• Prevent Pekko stream lifecycle context retention #12470
• Prevent OkHttp connection pools from retaining request contexts #12492
• Fix Java HTTP async continuation lifecycle #12498
• Prevent Netty periodic purge tasks from inheriting context #12499
• Close ServiceTalk captured context scope in test #12500
• Wait for Reactor Kafka test pipelines to finish #12501
• Test async RUM timer propagation as application code #12502
• Prevent Netty idle timers from retaining request context #12503
• Use context swap for Akka and Pekko HTTP flows #12507
• Avoid abandoning Guidewire test continuations #12518
• Prevent Couchbase Netty initialization from capturing context #12520
• Avoid duplicate context capture in OpenSearch executors #12521
• Fix Hystrix async error tests without relaxing trace validation #12570

Additional Notes

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	13.96 s	[-0.5%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent	12.92 s	13.03 s	[-1.5%; -0.3%] (maybe better)
startup:petclinic:appsec:Agent	17.09 s	16.84 s	[-4.5%; +7.4%] (unstable)
startup:petclinic:iast:Agent	17.40 s	17.42 s	[-0.9%; +0.7%] (no difference)
startup:petclinic:profiling:Agent	17.20 s	17.36 s	[-2.0%; +0.2%] (no difference)
startup:petclinic:sca:Agent	17.04 s	17.36 s	[-6.3%; +2.7%] (no difference)
startup:petclinic:tracing:Agent	16.20 s	16.78 s	[-7.6%; +0.7%] (no difference)

Commit: 8411dfdd · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[pr-commenter]

Kafka / consumer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/diag2
git_commit_date	1790036134	1790261323
git_commit_sha	dd9d42a	a96fbd8

See matching parameters

	Baseline	Candidate
ci_job_date	1790262492	1790262492
ci_job_id	2076863061	2076863061
ci_pipeline_id	139852046	139852046
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-o0p89yof 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-o0p89yof 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

scenario	Δ mean throughput
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume	worse [-13804.396op/s; -5026.515op/s] or [-4.515%; -1.644%]

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume	same

[pr-commenter]

Kafka / producer-benchmark

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	andrea.marziali/diag2
git_commit_date	1790349329	1790349388
git_commit_sha	220ff6c	8411dfd

See matching parameters

	Baseline	Candidate
ci_job_date	1790350793	1790350793
ci_job_id	2081182196	2081182196
ci_pipeline_id	140181185	140181185
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
jdkVersion	11.0.31	11.0.31
jmhVersion	1.36	1.36
jvm	/usr/lib/jvm/java-11-openjdk-amd64/bin/java	/usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs	-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=15002 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=15002 -Dhttp.nonProxyHosts=localhost	*.localhost
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-3d9ikjhq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-3d9ikjhq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
vmName	OpenJDK 64-Bit Server VM	OpenJDK 64-Bit Server VM
vmVersion	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu	11.0.31+11-post-1ubuntu1-22.04.2-Ubuntu

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results

scenario	Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce	same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce	same

[dd-octo-sts]

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

• Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-22T08:09:17.044780Z	13ee8f8	Manual request
🔒 Security Review	✅ Completed 2026-09-22T08:06:28.421867Z	13ee8f8	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c034c2c2b7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[amarziali]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-24 15:01:37 UTC ℹ️ Start processing command /code blockers

---

2026-09-24 15:01:39 UTC ℹ️ Devflow:

Checking merge blockers for #12458...

---

2026-09-24 15:01:44 UTC ℹ️ Devflow: /code blockers

Detected 2 merge blocker(s) to address:

🔴 Blocking

• Required check Check system tests success has not reported
  Hint: Ensure the workflow that produces this check is triggered.

🟠 Pending

• Required check dd-gitlab/default-pipeline is running

[amarziali]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-25 07:43:45 UTC ℹ️ Start processing command /code blockers

---

2026-09-25 07:43:46 UTC ℹ️ Devflow:

Checking merge blockers for #12458...

---

2026-09-25 07:43:55 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🔴 Blocking

• Required check dd-gitlab/default-pipeline failed
  Hint: Inspect the failure and re-run it once fixed.

[amarziali]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-09-25 17:06:03 UTC ℹ️ Start processing command /code blockers

---

2026-09-25 17:06:04 UTC ℹ️ Devflow:

Checking merge blockers for #12458...

---

2026-09-25 17:06:13 UTC ℹ️ Devflow: /code blockers

No merge blockers detected.