[GLM tests] Runtime-controlled B300 profiling benchmark

[hershg]

• Keeps merged [feat][tinker] Enable torch profiling on a running Tinker server #2190 as the sole trainer-profiler lifecycle, with acknowledged start/status/stop, owner-only optimizer-step advancement, overlap rejection, error-preserving cleanup, and selected-rank Megatron timers that cannot enter L1 barriers skipped by other ranks.
• Adds transport-neutral monotonic client receipts and two honest server aggregates: inference_engine_initialization_aggregate and sampler_weight_sync. Missing vLLM substages and standalone activation stay absent; distributed attribution uses rank maximum, never rank sum.
• Exposes skyrl.tinker.qualification.run_lora_qualification for the shared [GLM 5.3 Tests] Check trainer/inference LoRA logprob agreement #2182 evidence sequence, and updates the two-trainer-node 256K recipe to TP4/CP2/PP2.

Testing

uv run pytest examples/tinker/glm53/tests/test_profiles.py tests/backends/skyrl_train/utils/test_phase_timing.py -q  # 24 passed
uv run pytest examples/tinker/glm53/tests tests/backends/skyrl_train/inference_servers/test_build_vllm_cli_args.py tests/backends/skyrl_train/utils/test_phase_timing.py tests/backends/skyrl_train/utils/test_profiler.py tests/tinker/skyrl_train/test_profiling.py tests/tinker/skyrl_train/test_publication_timing.py tests/tinker/test_engine.py tests/tinker/test_model_info_tokenizer.py tests/tinker/test_profiling_receipts.py tests/tinker/test_qualification.py tests/tinker/test_server_timing.py -m 'not vllm' -q  # 250 passed, 4 deselected

Changed-file pre-commit and compileall passed. The local full Tinker collection reached 222 passed/22 skipped; its three setup errors were the CPU render fixture failing because optional local vllm was absent. Exact-head SkyRL-CPU and SkyRL-Train-CPU are green, including full pytest, engine benchmarks, code quality, gym, without-vLLM, with-vLLM, and Tinker jobs.

Qualification status

Head: 80a3d95d34519bb2bb4ea7210138f6820d19423f
Base: a7637de1fd71e1696f08cae2b9ffcacb3297b9a0

Profiling contract v10 is published and validator-clean: ready.json SHA-256 54a932acd3e4b4f38591247e5984b17881b976eb31fe704a442ec454cf086350; releases/v10/SHA256SUMS SHA-256 7e4fe907399eeb917e23784afdb2d30e80fed4019b45b0be38ae422dd35df60c. It preserves accepted v9 contract 1.3.0, both schema versions 1.1.0, #2190 dependency metadata, public endpoint/helper identities, and unchanged qualification source bytes while repinning current main/head and camera-readiness evidence. The timer and 256K topology review threads are resolved; no review threads remain open.

No image build or GPU/live campaign was run for this rebased head, by instruction. The automatic Anyscale GPU workflow failed on an empty token/invalid credentials before job submission or any product test; Vercel is an irrelevant deployment-authorization check for this Python-only PR.

Historical roughly 600-second warm-publication measurements remain old-stack context, not a current denominator. The final performance comparison is matched profiler-disabled baseline Arm A versus the selected independently numerics-qualified native candidate; profiler-enabled runs are attribution-only. Retired Arms B/D and #2195 are excluded.

[gemini-code-assist]

Code Review

This pull request introduces full-context profiling capabilities for GLM models through SkyRL's Tinker API, adding configuration profiles, server and client scripts, and comprehensive tests. It also implements a decorator to flush active torch profiler traces upon encountering a CUDA OutOfMemoryError. The review feedback suggests two important robustness improvements: safely accessing the profiler attribute using getattr in the OOM decorator to avoid masking the original error with an AttributeError, and adding a sleep delay in the client's model unloading loop when handling 408 status codes to prevent a tight busy-polling loop.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking defects identified.

Diagram

sequenceDiagram
    participant C as Tinker client
    participant A as SkyRL API
    participant P as Policy workers
    participant I as Inference engine
    C->>A: create LoRA model
    A->>P: initialize policy and start profiler
    C->>A: forward reference batch
    A->>P: forward-only scoring
    P-->>C: old-policy logprobs
    C->>A: forward_backward GSPO batch
    A->>P: forward and backward
    C->>A: optimizer step
    A->>P: apply update and advance profiler
    C->>A: publish weights
    A->>I: synchronize adapter
    C->>I: sample
    C->>A: save checkpoint and unload
    A->>P: stop profiler on runtime teardown

Loading

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit bada2c2. Configure here.}