Add pipeline recipe profiling scripts for optimization benchmarking#14144
Open
pzarzycki wants to merge 1 commit into
Open
Add pipeline recipe profiling scripts for optimization benchmarking#14144pzarzycki wants to merge 1 commit into
pzarzycki wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds new profiling utilities under examples/profiling to benchmark end-to-end Diffusers optimization “recipes” and to emit structured JSON/Markdown reports, plus a dedicated DiffusionGemma quantized run benchmark script. It also refactors the existing profiling CLI to use a shared pipeline registry.
Changes:
- Added a recipe-based benchmarking CLI (
profile_pipeline_recipes.py) that runs pipelines under optimization combinations and writes JSON + Markdown reports. - Introduced a shared
pipeline_registry.pyand updatedprofiling_pipelines.pyto consume it. - Added DiffusionGemma quantized profiling scripts (including a small Colab runner) and documented the new workflow in
examples/profiling/README.md.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/profiling/README.md | Documents the new recipe benchmarking CLI and its supported actions/output. |
| examples/profiling/profiling_pipelines.py | Switches pipeline config construction to the shared registry via import fallback logic. |
| examples/profiling/profile_pipeline_recipes.py | New end-to-end recipe benchmarking runner with JSON/Markdown report output. |
| examples/profiling/profile_diffusion_gemma_quantized.py | New benchmarking script for (optionally) 4-bit quantized DiffusionGemma generation runs. |
| examples/profiling/profile_diffusion_gemma_quantized_colab_run.py | Minimal Colab helper to invoke the quantized DiffusionGemma profiler. |
| examples/profiling/pipeline_registry.py | New shared registry of profiling pipeline configs used by the profiling CLIs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+102
to
+126
| def benchmark_pipeline_call(pipe, call_kwargs, num_runs: int, num_warmups: int) -> tuple[float, float, float]: | ||
| for _ in range(num_warmups): | ||
| pipe(**call_kwargs) | ||
| torch.cuda.synchronize() | ||
|
|
||
| times = [] | ||
| peak_memories = [] | ||
|
|
||
| for _ in range(num_runs): | ||
| torch.cuda.reset_peak_memory_stats() | ||
| start = torch.cuda.Event(enable_timing=True) | ||
| end = torch.cuda.Event(enable_timing=True) | ||
|
|
||
| start.record() | ||
| pipe(**call_kwargs) | ||
| end.record() | ||
| torch.cuda.synchronize() | ||
|
|
||
| times.append(start.elapsed_time(end)) | ||
| peak_memories.append(torch.cuda.max_memory_allocated() / (1024**3)) | ||
|
|
||
| mean_ms = sum(times) / len(times) | ||
| variance = sum((t - mean_ms) ** 2 for t in times) / len(times) | ||
| return mean_ms, variance**0.5, max(peak_memories) | ||
|
|
Comment on lines
+183
to
+195
| if action_name == "model_cpu_offload": | ||
| pipe.enable_model_cpu_offload() | ||
| uses_offloading = True | ||
| continue | ||
|
|
||
| if action_name == "group_offload_leaf": | ||
| pipe.enable_group_offload( | ||
| onload_device=torch.device("cuda"), | ||
| offload_device=torch.device("cpu"), | ||
| offload_type="leaf_level", | ||
| ) | ||
| uses_offloading = True | ||
| continue |
Comment on lines
+296
to
+306
| for result in payload["results"]: | ||
| lines.append( | ||
| "| {recipe} | {status} | {mean} | {std} | {peak} | {notes} |".format( | ||
| recipe=result["recipe"], | ||
| status=result["status"], | ||
| mean=result["mean_ms"] if result["mean_ms"] is not None else "—", | ||
| std=result["std_ms"] if result["std_ms"] is not None else "—", | ||
| peak=result["peak_vram_gb"] if result["peak_vram_gb"] is not None else "—", | ||
| notes=result["notes"] or "—", | ||
| ) | ||
| ) |
Comment on lines
+86
to
+88
| scheduler = BlockRefinementScheduler() | ||
| pipe = DiffusionGemmaPipeline(model=model, scheduler=scheduler, processor=processor) | ||
|
|
Comment on lines
+1
to
+9
| import runpy | ||
| import sys | ||
| sys.argv = [ | ||
| "profile_diffusion_gemma_quantized.py", | ||
| "--gen_length", "32", | ||
| "--num_inference_steps", "4", | ||
| "--cache_implementation", "dynamic", | ||
| ] | ||
| runpy.run_path("/content/profile_diffusion_gemma_quantized.py", run_name="__main__") |
Contributor
|
Hi @pzarzycki, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds profiling utilities under
examples/profilingfor benchmarking end-to-end Diffusers optimization recipes and emitting structured JSON/Markdown reports.The goal is to make memory/performance tradeoffs measurable for real pipelines, instead of requiring users to manually guess which optimization combination works best on their hardware.
Added
examples/profiling/profile_pipeline_recipes.pyexamples/profiling/pipeline_registry.pyexamples/profiling/profile_diffusion_gemma_quantized.pyexamples/profiling/README.mdWhat it measures
The profiler reports:
Supported actions include:
torch.compileValidation
Local GPU
Tested on RTX 3080 Ti 12GB with real image-generation pipelines.
runwayml/stable-diffusion-v1-52.09s,3.18 GBmodel_cpu_offload:3.20s,2.32 GBchannels_last:1.91s,3.18 GBCompVis/stable-diffusion-v1-41.00s,3.18 GBmodel_cpu_offload:1.96s,2.32 GBstabilityai/sd-turbo0.10s,3.03 GBmodel_cpu_offload:0.98s,1.71 GBThese runs show the expected tradeoff: offloading reduces VRAM at the cost of latency.
Google Colab GPU
The largest accelerator available to this account was
Tesla T4(14.56 GBVRAM).H100,A100, andL4were not available.Real Diffusion Gemma profiler results on Colab T4 with branch-installed code using
trl-internal-testing/tiny-DiffusionGemmaForBlockDiffusion:dynamic+ 4-bit:pass,wall_time_s=12.88,tokens_per_second=2.49dynamic+ full precision:pass,wall_time_s=29.81,tokens_per_second=1.07static+ 4-bit:failwith reproducibledecoder_attention_masklength mismatchdynamic+ 4-bit +compile_decoder:pass, but slower atwall_time_s=42.8On this hardware:
I also attempted
google/diffusiongemma-26B-A4B-iton Colab T4. The run reached model-load stage and began shard loading, but did not reach generation in this free unauthenticated T4 setup.