Accept gradient iterators in global norm helpers - #8518
vineethsaivs wants to merge 1 commit into
Conversation
Materialize the input iterable once in each public helper before it is reused. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7127b87d3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Total norm of the tensors (viewed as a single vector). | ||
| """ | ||
| assert isinstance(input_tensors, Iterable), f'expected Iterable type not {type(input_tensors)}' | ||
| input_tensors = list(input_tensors) |
There was a problem hiding this comment.
This non-merge commit has no Signed-off-by: trailer in its commit message, so it violates the repository’s mandatory commit-signoff requirement and will not meet the project’s DCO/CI policy. Recreate the commit with git commit --signoff.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
1 participant
Global norm calculation consumes a gradient iterator during validation and then raises; clipping a generator cannot complete a training step.
Cause: The helper advertises Iterable input but traverses it repeatedly and assumes len and indexing. The clipping wrapper needs the tensors again after norm calculation.
Fix: Materialize the input iterable once in each public helper before it is reused.
Test:
DS_ACCELERATOR=cpu python -m pytest tests/unit/runtime/test_runtime_utils.py -k 'global_norm_accepts_iterables or clip_tensor_generator or test_call_to_str' -q. Five iterator regressions fail before. Fourteen targeted tests pass after: list/tuple/iterator norms for p=1,2,3,infinity, two SGD steps compared with PyTorch clipping, and an existing utility control. Validated on Apple Silicon CPU with PyTorch 2.10.0. No GPU or distributed run. Changed-file pre-commit passes.