Conversation
Specifically:
* Median is reported, in addition to the existing mean+stdev, which is
significantly more resistant to skew by outliers.
* --metric {wall,cpu} (default wall): Enables profiling using CPU time
rather than wall-clock time. CPU profiling has roughly half the coefficient
of variation as wall-clock profiling equal run count.
* --workers1: Forces MYPY_NUM_WORKERS=1 (rather than the default 4) to
cut CPU scheduling variance. Strongly recommended when using --metric cpu.
* --warmup-runs N (default 1): Configurable number of leading cold runs to discard.
Previously was always 1. Higher run counts decrease outliers that skew
the reported mean.
* A new "Paired deltas vs <first commit>" section is added to the report,
showing per-round paired differencing against the first commit
to cancel round-level common-mode noise, reducing variance.
Reported as median +/-95% CI.
Also:
* --cache-binaries (default false): Caches each commit's compiled clone
to avoid ~5min recompile whenever comparing the same commit multiple times.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_parse_as_type_expression() Specifically: - If you set MYPY_TYPEFORM_PROFILE_FULL_PARSE environment variable, mypy will output a .tsv to that filepath which characterizes the kinds of Expressions that try_parse_as_type_expression() in semanal.py was forced to do a full parse of, which was not rejected early. - A misc/analyze_typeform_full_parse_profile.py script is added which takes those .tsvs and prints an expression-time summary (by total time) plus top-N descriptors per FAIL class. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…s_type_expression() Add a fast-rejection filter to SemanticAnalyzer.try_parse_as_type_expression(): a string literal that is an identifier naming a Var whose declared type is a concrete Instance (and is not a typing special form) is a value -- a local, parameter, or module-level constant -- never a type expression. Reject it before the expensive full-parse block (expr_to_analyzed_type + isolated_error_analysis). On the mypy self-check this filter rejects 157 of the 381 identifier-string literals that currently reach the full-parse block (e.g. "__doc__", "__name__", enum/constant members like "ROUND_DOWN", "GEN_CREATED"), all of which were failing full parses -- pure wasted work. Insertion point chosen empirically. The filter is placed AFTER the existing PlaceholderNode and unbound-tvar checks rather than before them. Its only expensive conjunct (get_proper_type(node.type)) runs solely for Var nodes, and all Var nodes already survive both earlier checks, so position cannot change how often the expensive part runs -- only how often the cheap isinstance(node, Var) conjunct is evaluated. Because 951 of the identifier-strings reaching this block are unbound type variables, evaluating the filter before the tvar check would force an extra isinstance onto ~951 nodes it can never catch. perf_compare.py (--metric cpu --workers1 --num-runs 100) confirms: paired median vs baseline was -15.6ms +/-4.6 here, vs -12.9ms (before placeholder) and -9.3ms (before tvar) -- matching the eval-count model's predicted ordering. Also add typing.Self / typing_extensions.Self to var_is_typing_special_form(). Self is a _SpecialForm-typed Var, so without this guard the new filter would wrongly reject a stringified "Self" type annotation (regressing testSelfRecognizedInOtherSyntacticLocations). This guard is a correctness prerequisite of the filter and is committed together with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…frequency
Move the unbound-type-variable check ahead of the PlaceholderNode check (and
ahead of the Var-value filter added in the previous commit) in
SemanticAnalyzer.try_parse_as_type_expression(). Final order:
unbound type variable -> value Var -> placeholder
These three checks are mutually exclusive -- a node is at most one of a
TypeVarExpr/ParamSpecExpr, a Var, or a PlaceholderNode -- so reordering cannot
change which expressions are rejected (verified: check-typeform and testsemanal
unchanged). Ordering them by descending rejection frequency, as measured on
mypy's self-check (unbound type vars ~951 >> value Vars ~157 > placeholders
~23), lets the commonest rejections exit first and minimizes total check
evaluations (~2700 -> ~1750 cheap isinstance calls over the self-check).
The win is below perf_compare's noise floor on its own (~10us), but the
reordering is free and behavior-preserving, and it makes the final ordering
self-documenting. A rationale comment is added at the head of the block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ression()
Add a second fast-rejection filter: a string literal that is an identifier
naming a FuncDef, OverloadedFuncDef, or MypyFile is a function or module, never
a type expression. Reject it before the expensive full-parse block.
On mypy's self-check this rejects 48 of the identifier-string literals that
reach the full-parse block (e.g. builtin functions like "classmethod",
"staticmethod", "hash"; user functions; module names like "platform"), all of
which were failing full parses.
Unlike the Var-value filter, this check is a single isinstance with no expensive
follow-on work, and FuncDef/OverloadedFuncDef/MypyFile are mutually exclusive
with the other early-reject node kinds, so it is freely positionable and its
rejection count is order-independent. It is placed by descending rejection
frequency: after the Var-value filter (~157) and before the placeholder check
(~23), i.e. the final order is
unbound type variable (~951) -> value Var (~157)
-> function/module (~48) -> placeholder (~23)
No companion guard is needed (a function or module name is never a valid type,
so nothing valid is rejected; check-typeform and testsemanal unchanged).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Contributor
Author
|
This PR is ready for review. (Disregard if you already saw a notification when this PR was opened. I'm commenting explicitly just in case GH issue notifications weren't working at the time this PR was opened.) |
Contributor
Author
ilevkivskyi
pushed a commit
that referenced
this pull request
Sep 18, 2026
References #21262. Replaces #21585 and obsoletes #21596. ## Summary Enabling `TypeForm` by default (referenced #21262) made `SemanticAnalyzer.try_parse_as_type_expression` run eagerly on every expression in certain syntactic positions. The cost is concentrated in the expensive full-parse block (`expr_to_analyzed_type` + `isolated_error_analysis`), which **fails ~87% of the time** - pure wasted work. This branch adds early-reject filters that eliminate **74% of full parses (2570 → 666)** on mypy's self-check, recovering **~46% of the regression**: **+1.57% → +0.84%** CPU time. **No new regexes** - per review feedback on replaced #21585. Every filter here is plain string/`isinstance` work, and the two shape tests that were regexes are now helper functions. <details> <summary>Why not do a type-context check?</summary> Review of replaced #21585 suggested skipping the call to `SemanticAnalyzer.try_parse_as_type_expression` entirely when the type context cannot be a `TypeForm`. That optimization already exists, but in the other *type checker* pass at `ExpressionChecker.try_parse_as_type_expression`. The same skip cannot be used in the *semantic analyzer* pass's function because the type context is not yet known. So cheaply filtering the inputs to `SA.try_parse_as_type_expression` is the only remaining (obvious) lever to reduce its runtime contribution. </details> ## Optimization Results CPU time, single worker, paired per-round deltas, n=300: ``` python misc/perf_compare.py --warmup-runs 3 --num-runs 300 -j 3 \ --metric cpu --workers1 \ <TypeForm-disabled-commit> 5bb72b7 <tip-of-this-pr-branch> ``` | Commit | Mean | Median | Δ vs baseline (paired median ±95% CI) | |---|---:|---:|---:| | baseline, `<TypeForm-disabled-commit>` | 2.679 s | 2.676 s | - | | master, `5bb72b788` | 2.720 s | 2.720 s | **+41.9 ms ±2.9 (+1.57%)** | | all filters, `<tip-of-this-pr-branch>` | 2.703 s | 2.699 s | **+22.4 ms ±2.9 (+0.84%)** | The feature branch recovers **19.5 ms of the 41.9 ms regression (~46% by paired median)** - leaving **+22.4 ms (~54%)**. Derivation: - +41.9 ms - +22.4 ms == 19.5 ms recovered - 19.5 ms / +41.9 ms == 46.5% (~46%) recovered - +22.4 ms / +41.9 ms == 53.5% (~54%) left A separate 2-way run of master vs `<tip-of-this-pr-branch>` measured **−21.1 ms ±3.2**, consistent with the 19.5 ms recovered that was derived above. <details> <summary>Notes on the measurement</summary> The baseline (`<TypeForm-disabled-commit>`) is current master (`5bb72b788`) with referenced #21262 (SHA: `dd851f559`) reverted, so all three arms share today's code and differ only in `TypeForm`. Measuring against the original pre-#21262 master commit instead of today's master would have conflated optimizations made during the following ~80 commits, including notably `c0cced35c`, which optimised `SA.try_parse_as_type_expression` specifically. Thus runtime regression measured here (+41.9 ms) is smaller than the +50.2 ms reported in replaced #21585: part of the original regression has already been absorbed upstream. </details> Full parses per self-check, identical corpus: | | master | branch | | |---|---:|---:|---| | full parses | 2570 | 666 | **−74.1%** | | - succeeded (produced a type) | 345 | 345 | **±0** ✅ | | - failed (wasted work) | 2225 | 321 | **−85.6%** | The successful-parse count is unchanged at every commit on the branch, as expected: No expression that previously parsed as a type stopped doing so. ## Overview of changes - Most changes are made to the `SemanticAnalyzer.try_parse_as_type_expression` function. All other changes occur within the same file. - 5 commits, each individually profiled: - 4 commits add a filter - 1 commit reorders existing filters - Any added filter can be dropped (if needed) without disturbing the other filters ### The filter commits Bare-identifier strings (`"Foo"`): 1. Reject a `Var` whose declared type is a concrete `Instance` - a value, not a type. 2. Reject `FuncDef` / `OverloadedFuncDef` / `MypyFile` - functions and modules are never types. 3. Reorder the mutually-exclusive checks by measured rejection frequency. Other strings: 4. Reject strings containing a character or boundary pattern that never appears in a type expression - leading/trailing `.`, or one of ``!:/<>@%$^?;&~`\``, or a `-` that is not a `Literal[...]` unary minus. Catches `"utf-8"`, `".pyi"`, `"error:"`, `"pkg/mod.py"`. 5. Dotted-name strings (`"builtins.tuple"`, `"typing.Mapping"`): look up the leftmost component and reject when it does not resolve, or resolves to a placeholder or a value `Var`. Filters 4 and 5 replace `_NONTYPE_PATTERN_RE` and `_DOTTED_IDENTIFIER_RE` from replaced #21585 with the helpers `has_nontype_char()` and `dotted_identifier_leftmost()`. Each was verified to agree with the regex it replaces on all 1171 distinct strings the full-parse profiler observes during a self-check. <details> <summary>Two specific hazards, and how they are handled</summary> `var_is_typing_special_form` was extended to recognize `typing.Self` / `typing_extensions.Self`, so filter 1 does not reject a stringified `'Self'` annotation (otherwise `testSelfRecognizedInOtherSyntacticLocations` regresses). In filter 4, `-` is treated as a unary minus wherever the preceding non-space character is `[` or `,`, so `"Literal[-1, -2]"` and `"Literal[1, -2]"` are still recognized. (`_NONTYPE_PATTERN_RE` in replaced #21585 used `(?<!\[)-`, which rejected those.) On the strings observed during a self-check the two rules reject identical sets, so the (improved) soundness costs nothing. </details> ## Notes - I don't think it's worth trying to recover the remaining +22.4 ms: - The 321 surviving failed parses are spread across four categories with no common cheap/obvious shape left - I experimented with adding some fancy `OpExpr` filters that actually gave a net *slowdown* of 1.9ms. - The profiling instrumentation and the `misc/perf_compare.py` improvements used to produce these numbers are in a separate PR: #21832. Happy to fold them in here instead if that is easier to review. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References #21262.
Includes 2 filters from the original combined 7-filter PR:
Summary
Enabling
TypeFormmadeSemanticAnalyzer.try_parse_as_type_expressionruneagerly on every expression in certain syntactic positions (~2.84M calls per
self-check). <5% of those reach an expensive full-parse block
(
expr_to_analyzed_type+isolated_error_analysis), and ~91% ofthose fail — pure wasted work.
This PR adds 2 cheap, early-reject filters for the
StrExpr-identifiercase, plus a reordering of the existing early-reject checks by decreasing
rejection frequency. Together the two filters eliminate 8% of full
parses (2,548 → 2,343) per self-check.
3 commits, all operating within the
str_value.isidentifier()branch of
try_parse_as_type_expression:isinstance(node, Var)+ more conditionsisinstance(node, (FuncDef, OverloadedFuncDef, MypyFile))Looking at the
str_value.isidentifier()branch alone,84% (244 → 39) of failing full parses are eliminated per self-check.
Performance
misc/perf_compare.py, single worker, paired per-round deltas.CPU time (canonical, lowest-variance) — master vs branch tip, n=100
Significant (CI excludes 0). This is the net effect of the branch.
Wall-clock — master vs branch tip
Borderline significant: the CI just barely includes 0. Consistent with a real
per-call win partly masked by multi-worker wall-clock.
Correctness
All tests pass.
Note that the
var_is_typing_special_formhelper needed to be extended torecognize stringified forms of
typing.Selfso that Filter A would notincorrectly reject a stringified
'Self'annotation, regressing thetestSelfRecognizedInOtherSyntacticLocationstest.Open Questions
Should the 2 infrastructure commits be moved to a separate PR?→ DoneThe full commit messages (after the subject line) on the 3 key commits→ Doneare rather verbose. Let me know if you'd like me to trim them down,
perhaps by removing everything after the subject.