Add language-agnostic emitter-diff tool by l0lawrence · Pull Request #11122 · microsoft/typespec

l0lawrence · 2026-06-30T21:41:41Z

What

eng/emitter-diff diffs the generated code produced by two versions of a TypeSpec emitter. It resolves a baseline and a head source tree (each a local folder or a GitHub url/sha), runs the emitter's own regenerate command in each, and renders the difference between the generated output as a clickable HTML report (default) plus a terminal summary.

For go/rust/languages in azure/typespec-azure you would add the language to the registry and update your package.json to reference this in the core submodule (or you can directly call the cli script (no registry needed) with the correct flags set in your package.json)

How it works

Refs (--baseline / --head): local:/path, github:owner/repo@<sha|branch>, or gh:<sha|branch> (this repo). Head defaults to the current working tree; baseline defaults to upstream/main (or origin/main).
Setup: a tree the tool fetches fresh from GitHub is prepared with the preset's --setup (install + build). The current working tree and local: paths are assumed already built and never installed by the tool. A commit-keyed cached worktree is prepared once, so reruns skip re-setup.
Baseline-output cache: baseline generated output is cached per profile + baseline commit, so iterative local runs only regenerate head. Disabled in CI with --ci. On a cache hit the baseline tree is not even checked out — its identity is resolved with a cheap shallow fetch.
Filtering a subset of tests: everything after -- is appended to the regenerate command verbatim on both sides, so you can forward the regenerate script's own filter flags (e.g. python's --name authentication/api-key --flavor azure) to diff only part of the suite.

Flags

--emitter <name> (preset: python, typescript/ts), --command, --emitter-path, --generated-code-path, --setup (repeatable) / --no-setup, --baseline / --head, --work-dir, --sequential (regenerate sides one at a time instead of in parallel), --ci, --html, --fail-on-diff, and -- <args> passthrough. See eng/emitter-diff/README.md.

Changes

eng/emitter-diff/ — the tool: CLI/orchestrator (cli.ts), ref resolver for local/github/gh: refs with a commit-keyed worktree cache (resolver.ts), emitter presets (registry.ts), a zero-dependency HTML/terminal diff renderer (diff.ts), the local baseline-output cache (baseline-cache.ts), shared types/utilities, tsconfig.json, and README.md.
packages/http-client-python/package.json — adds the diff-spector-tests script (tsx ../../eng/emitter-diff/src/cli.ts --emitter python).
.github/workflows/ci-emitter-diff-python.yml — the PR workflow (below).
cspell.yaml, .chronus/changes/… — a spelling allow-list entry and an internal changeset.

CI behavior

The python / emitter diff workflow runs on PRs that touch http-client-python or the tool. It diffs this PR's emitter against the PR merge-base (overridable via workflow_dispatch), uploads the rendered HTML report as an artifact, and posts/updates a sticky PR comment with the diff summary. It is informational: the job fails only on a tool/build error — never on a diff.

Ports the eng/emitter-diff tool: diffs generated code between two emitter versions (npm version, local folder, or github ref) and optionally runs the emitter's generated-code test suites. Ships the python adapter for @typespec/http-client-python, driving its native two-phase regenerate pipeline with a venv per emitter version. - eng/emitter-diff: @typespec/emitter-diff package (CLI, ref resolver, diff engine with HTML/VS Code/terminal output, adapter registry). - regenerate.ts: add --httpSpecsDir, --azureSpecsDir, --no-baseline flags. - pnpm-workspace.yaml: add eng/emitter-diff member + diff2html catalog entry. - ci-emitter-diff-python.yml: PR workflow that diffs head vs the PR base commit, uploads the rendered HTML artifact, and posts a sticky PR comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

CI now diffs the PR's emitter against the emitter at the commit recorded in eng/emitter-diff/baselines/python.sha and fails when generated output differs. To approve an intended change, update that SHA to a commit on the branch that contains the emitter changes and push; once the baseline matches head the diff is empty and the check passes. - cli.ts: --fail-on-diff now exits 2 for "diff present" (vs 1 for hard errors) so CI can distinguish an unapproved diff from a failure. - baselines/python.sha: the approved baseline commit. - ci-emitter-diff-python.yml: read the SHA file as baseline, run with --fail-on-diff, enforce the gate, and explain approval in the summary/comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Use a non-literal import specifier + local type so a typecheck that doesn't install this package's deps (e.g. the parent repo's check:eng, which includes core/eng) doesn't fail with TS2307; availability is validated at runtime. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

run_batch.py was always passed PLUGIN_DIR as --generated-dir, so when the emitter-diff tool redirects output via --generatedFolder the batch phase looked in the wrong tree, found no .tsp-codegen config files, and wrote zero .py files (leaving the path-bearing config files behind as spurious diffs). Pass the parent of GENERATED_FOLDER instead; it equals PLUGIN_DIR in the default case so normal regeneration is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The two sides write to isolated output dirs, use separate venvs, and emit uniquely-named temp YAML, so they can regenerate concurrently. Run them with Promise.all and tag each side's streamed output with a [baseline]/[head] prefix so the interleaved logs stay attributable. Add --sequential to opt back into one-at-a-time generation (quieter logs, lower peak CPU/memory). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Temporarily set the approved baseline to #10947 (timedelta duration encoding) so the diff PR comment and approval gate can be exercised end-to-end. Revert to the blessed HEAD SHA after the demo. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

pkg-pr-new · 2026-06-30T21:43:20Z

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-python@11122

commit: 0bb065e

github-actions · 2026-06-30T21:44:23Z

All changed packages have been documented.

✅ @typespec/http-client-python

Show changes

`@typespec/http-client-python` - internal ✏️

Add a diff-spector-tests script wired to the new language-agnostic emitter-diff tool, plus its PR CI workflow

github-actions · 2026-06-30T21:45:25Z

Python emitter diff

Baseline gh:4751383fd59eb64f1d443376c5db7948cfe974eb vs this PR.

No changes to generated output.

Rendered diff: inline on the run summary, or the emitter-diff-html artifact.

Informational check (eng/emitter-diff); does not block the PR.

A non-zero exit that is not the dedicated 'diff present' code (2) means the tool failed to run (build/venv/generate threw), not that output is unchanged. Drive the comment off the diff step's exit status so a hard error shows a clear failure notice instead of a misleading 'no changes'. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Revert the approved baseline to the blessed HEAD SHA and add a one-line comment to the generated package __init__.py so the PR's generated output differs from the baseline, exercising the emitter-diff PR comment + approval gate end to end. Both changes are temporary and must be reverted before merge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

azure-sdk-automation · 2026-06-30T21:59:11Z

You can try these changes here

🛝 Playground	🌐 Website	🛝 VSCode Extension

pnpm's recursive 'pnpm --filter <pkg> exec' collapses any non-zero child exit into its own generic exit 1 (ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL), which masked the tool's dedicated 'diff present' code (2) as a hard error (1). Run the CLI from the package directory via plain (non-recursive) 'pnpm exec' so the real exit code propagates and the approval gate can distinguish a diff from a crash. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ocs) - resolver: on the full-fetch fallback, check out a github ref by name rather than FETCH_HEAD (which could be a branch head), so a SHA baseline/head never silently diffs the wrong commit. - resolver: gh:<ref> now resolves the origin repo from the local remote instead of hard-coding microsoft/typespec, so forks/renames work. - python adapter: recognize packages/http-specs and packages/azure-http-specs layouts for external --specs, and warn (instead of silently falling back to the current checkout) when an explicit --specs dir can't be resolved. - diff: --html always writes a file (a 'No differences' stub when unchanged) and inlines the diff2html stylesheet so the CI artifact renders offline. - cli: reject an invalid --test-target instead of silently running no suites. - docs: fix the broken github-sha example, note --specs doesn't accept npm refs, soften the 'no core changes' wording, and correct the stale workflow comment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- cli.ts: absolutize --work-dir (resolve, not bare join). The python adapter runs regenerate.ts with a different cwd, so a relative work-dir made --generatedFolder/--pluginDir resolve against the wrong directory: outputs landed elsewhere while the diff engine compared empty dirs -> silent false 'no differences'. Now every path handed to an adapter is absolute. - cli.ts: reject --specs npm: before building emitters (fast-fail). - python.ts: ensureDeps() runs 'npm ci' when node_modules is missing so a fresh gh:/github: clone builds instead of failing with 'cannot find module'. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The sticky comment now shows only the one-line summary (file count, +/-) and the emitter-diff-html artifact link, not an enumerated list of changed files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ent) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Security: - Pass the untrusted workflow_dispatch baseline input (and derived values) via env vars instead of interpolating GitHub Actions expressions into run: bash (Actions expression injection) - Validate git ref/repo and use "git checkout --end-of-options" (arg injection) - Add "npm install --ignore-scripts" when materializing a baseline package - Drop the remote CDN CSS fallback so the HTML report never fetches off-box Cleanup: - De-duplicate the two HTML report templates into one htmlDoc() helper - Remove emojis from CLI output, job summary, and PR comment Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The baseline is now the git merge-base with the PR base branch instead of a pinned SHA file, so it survives squash-merge / rebase / force-push. Diff approval is granted by adding the `emitter-diff-approved` label to the PR; labeled/unlabeled events re-run the check. Removes eng/emitter-diff/baselines/python.sha. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The workflow no longer fails on a generated-output diff or gates on a label. It runs the tool without --fail-on-diff, so a diff exits 0 and only a real tool/build error fails the job. Diffs are reported via job summary, a sticky PR comment, and the HTML artifact. Drops the label trigger, the approval gate, and the emitter-diff-approved label requirement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Replace diff2html with a self-contained, GitHub-style HTML renderer (inline CSS, light/dark, no external requests). Runtime deps now empty. - Remove diff2html from package.json, pnpm catalog, and lockfile. - Drop the --run-tests/--test-env/--test-target surface and the adapter runTests contract; parallel generation and --sequential are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Now that the tool has zero runtime dependencies, it no longer needs to be a workspace package. Deleting the package.json and its name removes any npm dependency-confusion surface, and the tool runs with plain ode (Node 24 strips TypeScript natively) instead of tsx. - Delete eng/emitter-diff/package.json; drop it from pnpm-workspace.yaml. - Rewrite relative imports to .ts extensions and set allowImportingTsExtensions so ode runs it and sc typechecks. - Invoke via ode src/cli.ts in CI, the shebang, and the README. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Running emitter-diff with plain node requires .ts import specifiers, but the repo-wide check:eng gate (tsconfig.eng.json, --noEmit) rejected them. Enable allowImportingTsExtensions there so the tool stays covered by the existing typecheck. Also reword the ensureDeps comment: http-client-python commits its own package-lock.json to the repo (lockfiles aren't in the published tarball), which is why npm ci works on a source checkout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A local run now prints the exact generated trees the diff compares, so developers can open the head (current checkout) output directly instead of inferring it from the work dir. Language-agnostic: lives in cli.ts, labeled per side by the resolved emitter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

l0lawrence · 2026-07-01T23:54:26Z

should emitter-diff live under eng/common bc we need this in tsp-azure too (like for rust/js/go etc)

l0lawrence · 2026-07-09T18:54:10Z

Another suggestion is can we merge similar diffs into one group to reduce the review effort? It is always a problem previously we review the generated code diff PR.

and

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The core PR microsoft/typespec#11122 adds the `eng/emitter-diff` tool, whose source uses Node-native `.ts`-extension imports (it runs under `node --experimental-strip-types`). This repo's `check:eng` typechecks `./core/eng` (which now includes `emitter-diff`) and was failing with **TS5097** (`An import path can only end with a '.ts' extension when 'allowImportingTsExtensions' is enabled`). **Fix:** enable `allowImportingTsExtensions: true` in `tsconfig.eng.json` — mirroring what core's own `tsconfig.eng.json` does. `check:eng` already runs with `--noEmit`, which this flag requires. Nothing is excluded; `emitter-diff` stays fully typechecked. Companion to microsoft/typespec#11122. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

tadelesh

Not checking the code too much. Reviewed the overall design and result. LGTM. Will try with Go emitter after merge.

tadelesh · 2026-07-20T06:13:32Z

@l0lawrence After this PR merged, how could typespec-azure repo reuse all the assets here?

l0lawrence · 2026-07-21T15:37:03Z

@l0lawrence After this PR merged, how could typespec-azure repo reuse all the assets here?

reference the script in the pkgs package.json from the core submodule in typespec-azure

## What Wires **typespec-go** into the language-agnostic [`emitter-diff`](https://github.com/microsoft/typespec/tree/main/eng/emitter-diff) tool (added in microsoft/typespec#11122, already vendored in the `core` submodule), so we can diff the generated Go code produced by two versions of the emitter — locally and as an informational PR check. Because the generated Go fixtures are **not committed** (`.gitignore`d), a plain `git diff` can't show how an emitter change affects the output. This tool regenerates a **baseline** tree and the **head** (working tree / PR), then renders the difference. ## Changes - **`packages/typespec-go/package.json`** — adds the `diff-regen-code` script. Since the tool's presets live in the `core` submodule, it passes flags directly: `--command "npm run tspcompile"`, `--emitter-path packages/typespec-go`, `--generated-code-path test/http-specs,test/azure-http-specs,test/local` (all three roots `tspcompile` writes), and `--setup` steps (submodule init + `pnpm install` + `build:deps`) that prepare the fetched baseline tree. - **`.github/workflows/ci-emitter-diff-go.yml`** — informational PR check modeled on `ci-emitter-diff-python.yml`, adapted for Go: `submodules: recursive` checkout, azure-sdk npm registry, `build:deps` build (matching `ci-go.yml`), diff against the PR merge-base, sticky PR comment + HTML artifact. **Fails only on a tool/build error, never on a diff.** Skips fork PRs. - **`packages/typespec-go/docs/development.md`** — documents the `diff-regen-code` workflow for reviewing generated-code changes. ## How the diff works `head` defaults to the current working tree (built by the workflow, since the tool runs the regenerate command as-is and never builds your checkout); `baseline` is `gh:<merge-base>`, fetched into a temp worktree and auto-prepared via the script's `--setup`. Each side runs `tspcompile`, the three generated roots are snapshotted, and `git diff --no-index` renders the report. ## Validation Ran a filtered self-diff locally (`pnpm diff-regen-code -- --sequential --baseline local:. -- --filter=bytesgroup`): the full pipeline regenerated both sides and reported **No differences** (exit 0), confirming the wiring. _Note: CI runners have network access for the two installs; the workflow points npm/pnpm at the azure-sdk public feed via `$HOME/.npmrc` so the out-of-repo baseline install inherits it too._ Co-authored-by: tadelesh <chenjieshi@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

l0lawrence and others added 7 commits June 30, 2026 13:22

Add gpgsign to cspell word list

244f134

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

microsoft-github-policy-service Bot added emitter:client:python Issue for the Python client emitter: @typespec/http-client-python eng labels Jun 30, 2026

l0lawrence changed the title ~~Add language-agnostic emitter-diff tool + python adapter (same-repo test)~~ Add language-agnostic emitter-diff tool + python adapter Jun 30, 2026

l0lawrence mentioned this pull request Jun 30, 2026

Add language-agnostic emitter-diff tool + python adapter #11119

Closed

l0lawrence and others added 2 commits June 30, 2026 14:53

l0lawrence and others added 12 commits June 30, 2026 15:10

Remove --vscode diff option from emitter-diff (HTML report is suffici…

17c65ee

…ent) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

timotheeguerin reviewed Jul 9, 2026

View reviewed changes

github-advanced-security AI found potential problems Jul 9, 2026

View reviewed changes

Comment thread eng/emitter-diff/src/diff.ts Fixed

l0lawrence added 2 commits July 9, 2026 11:56

tim comments (no simple-git)

f6b81fd

generate in groups by default can choose to not with --no-group

5c10b9d

l0lawrence force-pushed the l0lawrence-emitter-diff branch from 0840c26 to 5c10b9d Compare July 9, 2026 18:57

l0lawrence and others added 3 commits July 10, 2026 09:54

bump actions to Node 24 runtimes to clear deprecation warning

519c552

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

restrict emitter-diff workflow to non-fork PRs

755fba2

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

remove test line from version.py.jinja2 template

72491e0

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

tadelesh reviewed Jul 13, 2026

View reviewed changes

Comment thread eng/emitter-diff/README.md Outdated

l0lawrence mentioned this pull request Jul 13, 2026

Enable allowImportingTsExtensions for core/eng typecheck Azure/typespec-azure#4883

Merged

l0lawrence added 2 commits July 13, 2026 15:33

allow comma sep list for gen code path

e93b921

Merge branch 'main' into l0lawrence-emitter-diff

6444b05

Merge branch 'main' into l0lawrence-emitter-diff

0bb065e

l0lawrence requested review from msyyc, tadelesh and timotheeguerin July 15, 2026 15:53

tadelesh reviewed Jul 16, 2026

View reviewed changes

tadelesh approved these changes Jul 16, 2026

View reviewed changes

timotheeguerin approved these changes Jul 16, 2026

View reviewed changes

l0lawrence added this pull request to the merge queue Jul 21, 2026

Merged via the queue into main with commit 3af767f Jul 21, 2026
50 checks passed

l0lawrence deleted the l0lawrence-emitter-diff branch July 21, 2026 17:45

tadelesh mentioned this pull request Jul 22, 2026

Add emitter-diff wiring for typespec-go Azure/typespec-azure#5011

Merged

tadelesh mentioned this pull request Jul 31, 2026

Fix handling of utcDateTime types Azure/typespec-azure#5085

Merged

Uh oh!

Conversation

l0lawrence commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How it works

Flags

Changes

CI behavior

Uh oh!

pkg-pr-new Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

@typespec/http-client-python - internal ✏️

Uh oh!

github-actions Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python emitter diff

Uh oh!

azure-sdk-automation Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0lawrence commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

l0lawrence commented Jul 9, 2026

Uh oh!

Uh oh!

tadelesh left a comment

Choose a reason for hiding this comment

Uh oh!

tadelesh commented Jul 20, 2026

Uh oh!

l0lawrence commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

l0lawrence commented Jun 30, 2026 •

edited

Loading

pkg-pr-new Bot commented Jun 30, 2026 •

edited

Loading

github-actions Bot commented Jun 30, 2026 •

edited

Loading

`@typespec/http-client-python` - internal ✏️

github-actions Bot commented Jun 30, 2026 •

edited

Loading

azure-sdk-automation Bot commented Jun 30, 2026 •

edited

Loading