Fix memory leak in experimental mutator engine's seen cache

[Copilot]

The experimental mutator engine (packages/compiler/src/experimental/mutators.ts) holds a module-level, never-cleared, strongly-referenced seen cache. Every mutateSubgraph/mutateSubgraphWithNamespace call inserts Type values into it, pinning the entire type graph of every mutated program in memory for the process lifetime — heap grows unbounded for hosts that drive compile() in a loop (e.g. emitter test suites, versioning mutation via TCGC).

Why the cache can't simply be scoped per-engine

The obvious fix — moving the cache into createMutatorEngine so it dies with the engine — breaks recursive type graphs. Mutators such as @typespec/http's merge-patch transform call mutateSubgraph re-entrantly from inside their own mutate function, and each call spins up a fresh engine. The seen cache must be shared across those nested engines so that a self-referential type (e.g. model Resource { related?: Record<Resource> }) is cloned once instead of recursing forever. A per-engine cache regresses this into RangeError: Maximum call stack size exceeded (observed in the @typespec/http merge-patch and @typespec/samples visibility suites).

The fix: scope the cache per Program

The seen cache is now stored in a WeakMap<Program, SeenCache> instead of a single module-level map. Each program gets its own cache, so:

• The leak is fixed. The cache — and every Type it references — is only reachable through the Program key. Once the program is garbage collected, its cache goes with it, instead of accumulating for the process lifetime. Hosts that compile in a loop no longer grow the heap unboundedly.
• Cycle-breaking still works. Within a single program the cache is shared across the nested, re-entrant mutateSubgraph calls a mutator makes, so recursive/self-referential type graphs are cloned once and terminate.
• Cross-call clone reuse is preserved. Repeated top-level mutations of the same type with the same mutator within a program reuse the earlier clone, which matches the engine's existing behavior.

The type/mutator keyers (typeId/mutatorId) remain module-level: they are WeakMap-backed object keyers that never strongly retain their keys, so they don't leak and are safe to share across programs.

type SeenCache = CustomKeyMap<[MutableTypeWithNamespace, Set<Mutator> | Mutator[]], Type>;

const seenByProgram = new WeakMap<Program, SeenCache>();

function getSeenCache(program: Program): SeenCache {
  let seen = seenByProgram.get(program);
  if (seen === undefined) {
    seen = new CustomKeyMap<[MutableTypeWithNamespace, Set<Mutator> | Mutator[]], Type>(
      ([type, mutators]) => {
        const key = `${typeId.getKey(type)}-${[...mutators.values()]
          .map((v) => mutatorId.getKey(v))
          .join("-")}`;
        return key;
      },
    );
    seenByProgram.set(program, seen);
  }
  return seen;
}

Verification

A local GC harness (using WeakRef + --expose-gc) confirms the fix: after a mutation, dropping the program and forcing GC collects the mutation clone (WeakRef.deref() returns undefined). Running the same harness against the pre-fix module-level cache leaves the clone pinned — reproducing the leak.

Regression tests

packages/compiler/test/experimental/mutator.test.ts:

• reuses cached clones across top-level calls within a program — two top-level mutations of the same type/mutator within one program return the same clone (cache is shared per program).
• scopes the mutation cache per program — equivalent types compiled in two separate programs produce distinct clones in distinct realms; neither program sees the other's clone.
• breaks cycles across re-entrant mutateSubgraph calls — a self-referential model run through a mutator that re-enters mutateSubgraph (mirroring merge-patch) must not overflow the stack. Verified to fail with RangeError: Maximum call stack size exceeded under a per-engine cache, confirming it guards the regression.

Validated suites: compiler experimental (21), @typespec/http merge-patch (37), versioning (137), openapi3 (2547), samples (68) — all pass.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/compiler@11121

commit: 7f8d372

[azure-sdk-automation]

You can try these changes here

🛝 Playground	🌐 Website	🛝 VSCode Extension

[github-actions]

All changed packages have been documented.

• ✅ @typespec/compiler

Show changes

@typespec/compiler - fix ✏️

Fix memory leak in the experimental mutator engine where a module-level seen cache pinned the type graph of every mutated program in memory for the lifetime of the process. The cache is now scoped per Program (via a WeakMap), so it is released once the program is garbage collected while still being shared across the nested mutations required for recursive type graphs to terminate.

[timotheeguerin]

Actually didn't realized the test was added there