[RNE Rewrite] test(ts): add API tests for the TypeScript surface - #1355
Merged
Conversation
msluszniak
marked this pull request as draft
August 7, 2026 14:36
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 10, 2026 06:29
49728d3 to
4755b35
Compare
msluszniak
force-pushed
the
@ms/api-tests
branch
2 times, most recently
from
August 26, 2026 19:06
499a2ad to
5e6e7e9
Compare
12 tasks
msluszniak
marked this pull request as ready for review
August 26, 2026 20:30
barhanc
approved these changes
Aug 27, 2026
Every path through `src/` bottoms out in `__rnexecutorch_jsi__`, so stubbing those calls per test would only ever assert against the stub. Instead of stubbing, implement the native contract in JavaScript: typed-array-backed tensors with the real byte semantics, JS implementations of the math/cv/speech operators, and a `loadModel` that serves a program the test describes. Task pipelines therefore run end to end. Alongside it, an in-memory blob-util mock with a programmable server (status, body, Range support, and a gate to hold a download open) and a worklets mock that runs worklets inline. Native memory is not garbage collected, so the setup file asserts after every test that nothing allocated through the fake was left undisposed; each pipeline suite gets disposal coverage for free.
- `core/`: tensor byte semantics and copy windows, model execution and disposal, `wrapAsync` error propagation, and the spec matcher in depth — symbol binding, variant selection, runtime-constraint matching, and the authoring errors that fire before matching starts. - `fetcher/`: caching, forced re-download, byte-weighted progress, HTTP and transport failures, cancellation, requests shared between concurrent callers, iOS partial-file resume, and the Android DownloadManager backend. Telemetry gets its own suite, including the locale parsing that must not read a language-only tag as a country. - `extensions/`: box and point scaling under both resize modes, box decoding, and the seeded generators.
One suite per pipeline: which model signatures it accepts, that a mismatch is rejected with a message naming the mismatch, the postprocessing that is the pipeline's own work (softmax ordering, NMS suppression, argmax colormaps, sigmoid grayscale masks, coordinate scaling back through letterboxing), every option and per-call override, and that `dispose()` releases everything. The pipelines whose behavior depends on real model weights — Whisper, VAD, SDXS, keypoints, instance segmentation — get schema acceptance, rejection and full disposal instead, including Whisper's nested tokenizer and VAD pipeline. `tasks/constructionFailure.test.ts` records a leak the suites surfaced: a `create<Task>` factory that throws after `loadModel` abandons the native model, and the caller never receives a `dispose` to release it. The tests assert the current behavior so it flips loudly once fixed. Hook suites cover the lifecycle apps depend on: disposal on unmount and on config change, the create-after-unmount race, config identity by value, download cache hits, preventLoad, and errors surfaced through the shared field.
- A snapshot of every export, so an addition, rename or removal shows up in the diff of the pull request that causes it rather than in a user's app. - Registry rules that only fail on a device otherwise: https URLs on the software-mansion org, a pinned revision, the `modelname_backend_precision.pte` naming contract, a folder matching the backend suffix, and a default alias structurally identical to one of its own variants. - Label-array invariants, including the ImageNet duplicates that must stay because the array mirrors the model's output vocabulary. - Source-level conventions, parsed with the TypeScript compiler: the `'worklet'` directive on every JSI wrapper, and the core/extensions and hooks/native import boundaries from the architecture guide.
Adds an `api-tests` job to CI — no native libraries, no simulator, no `.pte`, so the existing TypeScript-only setup action is all it needs. Adds an `add-api-tests` skill covering the fake runtime, the helpers, what to cover for a new pipeline or hook, and the leak-checking contract; wires it into the skills index, the architecture guide and the maintenance list; and adds the test step to the verify-and-build workflow and checklist. Also refreshes the hook example in `add-task-pipeline`, which still showed the `localPath` shape from before the resource fetcher landed.
Rebasing onto the supertonic TTS pipeline (#1317) surfaced two gaps the suites themselves reported: - The registry walk assumed every task config names a single `modelPath`. Supertonic assembles four `.pte` files under `modelPaths`, so its category looked empty and its variant group went unchecked. Recognize both shapes, and compare variants on the whole set of files they name. - The export snapshots record the new TTS surface. Additions only — no removal or rename.
Adapts the suites to what landed on `rne-rewrite` since they were written: hooks report absence as `undefined` rather than `null`, aborts surface as a coded `DOWNLOAD_ABORTED` rather than an `AbortError` class, `schema.constr` is now `schema.constraint` with `equality` in place of `eq`, the CV ops moved to singular module names, `bool` joined `DType`, and the iOS download path reads its status through blob-util's `stateChange`, which the mock now serves. Two registry conventions were restated rather than relaxed. Variant groups name their default with an explicit `DEFAULT` key instead of spreading it, and a default may now live further down the tree (a YOLO26 family defaults through a scale and an input size), so the rule is that the default has to be a config the group actually offers. Model files are still required to declare their backend and precision, but Kokoro nests a variant folder under the backend one and publishes its grapheme-to-phoneme models per language, so the backend is looked for anywhere in the path and the untagged files are pinned to that one shape rather than left to opt out silently. The three source fixes the suites originally carried are all upstream already, so the commit that made them is gone.
`rne-rewrite` gained the privacy filter, the LLM chat session, Kokoro TTS, PaddleOCR and the coded error type since these suites were written, and none of them had a suite. Only the weights are out of scope, so most of this runs end to end. The privacy filter's logits come from a scripted `execute`, and everything above them is driven for real: the BIOES grammar, the Viterbi decode, the sliding window and its overlap policy, and the static-versus-dynamic export handling. The chat session runs against a runner whose KV cache is a token position, which is enough to check that a second turn prefills only what is new, that a tool loop feeds results back, and that a failed turn leaves the session usable. Kokoro gets its argument validation, chunking and streaming; OCR gets its quad decode, CTC collapse, confidence filter and reading order over a probability map the test paints. That needed the fake runtime to grow the parts those pipelines reach: `llm.createLLMRunner`, `speech.createPhonemizer`, `math.gather`, `cv.extractDbnetTextQuads`, `cv.rectifyQuad` and `fs.readFile`. The two CV decoders implement their documented contract over axis-aligned regions rather than tracing contours and warping perspective, which is the same answer for the rectangles a pipeline test draws and keeps every threshold the caller passes load-bearing. The leak check now also covers LLM runners and phonemizers, so the pipelines that own them get disposal coverage the same way.
msluszniak
force-pushed
the
@ms/api-tests
branch
from
August 27, 2026 07:10
5e6e7e9 to
f08af33
Compare
This was referenced Aug 27, 2026
Closed
msluszniak
added a commit
that referenced
this pull request
Aug 27, 2026
…ails (#1386) ## Description Fixes the finding recorded in #1355: a `create<Task>` that throws part-way through construction abandoned everything it had already allocated. A factory allocates as it goes, and only hands back a `dispose` at the very end. Anything that threw in between left the caller with no reference to what was already there, and native memory is not garbage collected, so it stayed alive for the rest of the process. Depending on the task that is a model, a tokenizer, a phonemizer, an LLM runner, or a whole nested pipeline (Whisper owns both a tokenizer and a VAD). `useModel` re-runs its factory whenever the config changes, so an app pointed at a mismatched model leaked a full resource set per attempt. `src/core/lifetime.ts` adds `createResourceScope`, which gives a factory one teardown path for both outcomes: it tracks each resource as it is created, releases them in reverse order, and the same function becomes the pipeline's `dispose`. Every factory now allocates through a scope and wraps its body in `try`/`catch`: ```typescript const scope = createResourceScope(); const dispose = scope.dispose; try { const model = scope.track(await wrapAsync(loadModel, runtime)(modelPath)); const { dims } = validateSpec(model.schema, { ... }); // may throw const tensors = [tensor('float32', outShape)] as const; tensors.forEach(scope.track); return { classify, classifyWorklet, dispose }; } catch (error) { dispose(); throw error; } ``` `createKokoroTextToSpeech` already did this with a local array and moves to the shared helper. That also closes a smaller hole it had: its two models were loaded before the `try` block, so one load rejecting stranded the other. Parallel loads elsewhere now track inside each promise for the same reason. `createTokenizer` is unchanged. It loads a tokenizer and returns with nothing in between that can throw, so it has no window to leak through. The helper is internal, so the public export surface is unchanged. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [x] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions ```sh yarn workspace react-native-executorch test yarn typecheck yarn lint ``` Expected: 28 suites, 3117 tests, 4 snapshots, 0 skipped; typecheck and lint clean. To see the tests bite, stash `src/` and re-run: every construction-failure case fails. ### Screenshots N/A ### Related issues Follows up the finding recorded in #1355. ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes Stacked on `@ms/api-tests` (#1355), because that is where the suite recording this behavior lives. Merge #1355 first and this retargets to `rne-rewrite` cleanly. `tasks/constructionFailure.test.ts` was written to record the leak as current behavior so it would fail loudly the day a factory started cleaning up. That is this PR, so it now asserts the opposite, and covers all fifteen factories rather than five. It checks every kind of handle, so a failure names the factory and the resource rather than just reporting that something leaked. The per-pipeline suites and `hooks/taskHooks.test.ts` drop the `allowNativeLeaks()` calls they needed for the same reason. Nothing in the suite leaks any more, so the setup file's global leak check now asserts this on every construction-failure test for free. The diff is large mostly because indenting a factory body inside `try` touches every line of it. The behavioral change per file is the scope, the `track` calls and the `catch`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds TS API tests:
Also adds necessary skills.
To find details about this approach please follow:
__tests__/README.md.No source changes: the three contract violations the suites originally surfaced (
SpecMatch.dim, the missing'worklet'directive ongetRegisteredBackends, andrandomNormal's millisecond-resolution default seed) have all since been fixed onrne-rewrite, sosrc/is untouched by this PR.During the testing I spotted a problem that is addressed in #1386.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
yarn workspace react-native-executorch test yarn typecheck yarn lintExpected: 28 suites, 3072 tests, 4 snapshots, 0 skipped; typecheck and lint clean.
yarn preparestill emits onlysrc/intolib/.Screenshots
N/A
Related issues
Closes #1352.
Checklist
Additional notes
Layout
core/fetcher/tasks/hooks/useModel,useResourceDownload, and the task hooks end to endextensions/api/support/Deliberately not covered: the numerical behavior of the native operators (that is
cpp/tests/, and duplicating it here would only test the fake); the parts of a pipeline whose behavior depends on real weights: Whisper's decode loop, the VAD rolling window and the SDXS diffusion step get schema acceptance, rejection and full disposal instead; and the worklet thread hop, since worklets run inline. The'worklet'directive convention that makes that hop possible is enforced by parsingsrc/with the TypeScript compiler.That line is drawn per pipeline, not per suite. The privacy filter's logits are weights but its BIOES decode and sliding window are not, so they run end to end; the LLM's generation belongs to the native runner but the chat session's history, KV cache bookkeeping and tool loop are covered against a scripted one; Kokoro's waveform is weights but its argument validation, chunking and streaming are not; PaddleOCR's probability map is weights but the quad decode, CTC collapse and reading order run over a map the test paints. The fake runtime grew
llm.createLLMRunner,speech.createPhonemizer,math.gather,cv.extractDbnetTextQuads,cv.rectifyQuadandfs.readFileto make that possible.