ENG-9148: feat(compiler): incremental compile cache & warm hot-reload daemon (REFLEX_COMPILE_CACHE) - #6688
ENG-9148: feat(compiler): incremental compile cache & warm hot-reload daemon (REFLEX_COMPILE_CACHE)#6688FarhanAliRaza wants to merge 28 commits into
Conversation
Add an experimental, flag-gated incremental frontend compile cache that recompiles only the pages whose source actually changed and reuses the rest. Two layers, both off by default and enabled by REFLEX_COMPILE_CACHE: - In-process per-page cache (page_cache.py): a Salsa-style dependency graph records the exact set of source files each page reads, so editing one file invalidates only the pages that depend on it. Pages are keyed by a small genuinely-global epoch (Reflex version + rxconfig + lockfile) plus the content hashes of their dependency set. Speeds up repeat compiles within a single process. - On-disk manifest (disk_cache.py): persists each page's serializable contribution and dependency hashes to .web/reflex_compile_cache.json so a fresh process — notably a `reflex run` hot-reload worker, which respawns on every edit — recompiles only changed pages and reuses the rest. Falls back to a full compile on any unsafe condition. REFLEX_COMPILE_CACHE_VERIFY runs a full compile alongside the cached one and asserts byte-identical output, falling back on mismatch — the backstop for gaps a static dependency graph cannot see (runtime importlib imports, data read at module-import time). Supporting changes required for safe page reuse: deterministic compile-time ref-name generation, and own-before-mutate page metadata injection.
Merging this PR will not alter performance
Comparing 
Footnotes
|
 The PR is not yet safe to merge because an outstanding app-import tracking issue can allow stale app-wide frontend output when the compile cache is enabled.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a453ccbd69
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Add a persistent compile daemon (REFLEX_COMPILE_CACHE) that imports the world once and forks a throwaway child per source change, instead of the reloader respawning a worker that cold-imports on every edit. The child re-imports first-party code fresh and runs the incremental rebuild, so correctness matches a respawn while the cold import is paid once. Supporting changes that make the daemon safe and complete: - write_file now writes atomically (temp + os.replace) so a reader (vite, a concurrent compile) never sees a half-written file, even when a forked child is killed mid-compile. - _run_dev launches the daemon and sets REFLEX_SKIP_COMPILE on the backend so it only evaluates pages to register state. - The daemon watches what the compiler reads (incl. sibling-dir markdown from the manifest); uvicorn reload_includes also covers *.md/*.mdx, so markdown edits finally trigger a reload. - Drop the per-rebuild console.info now that progress is shown inline.
multi_docs built every component's prop tables at module import, so the whole library reference was reconstructed on every import of the docs tree -- re-run on every dev hot-reload reimport, cold start, and backend respawn. Move the build into the page render closures (matching the non-library doc path) so a page builds its prop tables only when it is actually compiled. Docs cold import 9.7s -> 1.9s; hot-reload reimport 8.3s -> 0.6s.
- disk_cache: stop re-evaluating stateful HIT pages during an incremental rebuild. The compiling process never serves (the daemon, the initial compile, and CLI compiles all exit; the serving backend re-evaluates the marked stateful pages itself), so re-running their render pipeline was pure waste. The stateful-pages marker stays complete -- hits recorded from the manifest, misses from the fresh compile. - compile_daemon: poll faster (0.25s -> 0.05s) but cheaply -- stat the known file set each tick and rglob only every 1s for added/removed files, cutting detection latency without burning idle CPU. - compile_daemon: log per-edit timing (reset / reimport / compile).
|
@greptile please rereview |
Profiling the hot-reload compile on the docs app (418 pages) showed the disk-cache rebuild spent ~1.1s of its ~1.6s on manifest I/O: the manifest stored every page's rendered output_code (and output_path/frontend_imports), ballooning it to 46MB, yet those fields are never read back — only dep_hashes, app_wrap_keys, is_stateful, and the merged all_imports are consumed. Storing just the bookkeeping that is read drops the manifest to 14MB. The in-process page cache (_PAGE_STORE / validate_page / store_page) is never reached by the warm daemon (its fork child cleared it and the disk path returns first), and forcing it measured ~55x slower than the disk path because it re-runs the whole app-level pipeline (memo render, stylesheet, plugins) every edit. Remove it, the verify mode that only guarded it (REFLEX_COMPILE_CACHE_VERIFY), and the dead page_source_fingerprint. Manifest schema bumped 2->3 so stale fat manifests are ignored.
Remove page_module_files and file_hashes, now-dead after the compile-cache manifest rework, along with their tests.
Two correctness fixes for the incremental compile cache, plus a pass to trim verbose comments and docstrings across the cache modules. - global_epoch now fingerprints the app-level config files: the app entrypoint module plus the config-only modules it imports (theme, app-wraps, stylesheets), found by walking the import graph with page modules as barriers so per-page incrementality is preserved. An edit to app-wide config previously left every page a hit, so the reused on-disk app root / contexts / theme stayed stale. - The incremental manifest refresh now persists the complete install import set (page imports merged with the memo-component imports the rebuild generated, and threading root through dependency discovery), so a later all-hit compile installs every package the reused memo files need instead of dropping newly-introduced memo packages.
test_reset_model_metadata_allows_table_redefinition builds an
rx.Model(table=True), which needs the SQLModel stack. The "without db
dependencies" unit-test job uninstalls it, so the first model definition
raised TypeError before the test could exercise the reset. Guard with
pytest.importorskip("sqlmodel"), matching the repo's other db-dependent
tests, so it skips there instead of failing.
Add the news fragments the changelog check requires for the packaged source touched by this PR: a feature entry for the reflex package's REFLEX_COMPILE_CACHE flag and a misc entry for the reflex-base recorder hook / reproducible ref names that support it.
Patch __import__ and importlib.import_module while a page recorder is active so modules imported during page eval are recorded as source dependencies. Resolves names (including relative imports), skips stdlib and out-of-project files, and caches module-file lookups. Lets the incremental compile cache invalidate pages that depend on first-party modules pulled in at import time.
Replace the os.path string-comparison helpers used for runtime import tracking with pathlib.Path operations, dropping the redundant cached root-string globals (_recorder_root_str, _recorder_raw_root_str) and the sentinel-based module-file cache. Make the read-set target an explicit argument instead of falling back to the recorder context.
The compile-cache read tracker patches builtins.__import__ to record a page's runtime imports. While recording, _recordable_module_file resolves a module file via Path(...).absolute(), which on CPython 3.12 lazily runs `import ntpath` to compare path flavours. That import re-entered the patched hook -> resolved a path again -> imported again, recursing until the stack overflowed (32 unit tests failed only on ubuntu 3.12). Suspend read/import tracking (_suspend_tracking) while the tracker's own path/module bookkeeping runs, so imports and reads it triggers are neither recorded nor able to re-enter. Add a version-independent regression test that mimics the 3.12 lazy import.
|
@codex pelase review |
Config the app module pulls in dynamically (importlib calls, files read at import time) was invisible to app_dependency_files, which only walked the static import graph from the entrypoint. Editing such config could leave reused app-wide output stale. Record files read/imported while the app module loads (record_app_import, wired into prerequisites.get_app under REFLEX_COMPILE_CACHE) and fold that dynamic set into app_dependency_files, subtracting per-page static closures so ordinary page edits still invalidate per-page. Extract the shared graph walk into _walk_import_closure.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21e7d3ea9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #: Watchdog: kill a compile child that runs longer than this (a hung/deadlocked | ||
| #: child must never wedge the daemon). Generous enough for a real full compile. | ||
| _COMPILE_TIMEOUT = 300.0 | ||
| #: Source suffixes edited under the app roots that should trigger a recompile. |
There was a problem hiding this comment.
When a page reads an in-project data file with a suffix that record_reads() records (such as .json, .yaml, .toml, .txt, .csv, .html, or .rst), editing that file does not wake the compile daemon: this watched suffix list only includes .py, .md, and .mdx, and _external_dependency_files() filters out deps already under the reload roots. The manifest would make the page stale if a compile ran, but no compile is triggered, leaving the frontend stale until another watched file changes.
Useful? React with 👍 / 👎.
The incremental rebuild no longer re-emits the contexts file every time. It is rewritten only when a stateful page missed and that page's own state config actually changed (fingerprinted against the manifest); otherwise the on-disk file is reused untouched. When a rewrite is needed, the stateful hit pages are evaluated first so the state registry — and the frontend dispatch map compiled from it — stays complete. Supporting changes: - Reset the daemon's state registry surgically: states from modules that survive the purge (framework/installed/workspace packages) are re-registered in original order, while purged-module and reflex.istate.dynamic states are dropped so re-created local states get deterministic fresh-process names. - Patch react-router's served HMR runtime so an edit to a not-currently-open route no longer throws and poisons HMR until a full reload. - Store per-input epoch digests instead of one combined hash so a global-input mismatch can name the exact file that changed, and surface fallback reasons and detected file changes at info level. - Write the vite config atomically and regenerate it on the incremental path.
The hot paths of a warm rebuild did far more work than needed: - Collapse duplicate ImportVars when serializing the manifest and merge the miss pages' import sets once, instead of re-merging the ~100k-entry app-wide set per page. A full docs-app compile accumulates ~107k import entries of which only ~6k are unique. - Key the module-file cache by the raw __file__ value and check it before any Path construction, so every import statement under an active read recorder skips the resolve. Cleared when the recorder root changes. - Compare path strings in _under_roots instead of building a Path per ancestor per module, which dominated the daemon's reset phase on every hot reload.
The incremental-cache tests fingerprint state config via _contexts_snapshot, which walks the whole root state tree — picking up unrelated (and sometimes broken) state classes registered by other test modules. Stub it with a snapshot limited to the states this file defines, and stub externals before write_manifest so the fingerprint it records uses the same scoped snapshot.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dbfacec09
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if root in rf.parents: | ||
| file_to_mod[str(rf)] = name |
There was a problem hiding this comment.
When an app uses REFLEX_HOT_RELOAD_INCLUDE_PATHS for a sibling local package, the daemon watches that root, but this dependency graph keeps only modules whose files are under Path.cwd(). A page importing a component/helper from the sibling package therefore records no hash for that .py file; editing it wakes the daemon, partition_pages sees all stored deps unchanged, and the incremental path reuses stale .web output.
Useful? React with 👍 / 👎.
| _record_module_file(result, target) | ||
| result_id = id(result) | ||
| if (module := sys.modules.get(name)) and id(module) != result_id: | ||
| _record_module_file(module, target) |
There was a problem hiding this comment.
This records only the module returned by a runtime import. If a page dynamically imports a helper that was already loaded by an earlier page or by app import, the helper's own imports are not executed under this recorder and are not added to the page dependency set; editing one of those transitive first-party files can leave that page classified as a cache hit and keep its previously generated output stale.
Useful? React with 👍 / 👎.
masenf
changed the title
# Conflicts: # reflex/compiler/compiler.py
| if not config.app_module | ||
| else config.app_module |
There was a problem hiding this comment.
When config.app_module is already populated, this branch returns the existing module while record_app_import() is active, but no import or file read is replayed inside the recorder. That can leave the app-wide read set empty or incomplete. With REFLEX_COMPILE_CACHE enabled, a change to a dynamically loaded theme, app wrap, toaster, stylesheet, or provider module can leave the global inputs unchanged, so the cache can reuse stale app-root output instead of falling back to a full compile. This path needs to fingerprint the preloaded module's app-wide dependencies, or force a full compile when those reads cannot be captured.
masenf
left a comment
masenf
left a comment
There was a problem hiding this comment.
there is an issue when updating a page to use a memo component imported from another module
app/app.py
import reflex as rx
# from .m1 import m1
def index():
return rx.vstack(
"test",
# m1(),
)
app = rx.App()
app.add_page(index)Run as REFLEX_COMPILE_CACHE=1 reflex run
Then add the following file:
app/m1.py
import reflex as rx
@rx.memo
def m1() -> rx.Component:
return rx.text("hello")Then uncomment the lines in the app.py.
This gives a red error:
[plugin:vite:import-analysis] Failed to resolve import "$/app_components/ccminimal/m1" from "app/routes/_index.jsx". Does the file exist?
/Users/masenf/code/reflex-dev/reflex/examples/ccminimal/.web/app/routes/_index.jsx:1:26
1 | import RefreshRuntime from "virtual:react-router/hmr-runtime";const inWebWorker = typeof WorkerGlobalScope !== 'undef...
2 | import { M1_6aca64ba } from "$/app_components/ccminimal/m1";
| ^
3 | import { Flex as RadixThemesFlex } from "@radix-ui/themes";
4 | import { Fragment, useEffect } from "react";
at _formatLog (/Users/masenf/code/reflex-dev/reflex/examples/ccminimal/.web/node_modules/vite/dist/node/chunks/node.js:30602:42)
at error (/Users/masenf/code/reflex-dev/reflex/examples/ccminimal/.web/node_modules/vite/dist/node/chunks/node.js:30599:13)
at /Users/masenf/code/reflex-dev/reflex/examples/ccminimal/.web/node_modules/vite/dist/node/chunks/node.js:27905:35
at transform (/Users/masenf/code/reflex-dev/reflex/examples/ccminimal/.web/node_modules/vite/dist/node/chunks/node.js:27873:17)
at processTicksAndRejections (native)
The explicit memo component doesn't get compiled, but I'm not exactly sure why not.
| self.auto_memo_components.clear() | ||
|
|
||
| # Keep generated ref names stable across in-process compiles. | ||
| reset_unique_variable_names() |
There was a problem hiding this comment.
the practical problem with resetting this every time is that the generated names are order-dependent. so if the first compilation hits 3 pages and they each use get_unique_variable_name() once, then each page will get a unique name. but if the cached recompile only hits the third page next time, then the third page will get the same name that was generated for the (cached) first page, resulting in 2 pages that now have a reference to the same "unique" name.
i think we need a little more machinery here; off the top of my head thinking, _UNIQUE_NAME_RNG should be re-seeded for each page being compiled, and the seed should be based on the hash of the page route. That way the ordering would be deterministic per-route, so even if some pages don't get recompiled the generated names should realistically never conflict and within a given page compilation, the generated names would be stable based on the order the names are generated (but within a single page, this should be fine).
| console.info("Compile daemon ready (warm); watching for changes.") | ||
|
|
||
| while True: | ||
| time.sleep(_POLL_INTERVAL) |
There was a problem hiding this comment.
all supported OS provide a mechanism for watching a filesystem for changes without polling. we cannot be walking the filesystem and stat'ing files every 0.05 seconds. the comments say stat is cheap, but i disagree; we should be subscribing to changes in the file tree (it could be from the whole root, but probably better to subscribe to individual files and directories that we're explicitly watching -- i know recursive change_notify on windows can be slow in large trees).
we can fall back to polling, but it shouldn't be the first choice.
# Conflicts: # packages/reflex-base/src/reflex_base/compiler/templates.py
| app = ( | ||
| __import__(module, fromlist=(constants.CompileVars.APP,)) | ||
| if not config.app_module | ||
| else config.app_module |
There was a problem hiding this comment.
config.app_module has already been loaded before get_app() runs, this branch returns the cached module inside the recorder instead of replaying the app import. That means import-time dynamic imports and file reads for app-wide inputs are not captured for the compile cache. With REFLEX_COMPILE_CACHE enabled, editing a dynamically loaded theme, app wrap, toaster, stylesheet, or provider module can leave the global inputs unchanged and let the all-hit incremental path reuse stale app-root output. This path should force a fresh tracked import or fall back when the preloaded module's app-wide reads cannot be captured.
masenf's review repro: with REFLEX_COMPILE_CACHE=1, adding a new @rx.memo module and importing it from a page left the page importing $/app_components/<app>/<mod> while that file was never written (red Vite error). Root cause: memo emission derived dirtiness from _changed_dependency_files, which only reads the previous manifest's dep_hashes — a brand-new file can structurally never appear there. Rework the memo decision salsa-style. The manifest (schema 8) stores a per-output-file record: the definitions landing in it, content hashes of the source module's first-party import closure, and the owning route. A file whose record differs from the current registry's demand is rewritten — a new module has no record, so it is dirty by construction. Ownership (a registration_owner ContextVar set around each page evaluation, recorded on MemoDefinition.owner_route) extends this to modules imported inside page functions: route-owned entries are validated by their owner page's hit/miss status and recomputed post-evaluation for recompiled routes, while entries of hit owners carry forward untouched. Hit pages contributing auto memos to a rewritten file are re-evaluated so their exports survive. The closure-based record also fixes a latent staleness bug: editing a helper module used by a memo now rewrites the memo's generated JS. The import graph cache is dropped after evaluation when new first-party modules were loaded, so post-evaluation closures are complete; the old changed-files condition stays as a net for memos without a trackable module file (exec'd docs demos).
… main Squash of FarhanAliRaza:reflex-hmr at e765921 rebased onto current main. Conflict resolutions: keep main's sass cast and console.progress(), keep main's RegistrationContext-based decorated page reset in get_app, dedupe tuple-valued import lists in collapse_imports (the PR's behavior), and port the daemon reset and its test off the deprecated DECORATED_PAGES global. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…te stat-first Fixes the performance and accuracy issues found reviewing #6688: - The daemon parent was never single-threaded after its initial compile (telemetry's ThreadPoolExecutor worker stays alive), so `_can_fork` always fell back to a cold subprocess. Add `telemetry.shutdown_executor()` and quiesce the parent before every fork; log the offending threads otherwise. - Replace the 50 ms stat-polling loop (plus 1 s tree walks and assets rglob) with watchfiles notifications over the reload roots, external dependency dirs, global files and assets; polling remains only as a fallback. Any recorded dependency counts as a compile input, not just .py/.md/.mdx. - Hand the watcher's changed set to the compile child so pages depending on none of the changed paths are hits without touching the filesystem. - Manifest schema 7: one shared file table `{path: [sha256, mtime_ns, size]}` and per-page dependency path lists. Validation is stat-first; a file is hashed only when its stat moved, and a touched-but-identical file has its entry refreshed instead of being re-hashed on every reload. Global inputs (now including pyproject.toml/requirements.txt) go through the same table. - Persist the import-name parse cache in the manifest keyed by stat, so an unchanged first-party module is never re-parsed by a child. - Cache the per-`__file__` first-party classification and per-class module file, warmed in the daemon parent so forked children inherit them. - Store each stateful page's contexts contribution (initial-state and client-storage slice) instead of a fingerprint; `compile_contexts` accepts extra entries, so a state-config change rebuilds contexts from the manifest without re-evaluating any stateful hit page. - Hold a compile-in-progress lock while the child runs; a backend reload worker that skips compilation waits on it before reading the stateful pages marker, closing the race on the same file save. - First-party roots include REFLEX_HOT_RELOAD_INCLUDE_PATHS; dynamically imported .py reads expand through the import closure; read/import hooks are not installed in backend workers; atomic writes use a dot-prefixed temp name so route discovery and watchers ignore it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…ay unique Generated ref/loop names were drawn from one process-wide sequence that the compile reset at its start, so a page's names depended on which pages compiled before it: an incremental compile of page 3 alone reproduced the names page 1 already held (masenf's review on #6688). Seed the generator per page from its route, capture the state after evaluation and resume it for the page's render, and reset once for whatever compiles after the pages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…te stat-first Fixes the performance and accuracy issues found reviewing reflex-dev#6688: - The daemon parent was never single-threaded after its initial compile (telemetry's ThreadPoolExecutor worker stays alive), so `_can_fork` always fell back to a cold subprocess. Add `telemetry.shutdown_executor()` and quiesce the parent before every fork; log the offending threads otherwise. - Replace the 50 ms stat-polling loop (plus 1 s tree walks and assets rglob) with watchfiles notifications over the reload roots, external dependency dirs, global files and assets; polling remains only as a fallback. Any recorded dependency counts as a compile input, not just .py/.md/.mdx. - Hand the watcher's changed set to the compile child so pages depending on none of the changed paths are hits without touching the filesystem. - Manifest schema 7: one shared file table `{path: [sha256, mtime_ns, size]}` and per-page dependency path lists. Validation is stat-first; a file is hashed only when its stat moved, and a touched-but-identical file has its entry refreshed instead of being re-hashed on every reload. Global inputs (now including pyproject.toml/requirements.txt) go through the same table. - Persist the import-name parse cache in the manifest keyed by stat, so an unchanged first-party module is never re-parsed by a child. - Cache the per-`__file__` first-party classification and per-class module file, warmed in the daemon parent so forked children inherit them. - Store each stateful page's contexts contribution (initial-state and client-storage slice) instead of a fingerprint; `compile_contexts` accepts extra entries, so a state-config change rebuilds contexts from the manifest without re-evaluating any stateful hit page. - Hold a compile-in-progress lock while the child runs; a backend reload worker that skips compilation waits on it before reading the stateful pages marker, closing the race on the same file save. - First-party roots include REFLEX_HOT_RELOAD_INCLUDE_PATHS; dynamically imported .py reads expand through the import closure; read/import hooks are not installed in backend workers; atomic writes use a dot-prefixed temp name so route discovery and watchers ignore it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…ay unique Generated ref/loop names were drawn from one process-wide sequence that the compile reset at its start, so a page's names depended on which pages compiled before it: an incremental compile of page 3 alone reproduced the names page 1 already held (masenf's review on reflex-dev#6688). Seed the generator per page from its route, capture the state after evaluation and resume it for the page's render, and reset once for whatever compiles after the pages. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
Found by driving the daemon with a browser rather than unit tests: - The forked child could not re-import the app: the warm parent's App stays bound to the RegistrationContext across the fork and the context refuses a second App. Release the slot in the first-party reset. - `reflex.lock/` is a directory of derived lockfiles rewritten on every install; treating it as a global input made every reload fall back to a full compile. Drop it from the global inputs; absent inputs are validated with is_file(). - Backend reload workers ran the cache path too. On Python 3.14 granian's workers come from a forkserver that the CLI's own initial compile started before `reflex run` set the skip flag, so environment set later never reaches them. Ownership is now decided from an on-disk marker written by `reflex run` (`compile_daemon.owns_compilation()`); the daemon and its children carry REFLEX_COMPILE_DAEMON and always own the build. - The file watcher runs a native thread; fork while it is alive trips Python's multi-threaded fork warning. Tear the watcher down before each compile, re-arm it after, and catch up on edits made in between from a stat snapshot of the recorded inputs. `_can_fork` also waits briefly for native threads to finish. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…indows os.kill(pid, 0) terminates the target on Windows; use OpenProcess there and map ProcessLookupError/PermissionError explicitly elsewhere. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uV6WEQPemtsE9Nm6WFxyE
…ilures States defined at import time have no per-page context slice, so a change to their definitions never invalidated anything and the shared frontend contexts went stale. Hash their source files (and their import closure) as global inputs, and serialize the manifest with reflex_base's json_dumps so state defaults needing Reflex serialization no longer abort the manifest write. Manifest write/refresh failures now warn instead of logging at debug level, and an incomplete compile says which page blocked the save. Key the module-file cache on (class, roots) so a different project root can't read another root's answer, and memoize first-party module resolution. In the daemon, snapshot every watchable input at the point the watcher stops and reconcile that checkpoint once it subscribes again, so edits made during a compile are not lost. When native threads are still alive past the deadline, fall back to a cold subprocess instead of forking anyway. Claude-Session: https://claude.ai/code/session_01VZ2gyiCc97WDZSNw8ozZxB
| mark_daemon_active() | ||
| proc = subprocess.Popen( | ||
| [sys.executable, "-m", "reflex.utils.compile_daemon"], env=env | ||
| ) |
There was a problem hiding this comment.
The ownership marker is written before the daemon subprocess starts and contains the long-lived reflex run PID rather than the daemon PID. If startup fails or the daemon later exits unexpectedly, backend workers still consider it active and skip frontend compilation. Nothing then updates .web or the stateful-pages marker for the rest of the session. Record the daemon PID after successful startup, or clear and validate the marker when the supervised process exits.
Knowledge Base Used: Application lifecycle and configuration
| sys.executable, [sys.executable, "-m", "reflex.utils.compile_daemon"] | ||
| ) | ||
|
|
||
| before = _dependency_snapshot(state) |
There was a problem hiding this comment.
An edit can occur after _next_changes() closes the watcher but before this baseline snapshot is taken. That edit is missing from the current changed set, yet its new timestamp appears in both this baseline and the post-compile checkpoint, so it is also absent from pending. The generated frontend then remains stale until another edit triggers compilation. Preserve the watcher's last reconciled snapshot or capture the baseline before stopping it.
Knowledge Base Used:
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
15 issues found across 34 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/units/compiler/test_page_cache.py">
<violation number="1" location="tests/units/compiler/test_page_cache.py:570">
P3: This assertion is a tautology and tests nothing: `assert v.unchanged([str(f)]) is False or True` parses as `assert (v.unchanged([str(f)]) is False) or True`, which is always True regardless of `unchanged`'s result. The comment claims it verifies the call is memoized (no file read), but if memoization regressed and a read occurred, this line would still pass. Assert `is True` instead so the memoization guarantee is actually enforced.</violation>
</file>
<file name="reflex/compiler/page_cache.py">
<violation number="1" location="reflex/compiler/page_cache.py:402">
P2: When a page dynamically imports a submodule with `importlib.import_module`, the recorder omits its parent packages. Track the loaded parent modules too, because changes to `pkg/__init__.py` can change the page while the cache reports a hit.</violation>
<violation number="2" location="reflex/compiler/page_cache.py:715">
P2: When a first-party package uses relative imports in `__init__.py`, the static import graph resolves them against the wrong package. Correctly resolve package-module relative imports so edits to those helpers invalidate dependent pages.</violation>
<violation number="3" location="reflex/compiler/page_cache.py:992">
P1: When `rx.App` and a page are defined in the same module, edits to app-wide configuration invalidate only the page dependency. The incremental path reuses the stale app root, theme, or stylesheet; do not treat the entrypoint itself as a page barrier.</violation>
<violation number="4" location="reflex/compiler/page_cache.py:1256">
P2: When a page handles a missing file read, `page_dependency_entries` drops that path from `deps`. If the file appears later, the cached page remains a hit and its output stays stale; retain unreadable paths in `deps` so the validator forces a safe miss.</violation>
</file>
<file name="reflex/compiler/compiler.py">
<violation number="1" location="reflex/compiler/compiler.py:1248">
P1: When the cache manifest is reusable, this early return skips configured plugin lifecycle hooks and static-asset/modify-file outputs. Plugin-generated files can therefore remain stale or be missing after an incremental reload; make the incremental path execute the same plugin output hooks before returning.</violation>
</file>
<file name="reflex/reflex.py">
<violation number="1" location="reflex/reflex.py:219">
P2: When `REFLEX_COMPILE_CACHE` is enabled, startup compiles twice: `_run_dev` already completed `_compile_app()`, then this daemon runs another initial `get_compiled_app()` before watching. Reuse the completed initial build and only warm the daemon, or skip its initial compile when the parent precompile succeeded.</violation>
<violation number="2" location="reflex/reflex.py:226">
P1: If the daemon exits unexpectedly, backend reload workers keep `REFLEX_SKIP_COMPILE=true` after the marker is cleared, so `_should_compile()` never reaches its ownership fallback and subsequent edits cannot regenerate `.web`. Remove this persistent skip flag and rely on `owns_compilation()` (or explicitly unset it when the daemon exits).</violation>
</file>
<file name="reflex/compiler/disk_cache.py">
<violation number="1" location="reflex/compiler/disk_cache.py:916">
P2: After `compile_app` clears `app._pages`, a successful incremental rebuild returns without restoring the compiled-page bookkeeping used by telemetry and post-compile consumers. Keep this in-memory metadata consistent with the full compile, or provide an equivalent cache-aware representation before returning.</violation>
</file>
<file name="reflex/utils/exec.py">
<violation number="1" location="reflex/utils/exec.py:606">
P2: This reload-watch change is applied unconditionally to the default immutable dev path and is not gated behind `REFLEX_COMPILE_CACHE`. With the feature off (the default), uvicorn users now get a backend reload + full recompile whenever any `.md`/`.mdx` file under a reload path is edited—behavior they never had before (uvicorn's default `reload_includes` is `["*.py"]`). That contradicts the PR's stated "no breaking changes to the default behavior" guarantee and can trigger unexpected restarts for projects with many markdown/docs files. Confirm this default-path change is intended and, if so, document it; otherwise scope it behind the feature flag or keep it opt-in so the default dev behavior stays unchanged.</violation>
</file>
<file name="reflex/utils/compile_daemon.py">
<violation number="1" location="reflex/utils/compile_daemon.py:64">
P2: When daemon startup fails, this line records the live `reflex run` PID before any daemon exists, so backend workers can skip frontend compilation for the rest of the session. Start the child first and record its PID only after `Popen` succeeds, with cleanup for startup failures.</violation>
<violation number="2" location="reflex/utils/compile_daemon.py:383">
P2: When a new source file is created directly under the project root, the watchfiles path misses it because the daemon watches existing children rather than the root directory. Watch the project root and filter it, or retain a root-level rescan alongside watchfiles.</violation>
<violation number="3" location="reflex/utils/compile_daemon.py:975">
P1: When a global input changes, the restarted daemon compiles without the lock that backend workers wait on. A concurrent backend reload can therefore register the previous stateful-page set; hold `_compile_lock` around the initial compile after restart, including failure cleanup.</violation>
<violation number="4" location="reflex/utils/compile_daemon.py:979">
P2: An edit between `_next_changes()` stopping the watcher and this snapshot can be absent from both `changed` and `pending`, leaving `.web` stale until another edit. Preserve the watcher's last reconciled snapshot and use it as the compile baseline, or capture that baseline before stopping the watcher.</violation>
</file>
<file name="docs/app/reflex_docs/pages/docs/component.py">
<violation number="1" location="docs/app/reflex_docs/pages/docs/component.py:963">
P1: When the incremental cache recompiles only one component-doc page, these deferred calls restart the shared `PropDocsState` ID sequence for that page. Cached pages retain the old IDs, so state slices and interactive prop controls can collide or update another component's control; derive IDs from stable component/property identity (including the expansion key) instead of the process-global counter, or otherwise preserve the full registration sequence.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| # version); re-exec the daemon so the new world is actually loaded. | ||
| if changed & state.globals_: | ||
| console.info("Global config changed; restarting compile daemon.") | ||
| os.execv( |
There was a problem hiding this comment.
P1: When a global input changes, the restarted daemon compiles without the lock that backend workers wait on. A concurrent backend reload can therefore register the previous stateful-page set; hold _compile_lock around the initial compile after restart, including failure cleanup.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/utils/compile_daemon.py, line 975:
<comment>When a global input changes, the restarted daemon compiles without the lock that backend workers wait on. A concurrent backend reload can therefore register the previous stateful-page set; hold `_compile_lock` around the initial compile after restart, including failure cleanup.</comment>
<file context>
@@ -0,0 +1,1030 @@
+ # version); re-exec the daemon so the new world is actually loaded.
+ if changed & state.globals_:
+ console.info("Global config changed; restarting compile daemon.")
+ os.execv(
+ sys.executable, [sys.executable, "-m", "reflex.utils.compile_daemon"]
+ )
</file context>
| page_deps: set[str] = set() | ||
| for page in pages or (): | ||
| starts = _component_source_files(getattr(page, "component", None), root) | ||
| barriers |= starts |
There was a problem hiding this comment.
P1: When rx.App and a page are defined in the same module, edits to app-wide configuration invalidate only the page dependency. The incremental path reuses the stale app root, theme, or stylesheet; do not treat the entrypoint itself as a page barrier.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/compiler/page_cache.py, line 992:
<comment>When `rx.App` and a page are defined in the same module, edits to app-wide configuration invalidate only the page dependency. The incremental path reuses the stale app root, theme, or stylesheet; do not treat the entrypoint itself as a page barrier.</comment>
<file context>
@@ -0,0 +1,1259 @@
+ page_deps: set[str] = set()
+ for page in pages or ():
+ starts = _component_source_files(getattr(page, "component", None), root)
+ barriers |= starts
+ page_deps |= _walk_import_closure(graph, starts)
+
</file context>
| barriers |= starts | |
| barriers |= starts - {str(entrypoint)} |
| prerender_routes=prerender_routes, | ||
| use_rich=use_rich, | ||
| ): | ||
| return True |
There was a problem hiding this comment.
P1: When the cache manifest is reusable, this early return skips configured plugin lifecycle hooks and static-asset/modify-file outputs. Plugin-generated files can therefore remain stale or be missing after an incremental reload; make the incremental path execute the same plugin output hooks before returning.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/compiler/compiler.py, line 1248:
<comment>When the cache manifest is reusable, this early return skips configured plugin lifecycle hooks and static-asset/modify-file outputs. Plugin-generated files can therefore remain stale or be missing after an incremental reload; make the incremental path execute the same plugin output hooks before returning.</comment>
<file context>
@@ -1185,20 +1225,30 @@ def compile_app(
+ prerender_routes=prerender_routes,
+ use_rich=use_rich,
+ ):
+ return True
+
+ progress = make_compile_progress(use_rich)
</file context>
| # Backend workers are recognised through the daemon's on-disk marker | ||
| # (see ``compile_daemon.owns_compilation``): environment set here is | ||
| # not inherited by workers a forkserver started earlier. | ||
| environment.REFLEX_SKIP_COMPILE.set(True) |
There was a problem hiding this comment.
P1: If the daemon exits unexpectedly, backend reload workers keep REFLEX_SKIP_COMPILE=true after the marker is cleared, so _should_compile() never reaches its ownership fallback and subsequent edits cannot regenerate .web. Remove this persistent skip flag and rely on owns_compilation() (or explicitly unset it when the daemon exits).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/reflex.py, line 226:
<comment>If the daemon exits unexpectedly, backend reload workers keep `REFLEX_SKIP_COMPILE=true` after the marker is cleared, so `_should_compile()` never reaches its ownership fallback and subsequent edits cannot regenerate `.web`. Remove this persistent skip flag and rely on `owns_compilation()` (or explicitly unset it when the daemon exits).</comment>
<file context>
@@ -209,6 +209,22 @@ def _run_dev(
+ # Backend workers are recognised through the daemon's on-disk marker
+ # (see ``compile_daemon.owns_compilation``): environment set here is
+ # not inherited by workers a forkserver started earlier.
+ environment.REFLEX_SKIP_COMPILE.set(True)
+
# Start the frontend and backend.
</file context>
| @docpage(set_path=path, t=title, description=description, image=image) | ||
| def out(): | ||
| # Build prop docs during page eval so imports stay cheap. | ||
| components = [ |
There was a problem hiding this comment.
P1: When the incremental cache recompiles only one component-doc page, these deferred calls restart the shared PropDocsState ID sequence for that page. Cached pages retain the old IDs, so state slices and interactive prop controls can collide or update another component's control; derive IDs from stable component/property identity (including the expansion key) instead of the process-global counter, or otherwise preserve the full registration sequence.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/app/reflex_docs/pages/docs/component.py, line 963:
<comment>When the incremental cache recompiles only one component-doc page, these deferred calls restart the shared `PropDocsState` ID sequence for that page. Cached pages retain the old IDs, so state slices and interactive prop controls can collide or update another component's control; derive IDs from stable component/property identity (including the expansion key) instead of the process-global counter, or otherwise preserve the full registration sequence.</comment>
<file context>
@@ -966,6 +959,11 @@ def links(current_page, ll_doc_exists, path):
@docpage(set_path=path, t=title, description=description, image=image)
def out():
+ # Build prop docs during page eval so imports stay cheap.
+ components = [
+ component_docs(component_tuple, previews)
+ for component_tuple in component_list[1:]
</file context>
| reload_dirs=list(map(str, get_reload_paths())), | ||
| # uvicorn's reload filter defaults to *.py only, so markdown/data edits | ||
| # would never trigger a reload; include the content suffixes too. | ||
| reload_includes=["*.py", "*.md", "*.mdx"], |
There was a problem hiding this comment.
P2: This reload-watch change is applied unconditionally to the default immutable dev path and is not gated behind REFLEX_COMPILE_CACHE. With the feature off (the default), uvicorn users now get a backend reload + full recompile whenever any .md/.mdx file under a reload path is edited—behavior they never had before (uvicorn's default reload_includes is ["*.py"]). That contradicts the PR's stated "no breaking changes to the default behavior" guarantee and can trigger unexpected restarts for projects with many markdown/docs files. Confirm this default-path change is intended and, if so, document it; otherwise scope it behind the feature flag or keep it opt-in so the default dev behavior stays unchanged.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/utils/exec.py, line 606:
<comment>This reload-watch change is applied unconditionally to the default immutable dev path and is not gated behind `REFLEX_COMPILE_CACHE`. With the feature off (the default), uvicorn users now get a backend reload + full recompile whenever any `.md`/`.mdx` file under a reload path is edited—behavior they never had before (uvicorn's default `reload_includes` is `["*.py"]`). That contradicts the PR's stated "no breaking changes to the default behavior" guarantee and can trigger unexpected restarts for projects with many markdown/docs files. Confirm this default-path change is intended and, if so, document it; otherwise scope it behind the feature flag or keep it opt-in so the default dev behavior stays unchanged.</comment>
<file context>
@@ -601,6 +601,9 @@ def run_uvicorn_backend(host: str, port: int, loglevel: LogLevel):
reload_dirs=list(map(str, get_reload_paths())),
+ # uvicorn's reload filter defaults to *.py only, so markdown/data edits
+ # would never trigger a reload; include the content suffixes too.
+ reload_includes=["*.py", "*.md", "*.mdx"],
reload_delay=0.1,
)
</file context>
| sys.executable, [sys.executable, "-m", "reflex.utils.compile_daemon"] | ||
| ) | ||
|
|
||
| before = _dependency_snapshot(state) |
There was a problem hiding this comment.
P2: An edit between _next_changes() stopping the watcher and this snapshot can be absent from both changed and pending, leaving .web stale until another edit. Preserve the watcher's last reconciled snapshot and use it as the compile baseline, or capture that baseline before stopping the watcher.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/utils/compile_daemon.py, line 979:
<comment>An edit between `_next_changes()` stopping the watcher and this snapshot can be absent from both `changed` and `pending`, leaving `.web` stale until another edit. Preserve the watcher's last reconciled snapshot and use it as the compile baseline, or capture that baseline before stopping the watcher.</comment>
<file context>
@@ -0,0 +1,1030 @@
+ sys.executable, [sys.executable, "-m", "reflex.utils.compile_daemon"]
+ )
+
+ before = _dependency_snapshot(state)
+ state.roots = roots = _reload_roots()
+ with _compile_lock(root):
</file context>
| env[environment.REFLEX_COMPILE_DAEMON.name] = "1" | ||
| if prerender_routes: | ||
| env["REFLEX_PRERENDER_ROUTES"] = "1" | ||
| mark_daemon_active() |
There was a problem hiding this comment.
P2: When daemon startup fails, this line records the live reflex run PID before any daemon exists, so backend workers can skip frontend compilation for the rest of the session. Start the child first and record its PID only after Popen succeeds, with cleanup for startup failures.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At reflex/utils/compile_daemon.py, line 64:
<comment>When daemon startup fails, this line records the live `reflex run` PID before any daemon exists, so backend workers can skip frontend compilation for the rest of the session. Start the child first and record its PID only after `Popen` succeeds, with cleanup for startup failures.</comment>
<file context>
@@ -0,0 +1,1030 @@
+ env[environment.REFLEX_COMPILE_DAEMON.name] = "1"
+ if prerender_routes:
+ env["REFLEX_PRERENDER_ROUTES"] = "1"
+ mark_daemon_active()
+ proc = subprocess.Popen(
+ [sys.executable, "-m", "reflex.utils.compile_daemon"], env=env
</file context>
| v = page_cache.FileValidator(files) | ||
| assert v.changed(str(f)) is False | ||
| assert reads == 0 # stat matched, never read | ||
| assert v.unchanged([str(f)]) is False or True # memoized, still no read |
There was a problem hiding this comment.
P3: This assertion is a tautology and tests nothing: assert v.unchanged([str(f)]) is False or True parses as assert (v.unchanged([str(f)]) is False) or True, which is always True regardless of unchanged's result. The comment claims it verifies the call is memoized (no file read), but if memoization regressed and a read occurred, this line would still pass. Assert is True instead so the memoization guarantee is actually enforced.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/units/compiler/test_page_cache.py, line 570:
<comment>This assertion is a tautology and tests nothing: `assert v.unchanged([str(f)]) is False or True` parses as `assert (v.unchanged([str(f)]) is False) or True`, which is always True regardless of `unchanged`'s result. The comment claims it verifies the call is memoized (no file read), but if memoization regressed and a read occurred, this line would still pass. Assert `is True` instead so the memoization guarantee is actually enforced.</comment>
<file context>
@@ -0,0 +1,676 @@
+ v = page_cache.FileValidator(files)
+ assert v.changed(str(f)) is False
+ assert reads == 0 # stat matched, never read
+ assert v.unchanged([str(f)]) is False or True # memoized, still no read
+ assert reads == 0
+
</file context>
| assert v.unchanged([str(f)]) is False or True # memoized, still no read | |
| assert v.unchanged([str(f)]) is True # memoized, still no read |
3 participants
An experimental, flag-gated frontend compile pipeline that recompiles only the pages whose source actually changed and reuses the rest. Off by default; enabled by
REFLEX_COMPILE_CACHE. The compile path is completely unchanged when the flag is unset.What it does
On-disk incremental cache (
disk_cache.py+page_cache.py)A fresh compile process reuses the previous build already on disk in
.weband recompiles only the pages whose source changed. Each page is invalidated by a precise dependency set rather than a global timestamp:importlibimports and data read at import time),Pages are keyed by the content hashes of that set plus a small global epoch — Reflex version,
rxconfig/lockfiles, and the app entrypoint's config-only modules (theme, app-wraps, stylesheets, head components, plus config the app imports dynamically at load time). A change to a genuinely-global input forces a full recompile; per-file edits invalidate only the pages that depend on them. Any unsafe condition falls back to a full compile, so a cache miss is never a correctness risk.Warm fork-per-compile hot-reload daemon (
compile_daemon.py)reflex rundev imports the app once, then compiles each edit in an isolated child — forking on POSIX so third-party imports stay warm (falling back to a fresh process elsewhere). Combined with the disk cache, a hot reload skips the cold reimport and rebuilds only what changed.Supporting changes required for safe page reuse: deterministic compile-time ref-name generation, own-before-mutate page-metadata injection, and mirroring memo output paths to their Python source modules.
All Submissions:
Type of change
New Feature Submission:
Changes To Core Features: