Skip to content

perf(migrations): speed up state replay and snapshots - #2274

Open
tsifrer wants to merge 3 commits into
tortoise:developfrom
tsifrer:faster-migrations
Open

perf(migrations): speed up state replay and snapshots#2274
tsifrer wants to merge 3 commits into
tortoise:developfrom
tsifrer:faster-migrations

Conversation

@tsifrer

@tsifrer tsifrer commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Description

This change improves migration performance by avoiding repeated reconstruction of the complete historical model state.

It introduces lightweight migration state snapshots that:

  • Copy mutable ModelState data, including field dictionaries.
  • Reuse already rendered historical model classes until a model changes.
  • Replace changed and relation dependent models non-destructively, preserving the models available through old_state.
  • Preserve the existing old_state and new_state operation API for forwards migrations, rollbacks, custom operations, and SQL collection.

It also changes model reloads so initial querysets are rebuilt only for models that were actually re-rendered. Previously, every operation rebuilt querysets for every model accumulated in the migration state.

Motivation and Context

Migration execution scaled approximately with the number of operations multiplied by the size of the accumulated schema.

Two repeated operations caused most of the overhead:

  1. Real migration and SQL generation operations recreated every historical ORM model when taking an old-state snapshot.
  2. Replaying migration state rebuilt initial querysets for every accumulated model after every operation, even when only one model changed.

This was particularly visible in a large project containing over 2,500 migration operations and approximately 900 models.

Running SQL generation for a late migration previously took approximately three minutes:

tortoise sqlmigrate core 0168_auto_20260908_1042

real 188.59 seconds
user 186.23 seconds
sys    1.40 seconds

With lightweight snapshots alone, the command still took approximately three minutes because state replay, not snapshot creation was the dominant cost for this migration.
After limiting queryset rebuilding to changed models, the same command completed in approximately five seconds:

real 5.05 seconds
user 4.78 seconds
sys  0.16 seconds

How Has This Been Tested?

Testing was performed on macOS arm64 with Python 3.13.14.
The complete migration test directory was run:

pytest tests/migrations -q

results 304 passed, 6 skipped

Regression tests were added to verify that:

  • Snapshots do not re-render unchanged model classes.
  • Mutable ModelState and field dictionaries remain isolated.
  • Removing a field affects only the new state; the field remains available in old_state.
  • Altering models preserves independent old and new rendered models.
  • Foreign key relations point to the appropriate historical model version.
  • Deleted models remain accessible through old_state.
  • Unchanged models do not have their initial querysets rebuilt.
  • Dry-run state replay does not create unnecessary snapshots.

The sqlmigrate command above and the same migrate command was run using an editable installation of this branch to measure the complete CLI path.

Ruff formatting and targeted lint checks also passed.

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added the changelog accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@codspeed-hq

codspeed-hq Bot commented Sep 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 24 untouched benchmarks


Comparing tsifrer:faster-migrations (9d90505) with develop (f6f072b)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant