Skip to content

experimental/air: lazily page older runs in air list#5811

Open
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-list-pagination
Open

experimental/air: lazily page older runs in air list#5811
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-list-pagination

Conversation

@riddhibhagwat-db

Copy link
Copy Markdown

The interactive air list table only ever held the newest --limit (20) runs, so its ←/→ paging just scrolled that fixed window — older runs (e.g. anything before a recent date) were never fetched from the server and could never be reached.

Fix

Replace the one-shot listAirRuns with a stateful runFetcher that pages Jobs runs/list on demand:

  • It buffers a page's leftover runs and keeps its page-token cursor, so successive next() calls resume where the last stopped.
  • The interactive table (list_tui.go) holds the fetcher and calls next(listPageRows) in the background as the cursor nears the end of the loaded rows (maybeFetchfetchCmdmoreRowsMsg). New rows and their MLflow links are appended and column widths recomputed; only one fetch runs at a time.
  • The hint line shows row N/M plus (loading…) / (load failed).
  • maxListScan (2000) remains the safety ceiling on total runs scanned.

One-shot output paths (JSON, piped, and explicit --limit) are unchanged: they call fetcher.next(limit) once, so acceptance output is identical.

Tests

  • Updated the existing list/model tests to the new runFetcher/newListModel signatures.
  • Added TestRunFetcherResumesAcrossCalls: a next() that stops mid-page buffers the rest, hands it back on the next call, then reports exhaustion — without re-fetching.
  • air list / air get acceptance tests unchanged and green; build + vet clean.

This pull request and its description were written by Isaac.

The interactive table only ever held the newest `--limit` (20) runs, so
its ←/→ paging scrolled the loaded window and could never reach older
runs. Replace the one-shot listAirRuns with a stateful runFetcher that
pages runs/list on demand and buffers a page's leftover runs, so the
table fetches the next batch (and its MLflow links) as the cursor nears
the end. One-shot JSON and piped/`--limit` output is unchanged.

Also tighten a few comments that had drifted after MLflow enrichment was
added.

Co-authored-by: Isaac
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 29208e4

Run: 28607048240

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 10 13 261 1016 7:14
💚​ aws windows 10 13 263 1014 9:30
💚​ aws-ucws linux 10 13 357 930 7:26
💚​ aws-ucws windows 10 13 359 928 11:10
💚​ azure linux 4 15 264 1014 7:03
💚​ azure windows 4 15 266 1012 10:09
💚​ azure-ucws linux 4 15 362 926 7:47
💚​ azure-ucws windows 4 15 364 924 11:09
💚​ gcp linux 4 15 260 1017 7:52
💚​ gcp windows 4 15 262 1015 10:35
23 interesting tests: 13 SKIP, 10 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/replace_existing 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 30 slowest tests (at least 2 minutes):
duration env testname
5:02 gcp windows TestAccept
5:02 aws-ucws windows TestAccept
5:02 aws windows TestAccept
5:02 azure windows TestAccept
4:56 azure-ucws windows TestAccept
4:35 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:33 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:24 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:16 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:25 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:20 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:08 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:05 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:04 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:58 gcp linux TestAccept
2:54 aws linux TestAccept
2:52 azure linux TestAccept
2:50 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:50 aws-ucws linux TestAccept
2:50 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:49 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:49 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:44 azure-ucws linux TestAccept
2:39 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:38 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:38 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:33 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:29 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

if url := m.rows[m.cursor].MLflowURL; url != "" && url != "-" {
return m, openURL(url)
}
if url := m.rows[m.cursor].MLflowURL; url != "" && url != "-" {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops the previous len(m.rows) > 0 guard, so m.rows[m.cursor] will panic if the model is ever created/updated with zero rows.
Is it intended?

m.offset = m.clampedOffset()
return m, m.maybeFetch()

case moreRowsMsg:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lazy-paging logic added here has no model-level test — the list_tui_test.go cases all pass a nil fetcher. A test that drives Update with a moreRowsMsg (asserting rows are appended, loading is cleared, and the error branch sets loadErr) would lock in this behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants