ability to share a cache amongst different transfers

[swelborn]

---

This is part 13 of 23 in a stack made with GitButler:

•  23  use historical=True as a fallback if job not found #52
•  22  test coverage for credential boundary #51
•  21  remove "force" option for cache shutdown #50
•  20  client/service/workflow changes for IRI token delegation changes #49
•  19  add cache table to track caches #48
•  18  authenticated principal + encrypted delegated-credential capture #47
•  17  OIDC transfer ownership and delegated user credentials #46
•  16  WIP for shared cache shutdown #45
•  15  add jobspec override #44
•  14  add mostly claude-gen usage example package #43
•  13  ability to share a cache amongst different transfers #42 👈
•  12  add precommit hook for checking openapi client drift #40
•  11  pin python version to 3.14 #41
•  10  build frontend image in CI #39
•  9  add frontend #38
•  8  add frontend client (autogenerated) #37
•  7  lazily construct settings #36
•  6  add autogenerated client #35
•  5  add alembic #34
•  4  add ability to get logs from v2 api services thru iri #29
•  3  functional core tests for transfer and producer #30
•  2  api with durable execution and saga #28
•  1  deps for v2 #27

[Copilot]

Pull request overview

Adds support for selecting whether transfers use a per-transfer cache or a shared per-experiment cache, and persists enough transfer context (experiment/run + cache mode) to make provisioning and log resolution deterministic without re-parsing request parameters.

Changes:

• Introduces CacheMode (per_transfer vs shared) across API models, DB schema, and provisioning/teardown logic.
• Persists experiment, run, and cache_mode on transfers, including Alembic migrations to backfill existing rows.
• Updates fastcache integration to use a string key, and adjusts log path resolution to account for shared-cache log location; regenerates OpenAPI clients.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/v2/test_transfer_core.py	Updates provisioning progress tests for cache-mode-aware compensation behavior.
tests/v2/conftest.py	Minor formatting change (blank line).
src/lclstream_api/v2/workflows.py	Implements shared-cache provisioning semantics, cache key selection, and teardown rules.
src/lclstream_api/v2/tables.py	Adds experiment, run, and cache_mode columns to transfers.
src/lclstream_api/v2/service.py	Extends transfer creation to persist experiment/run/cache_mode; updates log resolution to use stored context.
src/lclstream_api/v2/routers/v1/transfer.py	Passes parsed exp/run and requested cache mode into service layer on create.
src/lclstream_api/v2/repo.py	Persists experiment/run/cache_mode when inserting transfers.
src/lclstream_api/v2/models.py	Adds cache_mode to create request and includes experiment/run/cache_mode in public/detail responses.
src/lclstream_api/v2/core/transfer.py	Makes cache compensation conditional on cache mode; adds TransferSetup and cache_mode in resource refs.
src/lclstream_api/v2/core/producer.py	Defines CacheMode and adds shared_cache_dir helper.
src/lclstream_api/v2/core/logs.py	Adds cache-mode-aware cache log path resolution and refactors log path helpers.
src/lclstream_api/v2/clients/fastcache.py	Switches cache creation payload from transfer_id to string key and updates response model.
src/lclstream_api/v2/alembic/versions/aa298396d6a0_add_cache_mode_to_transfers.py	Migration to add cache_mode to transfers with default.
src/lclstream_api/v2/alembic/versions/0236c97ae695_add_experiment_and_run_to_transfers.py	Migration to add/backfill experiment and run from parameters.
openapi.json	Publishes CacheMode schema and adds experiment/run/cache_mode fields to transfer schemas.
frontend/src/client/zod.gen.ts	Regenerates frontend zod schemas/types to include CacheMode and new fields.
frontend/src/client/types.gen.ts	Regenerates frontend TS types for CacheMode and transfer fields.
frontend/src/client/schemas.gen.ts	Regenerates frontend JSON-schema constants to include CacheMode and new fields.
frontend/src/client/index.ts	Re-exports new CacheMode type from generated client.
client/src/lclstream_api_client/_generated/models/transfer_public.py	Regenerates Python client model to include experiment/run/cache_mode.
client/src/lclstream_api_client/_generated/models/transfer_detail.py	Regenerates Python client model to include experiment/run/cache_mode.
client/src/lclstream_api_client/_generated/models/transfer_create.py	Regenerates Python client model to include optional cache_mode in request.
client/src/lclstream_api_client/_generated/models/cache_mode.py	Adds generated Python enum for CacheMode.
client/src/lclstream_api_client/_generated/models/init.py	Exports generated CacheMode model.
client/src/lclstream_api_client/_generated/init.py	Exposes generated CacheMode at package root.

Comments suppressed due to low confidence (1)

src/lclstream_api/v2/workflows.py:303

• In shared cache mode, cache_log_path points to shared_cache_dir(...), so _create_cache no longer implies the per-transfer work_dir exists. Recording DeleteWorkDir(work_dir=...) before any IRI operation creates that directory can make rollback attempt to delete a non-existent path and fail compensation. Consider only adding the work_dir compensation after a step that guarantees the directory exists (e.g. after _upload_config for shared mode).

        work_dir = pcore.transfer_work_dir(
            config.get_producer(), setup.exp, setup.run, transfer_id
        )
        # if _create_cache succeeds, then we know the working directory exists
        progress = progress.with_work_dir(work_dir).with_cache(
            endpoint.cache_id, mode=setup.cache_mode
        )
        await _save_cache(transfer_id, endpoint)

        inputs = await _load_producer_inputs(transfer_id, endpoint)
        if inputs is None:
            raise LookupError(f"transfer {transfer_id} disappeared during setup")
        plan = pcore.plan_producer(inputs, config.get_producer(), transfer_id)

        await _upload_config(plan.config_path, plan.config_yaml)
        progress = progress.with_config(plan.config_path)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.