Adding explicit job_runs dependency ordering

[hejcman-enverus]

Changes

Add a depends_on field for ordering bundle Job Runs:

resources:
  job_runs:
    prepare:
      job_id: ${resources.jobs.prepare.id}

    publish:
      job_id: ${resources.jobs.publish.id}
      depends_on:
        - ${resources.job_runs.prepare.id}

Dependencies must reference an existing ${resources.job_runs.<name>.id}. A dependent run starts only after all referenced runs finish successfully. The resolved run IDs are stored in local deployment state and are not sent to the Jobs API. When an upstream run receives a new run ID, its dependents are re-triggered. This also resumes waiting for an in-progress upstream run when the previous deployment was interrupted.

Why

The direct deployment engine already orders resources through cross-resource references, and Job Run deployment already waits for each run to finish successfully.

However, there was no explicit way to declare ordering between Job Runs. Users have to place a reference such as ${resources.job_runs.prepare.state.result_state} in an unrelated request field, such as a job parameter. This created the necessary dependency-graph edge, but mixed deployment ordering with data sent to the job.

depends_on provides a clear, local-only mechanism while reusing the existing reference-based dependency graph and cycle detection.

The interrupted-deployment handling is also necessary because an in-progress Job Run is planned as unchanged. Without resuming its abandoned wait, the deployment could release its dependents before that run finished.

Tests

• Added validation tests for valid, malformed, and undefined Job Run dependencies.
• Added planner tests verifying that depends_on creates dependency-graph references.
• Added resource tests covering state persistence and ensuring depends_on is not sent to the Jobs API.
• Extended the interrupted Job Run acceptance test to verify that:
  • downstream runs remain blocked when the upstream wait fails;
  • the next deployment resumes waiting for the existing upstream run;
  • the downstream run starts only after the upstream run succeeds.
• Regenerated the bundle JSON schema, reference schema, and PyDABs models.
• Ran focused Go tests.
• Ran Job Run and reference-schema acceptance tests.
• Ran the PyDABs test suite: 496 passed.
• Ran ./task lint-q, ./task checks, and git diff --check.

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

5 files changed
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

/bundle/ - needs approval

13 files changed
Suggested: @denik
Also eligible: @janniklasrose, @pietern, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

General files (require maintainer)

Files: python/databricks/bundles/job_runs/_models/job_run.py, python/databricks_tests/core/_generated/job_runs.py
Based on git history:

• @denik -- recent work in bundle/direct/dresources/, bundle/direct/, bundle/config/validate/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.}

[denik]

Thanks @hejcman-enverus

We have a plan for a different format that is also applicable to all resources.

Tentative:

lifecycle:
  depends_on:
    - resource: foo # or jobs.foo

[github-actions]

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 6741
• Commit SHA: 2fa35546aa8be0648867f2792fa35362e36f9141

Checks will be approved automatically on success.