ci: build PRs against the pinned DuckDB submodule, not the branch tip

[NickCrews]

ci: build PRs against the pinned DuckDB submodule, not the branch tip

The problem

PR CI doesn't build the DuckDB commit this repo pins. on_pr.yml passes duckdb-sha: ${{ github.base_ref }} into the packaging workflow (and duckdb_git_ref into coverage), which makes the build step do:

cd external/duckdb
git fetch origin
git checkout main   # or v1.5-variegata — whatever the PR targets

So every PR builds against the tip of the matching duckdb/duckdb branch at the moment the workflow runs. In practice:

• A PR goes green, sits for a week of review, then turns red on the next push — not because the diff changed, but because duckdb main moved underneath it. CI results are a function of the wall clock, not of the PR.
• The failure lands on whichever PR author happens to be unlucky, even when the PR touches only this repository and has nothing to do with the upstream change.
• Failures don't reproduce locally, because local builds use the pinned submodule. You have to know to git checkout main inside external/duckdb to see what CI sees.

What this PR changes

Drop the two base_ref overrides from on_pr.yml. The checkout-override steps in packaging_sdist.yml, packaging_wheels.yml, and coverage.yml are already conditional on the input being set, so PR CI simply falls back to building the commit the submodule actually pins. Reproducible: same diff, same result, regardless of when it runs.

Doesn't this lose the early-warning signal against duckdb main?

I don't think so, for two reasons:

1. The pin already tracks duckdb head closely. The duckdblabs bot opens submodule-bump PRs roughly weekly against both main and v1.5-variegata, and they merge regularly. Drift between the pin and the branch tip is bounded to days by that automation, independent of what PR CI builds. Incompatibilities with new duckdb changes surface on the bump PR — which is exactly where they have an owner — instead of on unrelated contributor PRs.

2. Today, nothing in CI ever tests the pinned commit — including the bump PRs themselves. A bump PR's own CI also checks out base_ref, i.e. the moving branch tip, not the SHA it is pinning. With this change, bump-PR CI starts validating the exact commit being pinned, which arguably makes the bump automation more trustworthy, not less.

If a standing canary against duckdb head is still wanted, a small scheduled workflow calling the existing packaging.yml with duckdb-sha: main (filing/updating a tracking issue on failure) would provide that signal without coupling it to contributor PRs. Happy to add that here or in a follow-up if there's interest.

Precedent

duckdb-java already works this way: PR CI (Java.yml) builds only the vendored (pinned) engine sources and never checks out duckdb/duckdb; a separate Vendor.yml workflow advances the vendored copy. This PR brings duckdb-python in line with that model.

For history: the base_ref override dates back to #4 in the first week after the split from duckdb/duckdb, as part of getting CI running at all — I couldn't find any discussion choosing branch-tip builds over pinned builds, so I'm assuming it was incidental rather than intentional. If it was deliberate and I'm missing context, happy to close.

🤖 Generated with Claude Code