feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API

[XuQianJin-Stars]

• Bump opendal / object_store_opendal to 0.57.0 (in the new vortex-object-store-opendal crate); this version enables COS versioning by default.
• Use the upstream pyo3-object_store 0.11.0 from crates.io (no longer vendored via [patch.crates-io]). The built-in store classes (S3/Azure/GCS/HTTP/Local/Memory) keep their PyObjectStore interop, and obstore users can still pass external stores.
• Introduce vortex-object-store-opendal, a new workspace crate that builds an object_store::ObjectStore from an OpenDAL services::Cos OpendalStore (make_opendal_store(url, properties)), reused by both the URL registry and the Python store class.
• Add a CosStore pyclass (opendal feature) that wraps an OpenDAL-backed Tencent COS OpendalStore and holds it as Arc<dyn object_store::ObjectStore>. It is constructed with explicit config (bucket, endpoint, secret_id, secret_key, root, disable_config_load) and can be handed to vortex.io.read_url(store=...) / vortex.io.write(..., store=...).
• Introduce AnyVortexStore in io.rs so read_url / write accept both the built-in pyo3-object_store classes and the new OpenDAL-backed CosStore via a single store= argument.
• cos:// and oss:// URL resolution now go through the OpenDAL-backed store, configured via OpenDAL environment variables (TENCENTCLOUD_SECRET_ID / TENCENTCLOUD_SECRET_KEY / COS_ENDPOINT, and ALIBABACLOUD_ACCESS_KEY_ID / ALIBABACLOUD_ACCESS_KEY_SECRET / OSS_ENDPOINT).
• Document the standalone-store usage and add unit tests (cos_store_builds_object_store).

Rationale for this change

Closes: #8844

Vortex's Python API already exposes concrete, standalone store objects for S3/Azure/GCS/HTTP/Local/Memory, but Tencent COS and Alibaba OSS were only reachable through URL-based resolution that relied on environment variables. This adds first-class OpenDAL-backed stores so COS/OSS can be used exactly like the other stores (construct once, pass via store=), with better ergonomics and type checking.

What changes are included in this PR?

• new vortex-object-store-opendal crate (OpenDAL services-cos, opendal/object_store_opendal 0.57.0)
• pyo3-object_store pinned to the upstream 0.11.0 (object_store 0.13.x, matching the workspace) — vendored fork removed
• new CosStore pyclass (gated on the opendal feature) wrapping an OpenDAL COS OpendalStore, holding Arc<dyn object_store::ObjectStore>
• AnyVortexStore extractor in io.rs so read_url / write accept the new store
• cos:// / oss:// URL resolution backed by OpenDAL
• docs + unit tests for the standalone-store path

What APIs are changed? Are there any user-facing changes?

New public API (Python, opendal feature):

• vortex._lib.CosStore(bucket, endpoint, *, secret_id=None, secret_key=None, root=None, disable_config_load=False) — a standalone ObjectStore you can pass to vortex.io.read_url(url, store=...) / vortex.io.write(arrays, path, store=...).
• vortex.store.CosStore / vortex.store.from_url now document cos:// and oss:// OpenDAL-backed support.

[XuQianJin-Stars]

Hi @robert3005, could you please help review this PR when you have a moment? Thanks!

[codspeed-hq]

Merging this PR will not alter performance

✅ 1849 untouched benchmarks
⏩ 46 skipped benchmarks¹

---

_{Comparing XuQianJin-Stars:feature/opendal-cos (952fcb6) with develop (b4f8d34)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

Thanks for this — and for picking a genuinely good integration shape.

The approach is right, and I want to say that clearly before the list of problems. The two things that could have sunk an OpenDAL bridge both check out: object_store_opendal's OpendalStore implements real ranged reads (get_opts, honouring Bounded/Offset) and real multipart writes (put_multipart_opts), which is what vortex-io/src/object_store/write.rs:35 needs. That means a new provider costs ~160 lines in a leaf crate behind an off-by-default feature, with no changes to vortex-io at all. That's a shape we'd be happy to maintain.

The implementation isn't there yet. Two things need to be fixed before this can land in any form:

• import vortex is broken in every build. vortex-python/python/vortex/store/_cos.py has an IndentationError at line 68, and the chain vortex/__init__.py → vortex/file.py → vortex/store/__init__.py → ._cos is unconditional. Even after fixing the syntax, line 29's from vortex._lib import CosStore raises ImportError in any build without --features opendal, which includes every maturin wheel.
• The cos:// flow the PR documents cannot succeed. make_cos_store hard-requires endpoint in properties and never reads COS_ENDPOINT, while Registry::resolve passes an empty map. I reproduced this with a scratch test on your branch: with COS_ENDPOINT, TENCENTCLOUD_SECRET_ID and TENCENTCLOUD_SECRET_KEY all set, it fails with missing required OpenDAL store configuration: endpoint.

Behind those, the design question worth settling first: the PR ships two different CosStore classes — the Rust pyclass (a real store) and a pure-Python os.environ mutator that shadows it. The env-var one is what vortex.store.CosStore resolves to, what the ObjectStore type alias advertises, and what the new docs autoclass, but read_url(store=...) rejects it at runtime, so the docs demonstrate a TypeError. I'd suggest keeping the pyclass and dropping the env-var helper; process-global env mutation is a side channel no other store has, and it races the Rust side's std::env::vars() read in Registry::resolve.

I've left 14 inline comments. Grouped by kind:

Blocking

#	Where	Issue
1	store/_cos.py:68	IndentationError; also Any and os unimported
2	store/_cos.py:29	unconditional import of a feature-gated pyclass
3	store/_cos.py:57	two conflicting CosStore designs
4	vortex-object-store-opendal/src/lib.rs:87	cos:// env-var configuration always fails
5	vortex-object-store-opendal/src/lib.rs:74	oss:// advertised in 4 places, implemented in none
6	vortex-python/src/opendal_store.rs:98	cargo clippy --all-features --all-targets fails
7	docs/api/python/store/opendal.rst:3	short title underline fails --fail-on-warning docs build
8	store/__init__.py:26	ruff format --check fails

Correctness and design

#	Where	Issue
9	vortex-jni/src/object_store.rs:57	COS bypasses the store cache; new Operator per request
10	vortex-python/src/object_store/registry.rs:85	registry precedence inverted; no caching
11	docs/api/python/store/opendal.rst:61	store= example passes a URL where a store-relative key is required
12	vortex-python/src/opendal_store.rs:86	synthetic cos://bucket URL to reach the builder
13	vortex-object-store-opendal/src/lib.rs:108	ClientOptions silently inapplicable to COS
14	vortex-python/src/opendal_store.rs:38	pyclass should be frozen

Smaller points, not worth inline threads

• Dependency weight. This adds a second full HTTP/TLS stack — reqwest 0.13.4 with rustls-platform-verifier alongside the existing reqwest 0.12.28 with rustls-native-certs — plus opendal-core, opendal-service-cos, three reqsign-* crates, mea, and js-sys/wasm-bindgen pulled into jiff. 145 new Cargo.lock lines. That's acceptable behind an off-by-default feature, but note CI's --all-features clippy job pays the compile cost on every run.
• opendal and object_store_opendal are version-pinned inline in the new crate rather than declared in [workspace.dependencies], unlike every other third-party dependency.
• vortex-utils is added to vortex-python as a non-optional dependency but only used under cfg(feature = "opendal"). Also two spellings of the same type: vortex::utils::aliases::hash_map::HashMap in registry.rs vs vortex_utils::aliases::hash_map::HashMap in opendal_store.rs — please pick one.
• write and VortexWriteOptions.write now accept AnyVortexStore, but only read_url's store : docstring line was updated to mention CosStore.
• Test coverage: cos_derives_bucket_from_host asserts only "not a MissingConfig error" and its comment ("build fails before we can assert bucket derivation") is wrong — I confirmed the build succeeds without credentials. Nothing covers the cos:// registry resolution path, which is the path that's actually broken, and there are no Python tests for CosStore.
• Stray blank line inserted between the read_url doc comment and #[pyfunction], and two trailing blank lines appended to the root Cargo.toml.
• The PR description says the vendored pyo3-object_store fork and [patch.crates-io] were removed, but develop already has neither — that part is a no-op in this diff. Worth trimming from the description so reviewers aren't looking for it.

Checks I ran (against 1e0e128 rebased on develop @ a12c310)

Check	Result
cargo check -p vortex-object-store-opendal --all-targets	pass
cargo test -p vortex-object-store-opendal	3 passed
cargo check -p vortex-python --features opendal --all-targets	pass
cargo clippy -p vortex-object-store-opendal -p vortex-python --features vortex-python/opendal --all-targets	fail (#6)
python -m py_compile _cos.py	fail (#1)
ruff check / ruff format --diff	fail (#1, #8)
docutils parse of opendal.rst	fail (#7)
env-var cos:// resolution repro (scratch test, reverted)	fail (#4)

Not run: full Sphinx build, pytest, and basedpyright — all blocked by the import failure. No COS credentials available, so nothing end-to-end.

Happy to look again once the import break and the endpoint plumbing are sorted, and glad to discuss the one-CosStore-or-two question first if that helps — it changes what the docs and the type alias should say.

---

Generated by Claude Code

[robert3005]

Thanks for the round of fixes — 12 of the 14 items from my earlier review are addressed and verified on fe4d545: the _cos.py syntax error, the unconditional pyclass import (now a guarded try/except ImportError with an actionable message), the two conflicting CosStore designs, the unresolvable cos:// endpoint, the oss:// claims, the clippy tests_outside_test_module lint, the RST underline, ruff format, the JNI cache bypass, the store= docs examples, the synthetic-URL round-trip (nice — CosConfig + make_cos_store is the right shape), the ClientOptions limitation, and frozen. The smaller points came along too: vortex-utils gated on the feature, one spelling of vortex_utils, updated write docstrings, taplo sorting.

Four things left, one of them a regression the caching change introduced.

The COS caching in Registry::resolve is broken and shouldn't exist as a separate path. The second resolve of a cos:// URL returns an empty path, and a sibling object in the same bucket still rebuilds the client. The underlying reason is worth more than the fix: cos:// is a bucket-in-authority scheme exactly like s3://, and s3:// already caches at depth 0 through the generic formula — so COS needs no bespoke caching and no hand-written depth. Details and a verified patch in the inline comment.

A flaky test in vortex-object-store-opendal (fails ~3 of 4 cargo test runs; nextest's process isolation masks it, so CI may not catch it).

3 new basedpyright errors, gated by CI at .github/workflows/ci.yml:94, because the new pyclass has no entry in vortex/_lib/__init__.pyi.

The docs page documents the placeholder, not the real class, since no build enables the opendal feature — so the rendered parameters contradict the prose beside them. The same stub fixes this one.

Checks I ran on fe4d545: cargo clippy -p vortex-python --all-targets (with and without --features opendal), cargo test -p vortex-object-store-opendal, cargo test -p vortex-python --lib, py_compile, ruff check, ruff format --check, basedpyright vortex-python, a docutils parse of the new RST, and a live import vortex + CosStore(...) on a default build to confirm the ImportError guard. I did not run a full Sphinx build, so the docs finding comes from inspecting what autodoc resolves to rather than from rendered HTML.

---

Generated by Claude Code

[robert3005]

We can merge this once we fix the python lint failures. Thanks for the contribution!

[robert3005]

Looks like you will also need dco remediation commit

[XuQianJin-Stars]

Looks like you will also need dco remediation commit

done