Skip to content

feat(clickpipes): add advanced CDC table mappings - #740

Open
sdairs wants to merge 1 commit into
stack/691-mapping-engine-apifrom
stack/691-cdc-table-mappings
Open

feat(clickpipes): add advanced CDC table mappings#740
sdairs wants to merge 1 commit into
stack/691-mapping-engine-apifrom
stack/691-cdc-table-mappings

Conversation

@sdairs

@sdairs sdairs commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

MySQL, MongoDB, and BigQuery CDC creates can now send the full table-mapping objects supported by ClickHouse Cloud through repeatable --table-mapping-json flags. Simple --table-mapping values remain supported and can be mixed with JSON mappings; requests preserve simple-first ordering, optional and explicitly empty values, MySQL partitionKey versus partitionByExpr, and BigQuery mappings for both service-account and workload-identity authentication.

The CLI strictly validates each source-specific object before reading credential or CA files or issuing an HTTP request. It rejects malformed JSON, unknown fields and engine values, missing or blank identifiers, contradictory custom-sorting settings, blank array entries, and duplicate MySQL arrays. The preceding API prerequisite exposes analyzer-checked engine value lists for CLI validation.

Documentation now includes complete examples for all three sources and explains table engines, sorting-key behavior, and the two MySQL partition fields.

Validation:

  • cargo fmt --all --check
  • cargo clippy -p clickhousectl -- -D warnings
  • cargo test -p clickhousectl (1,085 unit tests passed, 1 ignored; 319 cloud request-shape tests passed, 1 ignored; all local and telemetry binaries passed)
  • cargo check -p clickhousectl --no-default-features
  • cargo clippy -p clickhousectl --all-targets --no-default-features -- -D warnings
  • cargo clippy -p clickhouse-cloud-api -p clickhouse-openapi-analyzer --all-targets -- -D warnings
  • cargo test -p clickhouse-cloud-api -p clickhouse-openapi-analyzer
  • python3 -m unittest discover -s scripts/tests -p 'test_*.py' (90 tests passed in the aggregate; the sole sandbox-blocked temporary-Git fixture passed on an exact outside-sandbox rerun)

Closes #691

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose table mapping options for MySQL, MongoDB and BigQuery ClickPipes

1 participant