Skip to content

feat(clickpipes): expose streaming table definitions - #737

Merged
sdairs merged 1 commit into
stack/586-cdc-scalingfrom
stack/594-streaming-table-definition
Sep 7, 2026
Merged

sdairs merged 1 commit into
stack/586-cdc-scalingfrom
stack/594-streaming-table-definition

Conversation

@sdairs

@sdairs sdairs commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Streaming ClickPipe creates always sent a managed MergeTree destination with no way to select another engine or define partition, primary, and sorting keys. This adds shared --managed-table <true|false> and --table-definition-file <PATH|-> options to Kafka, Kinesis, object-storage, and Pub/Sub creates while preserving the existing managed MergeTree request when both are omitted.

The table-definition file is strictly deserialized into the typed Cloud API model, including engine.type, nullable versionColumnId, columnIds, partitionBy, primaryKey, and sortingKey. Unknown fields at the top level or inside engine, unknown engine variants, malformed JSON, and missing typed fields fail locally before an API request.

Validation:

  • cargo fmt --all -- --check
  • cargo clippy -p clickhousectl -- -D warnings
  • cargo check -p clickhousectl --no-default-features
  • cargo clippy -p clickhousectl --all-targets --no-default-features -- -D warnings
  • cargo test -p clickhousectl

Closes #594

@iskakaushik iskakaushik left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sdairs
sdairs merged commit e4a5c8b into main Sep 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose destination table definition on streaming ClickPipe create (engine, partition/sorting/primary key)

2 participants