Skip to content

feat(clickpipes): expose source tuning options - #731

Open
sdairs wants to merge 1 commit into
stack/584-postgres-prometheusfrom
stack/593-source-tuning
Open

feat(clickpipes): expose source tuning options#731
sdairs wants to merge 1 commit into
stack/584-postgres-prometheusfrom
stack/593-source-tuning

Conversation

@sdairs

@sdairs sdairs commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

ClickPipe create commands could not send several source settings that the Cloud API supports. Kafka also dropped exactlyOnce, had no inline Protobuf schema path for create or discovery, and could not build the Azure Event Hubs connection-string credential shape.

This change exposes the missing controls:

  • MySQL create accepts nullable-column, delete-on-merge, compression, CDC batch/interval, and snapshot parallelism settings.
  • MongoDB create accepts delete-on-merge, native JSON, CDC batch/interval, snapshot parallelism, and certificate-verification controls.
  • Kafka create accepts --exactly-once; create and schema discovery accept a Protobuf schema file or stdin, base64-encode it, enforce the encoded 1 MiB API limit, and reject it outside Protobuf or alongside schema-registry flags.
  • Kafka create and discovery accept Azure Event Hubs connection strings and send the { "connectionString": ... } credential object with PLAIN authentication.

The request builders preserve omission for every optional setting, preserve explicit false, enforce the API's integer minima before file or HTTP work, and validate Kafka/MySQL/MongoDB enum inputs against analyzer-checked API VALUES. Kafka workload identity remains reserved for #690.

Validation:

  • cargo fmt --all --check
  • cargo clippy -p clickhousectl -- -D warnings
  • cargo check -p clickhousectl --no-default-features
  • cargo clippy -p clickhousectl --all-targets --no-default-features -- -D warnings
  • cargo test -p clickhousectl --bin clickhousectl cloud::clickpipes::tests (139 passed)
  • cargo test -p clickhousectl --test cli_request_shape_test issue_593 (5 passed)
  • cargo test -p clickhousectl (full suite passed; 2 explicitly ignored tests)

Closes #593.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose remaining per-source options on clickpipe create (MySQL/MongoDB settings, MongoDB TLS, Kafka exactly-once)

1 participant