[jdbc-v2,client-v2] Build callback interface to for spans

[chernser]

Descriptions

Here is semantic convention for DB operations spans https://opentelemetry.io/docs/specs/semconv/db/database-spans/
Here is what client can set in this adapter:

Attribute	Available?	Where from
db.system.name	always	constant clickhouse
db.namespace	always	ClientConfigProperties.DATABASE from merged settings
db.query.text	query/command only	the sqlQuery argument; stream inserts have no user SQL
db.collection.name	insert / getTableSchema only	the tableName argument
db.operation.name	insert/ping/schema only	known by which API was called
db.query.summary	no (client-v2)	requires a SQL parser
server.address / server.port	per attempt; at start only if single-endpoint	Endpoint from nodeSelector
db.response.status_code	on failure	X-ClickHouse-Exception-Code header, read in readError
error.type	on failure	exception class or ClientFaultCause
db.response.returned_rows	on success	result_rows from X-ClickHouse-Summary
db.operation.batch.size	POJO insert only	List#size()
db.query.parameter.	query only	the queryParams map
network.peer.*	no	Apache HttpClient doesn't surface it at this layer
clickhouse.query_id	usually	assigned at Client.java:1819-1821

Goal

Let an application observe every client-v2 operation and every HTTP request the client makes, through a
backend-agnostic SPI, plus one ready-made OpenTelemetry implementation of it.

Constraints

• Nothing is recorded, and nothing changes, when no recorder is configured. This is the default path.
• No new runtime dependency. The OpenTelemetry API must be compile-only, so it is neither packaged with nor
  transitively pulled in by client-v2, and no OpenTelemetry type may leak into any signature outside the
  OpenTelemetry recorder itself. The SPI must be usable with OpenTelemetry absent from the classpath.
• Java 8, public API additions only, no existing signature changed or removed.

Contract to implement

New public package com.clickhouse.client.api.observability, installed via a new
Client.Builder.setSpanRecorder(...) (null means record nothing):

• A recorder starts a span for a query operation, for an insert operation, and for a single transport
  request, and ends spans. Every started span is ended exactly once — on success, on failure, and across
  retries.
• A span receives attributes. Give it a no-op default implementation so the client can record more over
  time without breaking existing recorders, and provide a no-op recorder that doubles as the base class for a
  recorder interested in only some operations.
• Attribute keys and span names follow the OpenTelemetry semantic conventions for database and HTTP client
  spans, so a recorder can forward them to its backend verbatim. Keep those names and the operation-name
  constants on the SPI side, not inside the OpenTelemetry recorder, so every recorder reports the same values.
• Nesting: each HTTP request of an operation, including each retry, is a child span of that operation's
  span. Operation spans join the caller's ambient trace. Both hold only while the operation runs on the calling
  thread — with async_operations the operation span starts a new trace, which is a documented limitation, not
  a bug to solve here.
• The client records, at minimum: the db system, operation name, database, and query id on operation spans; the
  request method, server address and port, URL, and response status on request spans; and an error attribute on
  every failure path, which the OpenTelemetry recorder translates into an error span status.
• An HTTP request span cannot be named after its method until the request exists, so it must be renamed once the
  method is known.

Tests

• Unit-test the OpenTelemetry recorder against an in-memory exporter: span names, kinds, attributes and their
  value typing, parent/child nesting, joining and restoring an ambient context, error status, idempotent end,
  and rejection of bad input. Use a @DataProvider for the attribute-typing cases.
• Add one integration test that runs a real successful query and a real failing query and asserts the resulting
  span tree.
• Fix up the existing tests that construct the internal HTTP helper directly.

Docs

Update CHANGELOG.md and both the feature list and the compatibility-sensitive section of docs/features.md —
the shape of recorded spans (kinds, names, attributes, error semantics, the async caveat) is part of the
contract.

Do not copy these defects from the branch

• Two empty placeholder types in com.clickhouse.client.api.metrics are added and threaded through
  constructors without ever being called. Leave them out; they belong to unrelated metrics work.
• HttpAPIClientHelper.java was CRLF on main and got rewritten as LF, turning a ~30-line change into a
  ~2500-line diff. Preserve existing line endings.
• The query-id attribute key disagrees between the code and docs/features.md; pick one and make code, tests,
  and docs agree.
• server.port is recorded as a String and the db response status is prefixed with "Err:". Record both as
  plain values of the right type.

Acceptance

• mvn -pl client-v2 -am test passes and packages/clickhouse-jdbc-all still builds.
• The published client-v2 POM has no compile-scope OpenTelemetry dependency.
• This is new public API, so per AGENTS.md it needs an approved issue or proposal; link it, or say one is
  missing.

[chernser]

@polyglotAI-bot please implement

[polyglotAI-bot]

Thanks @chernser — happy to implement this. The behavioral spec is clear; before I start I'd like to lock two things, because this introduces a new public-API package (a precedent) and will land well over the AGENTS.md 800-LOC "split into smaller changes" guidance. Agreeing the SPI shape and the slicing up front avoids spending a large PR + your review on a shape that doesn't match what you had on the reference branch (which isn't pushed here, so I'm working from the prose spec only).

1. Proposed SPI shape — com.clickhouse.client.api.observability

// No-op base class: subclass and override only the operations you care about.
public class SpanRecorder {
    // Operation span (query / command / insert / ping / getTableSchema).
    // Recorder joins the caller's ambient trace as the parent.
    public Span startSpan(SpanKind kind, String operationName) { return Span.NOOP; }
    // Child transport-request span under the operation span (one per attempt, incl. retries).
    public Span startRequestSpan(Span operationSpan) { return Span.NOOP; }
}

public interface Span {
    Span NOOP = new Span() {};                    // no-op default
    default void setAttribute(String key, Object value) {}
    default void updateName(String name) {}       // request span renamed once HTTP method is known
    default void setError(String errorType) {}    // OTel recorder maps this to ERROR status
    default void end() {}                          // idempotent; exactly once on success/failure
}

// Names + keys live on the SPI side so every recorder reports identical values.
public final class SpanAttributes {
    public static final String DB_SYSTEM_NAME = "db.system.name";
    public static final String DB_NAMESPACE   = "db.namespace";
    public static final String CLICKHOUSE_QUERY_ID = "clickhouse.query_id";
    // ... db.query.text, db.collection.name, db.operation.name, server.address / server.port,
    //     db.response.status_code, error.type, db.response.returned_rows,
    //     db.operation.batch.size, db.query.parameter.<key> — per your table.
}

Entry point (null = record nothing, default fast path, zero overhead):

Client.Builder setSpanRecorder(SpanRecorder recorder)

Nesting / context model: startSpan(...) captures the caller's ambient context as the parent (OTel: Context.current()), so operation spans join the caller's trace; each transport attempt calls startRequestSpan(operationSpan) to create an explicit child. No OTel type appears in any of these signatures — the OTel recorder module owns all OTel wiring. async_operations starting a fresh trace stays a documented limitation, not addressed here.

Forks I'd like you to confirm (these are where a reference-branch shape would differ):

• end() on Span (try-with-resources friendly) vs. recorder.end(span) (recorder-centric — your text says "a recorder … ends spans"). I lean Span.end().
• Span as an interface with default no-ops + a NOOP constant vs. a no-op base class.
• SpanKind enum (CLIENT / INTERNAL) on the SPI vs. distinct startQuerySpan / startInsertSpan methods.
• Error hook setError(String errorType) + a db.response.status_code attribute on failure — OK, or do you want a richer/typed error signature?

2. Proposed delivery (respecting the ≤800-LOC split)

• PR 1 — SPI + client-v2 instrumentation (no OTel dependency). The observability package above, setSpanRecorder, and span instrumentation across the client-v2 operations (query/command, insert, ping, getTableSchema) plus per-request spans in HttpAPIClientHelper (retry nesting, rename-once-method-known, server.address/server.port, response status, error). Tests use a small in-repo capturing recorder against a real successful + failing query; fixes HttpAPIClientHelperTest. Docs: CHANGELOG.md + docs/features.md. → a fully usable, backend-agnostic SPI.
• PR 2 — OpenTelemetry recorder module. New Maven module (OTel API compile-only; aggregator + packages/clickhouse-jdbc-all wiring), OpenTelemetrySpanRecorder / OpenTelemetrySpan, in-memory-exporter unit tests (@DataProvider attribute typing, nesting, ambient join/restore, error status, idempotent end, bad input) + one integration test (success + failure span tree). Docs: docs/features.md compatibility section documenting the span shape.

If PR 1 still trends over ~800 LOC I'll split it further (SPI + no-op wiring, then the instrumentation).

Decisions I need

1. SPI shape — is the shape above (including the 4 forks) what you want, or should I match a specific shape from your reference branch?
2. Slicing — OK to deliver as the 2 PRs above (per the ≤800-LOC AGENTS.md guidance), or do you prefer a single PR / a different split?

I'll pick this up as soon as you confirm. — @polyglotAI-bot

[chernser]

@polyglotAI-bot
This is good spec and lets implement in two PRs as proposed.
Here is corrections we need to make in spec:

• class SpanAttributes -> enum SpanAttribute
• startSpan should overloaded method with argument QuerySettings or InsertSettings. Implementation should grab properties it needs.
• there should be default interface implementation with NOOP Span implementation - this is to avoid check for null and keep some default things.