feat(fastdeploy): OpenTelemetry instrumentation for fast-deploy (deco.deploy.*) - #35
Open
hugo-ccabral wants to merge 5 commits into
Open
feat(fastdeploy): OpenTelemetry instrumentation for fast-deploy (deco.deploy.*)#35hugo-ccabral wants to merge 5 commits into
hugo-ccabral wants to merge 5 commits into
Conversation
Add OTLP traces + metrics for the tanstack-kv fast-deploy pipeline, following
the deco o11y model (deco.deploy.*). Signals export to the in-cluster gateway
collector (OTEL_EXPORTER_OTLP_ENDPOINT) and land in ClickHouse/ClickStack.
- internal/telemetry: opt-in, fail-open OTel SDK bootstrap (no-op when no
endpoint is set), custom ID generator so the operator + syncer form one
distributed trace, and the fast-deploy instruments/recorders.
- Metrics (100%, alert source): deco.deploy.duration (end-to-end incl. k8s
setup), deco.deploy.schedule.duration (scheduling + cold start, from the
Pod), deco.deploy.count{status}, deco.deploy.active.
- Trace: operator mints a traceparent, injects it into the sync Job env +
annotation, and emits root (deco.deploy) + schedule child spans with explicit
timestamps derived from the Job/Pod. The syncer continues the same trace.
- Recording is guarded to the Syncing→terminal transition (no double count).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The syncer Job is created dynamically by the operator, so it can't get OTEL_EXPORTER_OTLP_ENDPOINT from Helm. Propagate the operator's endpoint (and CLUSTER_NAME) into the Job env so the syncer exports its spans to the same in-cluster collector and joins the operator's trace. Gated on the endpoint being set, so both stay no-ops without a collector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ready-to-build panel + alert spec (build needs the Grafana MCP / live datasource). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instruments the tanstack-kv fast-deploy pipeline with OpenTelemetry (traces +
metrics), following the deco o11y model (
decocms/context04_engineering/o11y—
deco.deploy.*). Signals export to the in-cluster gateway collector and landin ClickHouse/ClickStack + Grafana.
Fail-open: inert unless
OTEL_EXPORTER_OTLP_ENDPOINTis set — no behaviorchange without a collector.
What
internal/telemetry: opt-in OTLP SDK bootstrap; custom ID generator so theoperator + syncer form one distributed trace; fast-deploy instruments.
deco.deploy.duration(Job created →terminal, incl. k8s setup),
deco.deploy.schedule.duration(scheduling +cold start, from the Pod),
deco.deploy.count{status},deco.deploy.active.Labels
site/status/target. Recorded once on the Syncing→terminaltransition (guarded, no double-count).
TRACEPARENT, injects it + the OTLP endpoint into the syncJob; emits
deco.deployroot +deco.deploy.schedulechild with explicittimestamps from the Job/Pod. The syncer (separate PR) continues the trace.
docs/fast-deploy-webhook.md§4 (activation) +docs/fast-deploy-dashboard.md(dashboard/alert spec).Activate
Set on the operator (
deco-operatorSM secret,envFromwires it):OTEL_EXPORTER_OTLP_ENDPOINT=http://gateway-otlp.opentelemetry-collector:4317(+ optional
CLUSTER_NAME).Paired PRs
decocms/infra_applications: decofile-syncer spansdecocms/context:deco.deploy.*conventionsBuild, vet, and tests pass.
🤖 Generated with Claude Code
Summary by cubic
Adds OpenTelemetry traces and metrics to fast-deploy (
deco.deploy.*) and exports to the in-cluster OTLP collector; inert unlessOTEL_EXPORTER_OTLP_ENDPOINTis set. Propagates aTRACEPARENTand the OTLP endpoint to the sync Job so operator and syncer share one distributed trace, and ships an importable Grafana dashboard.New Features
internal/telemetry/*; non-fatal on init errors.deco.deploy.duration,deco.deploy.schedule.duration,deco.deploy.count{status},deco.deploy.activewith labelssite,status,target; recorded once on the Syncing→terminal transition.TRACEPARENT, injects it into Job env anddeco.cx/traceparent; emitsdeco.deployroot anddeco.deploy.schedulechild with explicit timestamps; custom ID generator keeps operator + syncer in one trace.OTEL_EXPORTER_OTLP_ENDPOINTand optionalCLUSTER_NAMEso the syncer exports to the same collector.docs/fast-deploy-webhook.md; dashboard + alerts spec indocs/fast-deploy-dashboard.md; importable Grafana dashboard JSON indocs/fast-deploy-dashboard.jsonwith example alert queries.Migration
OTEL_EXPORTER_OTLP_ENDPOINT=http://gateway-otlp.opentelemetry-collector:4317on the operator; optionally setCLUSTER_NAME.Written for commit edb9045. Summary will update on new commits.