Skip to content

chore: Add some HTTP outcalls metrics - #11233

Open
eichhorl wants to merge 5 commits into
masterfrom
eichhorl/http-outcalls-metrics
Open

chore: Add some HTTP outcalls metrics#11233
eichhorl wants to merge 5 commits into
masterfrom
eichhorl/http-outcalls-metrics

Conversation

@eichhorl

@eichhorl eichhorl commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Some metrics and logs for HTTPS outcalls:

  • Add a label to the metric counting delivered flexible errors
  • Add error and informational metrics to the pool manager
  • Add a detailed log whenever a timeout is included in a block
  • Track whenever the payload reaches the responses per payload limit (500)

@github-actions github-actions Bot added the chore label Aug 20, 2026
@eichhorl
eichhorl marked this pull request as ready for review August 20, 2026 11:06
@eichhorl
eichhorl requested a review from a team as a code owner August 20, 2026 11:06
@zeropath-ai

zeropath-ai Bot commented Aug 20, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to cdc1354.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/consensus/src/consensus/metrics.rs
    Change canister_http_flexible_errors_delivered from IntCounter to IntCounterVec and add labeling by type
► rs/https_outcalls/consensus/src/metrics.rs
    Add IntCounterVec metrics for pool_manager_events and pool_manager_errors with type labels, and constants for label type
► rs/https_outcalls/consensus/src/payload_builder.rs
    Introduce grouping of flexible_errors by kind using BTreeMap, and track new per-kind errors
► rs/types/types/src/batch/canister_http.rs
    Add kind() method to map FlexibleCanisterHttpError variants to string labels for metrics
► rs/https_outcalls/consensus/src/payload_builder.rs
    Update logic to increment new per-kind error counts when constructing consensus responses
► rs/https_outcalls/consensus/src/pool_manager.rs
    Add metrics emissions on various failure paths (boundary node lookup, socks proxy resolution, committee lookup, signing, etc.) and events (requests sent, responses received, etc.)
► rs/https_outcalls/consensus/src/pool_manager.rs
    Record additional pool manager events and error scenarios via metrics observations
► rs/https_outcalls/consensus/BUILD.bazel
    Include ic-test-utilities-metrics in test utilities dependencies
► rs/https_outcalls/consensus/Cargo.toml
    Add ic-test-utilities-metrics as a dependency
► rs/consensus/src/consensus/metrics.rs
    Adjust construction of canister_http_flexible_errors_delivered to be a vector with type label
► rs/https_outcalls/consensus/src/payload_builder.rs
    Add logging and metrics integration for timeout and per-kind error cases
► rs/https_outcalls/consensus/src/payload_builder.rs
    Emit warning and metrics when max responses per block is reached
► rs/https_outcalls/consensus/src/payload_builder.rs
    Record per-kind error counts and update tests to reflect BTreeMap-based stats

@eichhorl
eichhorl added this pull request to the merge queue Aug 20, 2026
Comment on lines +54 to +58
pool_manager_metrics: metrics_registry.int_counter_vec(
"canister_http_pool_manager_metrics",
"Canister http pool manager related metrics, by kind.",
&[LABEL_TYPE],
),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The doc comment on the field ("Notable, but expected events observed by the pool manager, by kind") actually suggests a better name:

Suggested change
pool_manager_metrics: metrics_registry.int_counter_vec(
"canister_http_pool_manager_metrics",
"Canister http pool manager related metrics, by kind.",
&[LABEL_TYPE],
),
pool_manager_events: metrics_registry.int_counter_vec(
"canister_http_pool_manager_events",
"Notable, but expected events observed by the pool manager, by kind.",
&[LABEL_TYPE],
),

As is, "metrics" is exceedingly vague. Everything is a metric, around here and in Prometheus.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, done: abcccb7

}

/// Records a notable, but expected event of the given kind.
pub(crate) fn pool_manager_metrics_inc(&self, label: &str) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observe_pool_manager_event?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 👍

}

/// Records a failed operation of the given kind.
pub(crate) fn pool_manager_errors_inc(&self, label: &str) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observe_pool_manager_error?

You could also merge the two IntCounterVecs into one by adding an error="true|false" label. Not sure whether that makes sense in this specific context, but it's an option.

@eichhorl eichhorl Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: cdc1354, I kept both metric separate, which is more consistent with what other protocols do

@eichhorl
eichhorl removed this pull request from the merge queue due to a manual request Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants