Skip to content

fix: Register custom metrics from eval config in LocalEvalSampler and centralize registration#6424

Open
ItsMacto wants to merge 1 commit into
google:mainfrom
ItsMacto:fix-custom-metrics-registration
Open

fix: Register custom metrics from eval config in LocalEvalSampler and centralize registration#6424
ItsMacto wants to merge 1 commit into
google:mainfrom
ItsMacto:fix-custom-metrics-registration

Conversation

@ItsMacto

@ItsMacto ItsMacto commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:
Custom metrics declared in eval_config.custom_metrics are only registered into the metric evaluator registry by an inline loop inside the adk eval CLI command. The adk optimize CLI and programmatic LocalEvalSampler users never run that loop, so any evaluation involving a custom metric fails with NotFoundError: <metric_name> not found in registry. Notebook users currently have to copy the CLI internals to work around it.

Solution:
Centralize the registration in a new helper and call it from LocalEvalSampler.__init__, as proposed in the issue:

  • New register_custom_metrics_from_config(eval_config, metric_evaluator_registry=None) in evaluation/metric_evaluator_registry.py. It reproduces the CLI loop's behavior exactly: the dict key overrides metric_info.metric_name, entries without a metric_info get a default one with a [0.0, 1.0] interval, and each entry registers _CustomMetricEvaluator. It registers into DEFAULT_METRIC_EVALUATOR_REGISTRY unless a registry is passed, and returns the registry it used.
  • LocalEvalSampler.__init__ calls the helper, so both adk optimize and direct construction get custom metrics registered up front, and a bad custom function path surfaces at construction time rather than mid run. The sampler keeps the returned registry and passes it explicitly to LocalEvalService, mirroring what the eval CLI does.
  • The adk eval command now calls the helper instead of the inline loop. No behavior change on that path.
  • get_default_metric_info moved from cli/cli_eval.py into evaluation/metric_evaluator_registry.py, since the evaluation package cannot import from cli. Its only caller was the removed loop; anything importing it from the old cli location would need to switch to the new one.
  • Constructing several samplers with the same metric name follows the registry's existing semantics: the entry is overwritten and an INFO log is emitted.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests in tests/unittests/evaluation/test_metric_evaluator_registry.py cover the helper directly: a provided metric_info is registered under the dict key (with the original name absent from the registry), a missing metric_info produces the default [0.0, 1.0] interval with the description propagated, an eval config without custom metrics is a no-op, and the helper defaults to DEFAULT_METRIC_EVALUATOR_REGISTRY when no registry is passed. A new test in tests/unittests/optimization/local_eval_sampler_test.py covers the fix end to end at the unit level: constructing a LocalEvalSampler whose eval config declares a custom metric makes that metric resolvable from the registry.

$ pytest tests/unittests/evaluation/test_metric_evaluator_registry.py tests/unittests/optimization/local_eval_sampler_test.py -q
30 passed

A wider sweep over tests/unittests/evaluation, tests/unittests/optimization and the CLI eval tests produces the same results as main (no new failures). pyink --check and isort --check-only are clean on all touched files, and mypy reports no new errors compared to main.

Manual End-to-End (E2E) Tests:

I ran adk optimize end to end against a minimal agent (gemini-2.5-flash, 2 eval cases) whose sampler config declares a custom metric:

"criteria": {"response_present_score": 0.5},
"custom_metrics": {
  "response_present_score": {
    "code_config": {"name": "hello_agent.custom_metrics.response_present_score"}
  }
}

On current main the run crashes with the error from the issue:

ERROR - local_eval_service.py:391 - Metric evaluation failed for metric
`response_present_score` for eval case id 'sum_question' with following error
`response_present_score not found in registry.`
...
    raise NotFoundError(f"{eval_metric.metric_name} not found in registry.")

which then surfaces as TypeError: type NoneType doesn't define __round__ method in _extract_eval_data and exit code 1.

With this branch, the identical command completes with exit code 0, no registry errors, the custom metric scoring both eval cases:

INFO - local_eval_sampler.py:64 - Evaluation summary: 2 PASSED, 0 FAILED
================================================================================
Optimized root agent instructions:
--------------------------------------------------------------------------------
You are a helpful assistant. Answer questions concisely.

One note for anyone reproducing this: the custom metric module must be importable at evaluation time (for example via PYTHONPATH); the agent loader does not put the agents dir on sys.path for metric functions. That applies equally before and after this change.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

The registry's _registry dict is a class attribute, so all MetricEvaluatorRegistry instances share one mapping. This PR does not change that; the new tests clean up the entries they register so the shared state stays untouched across tests.

@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Jul 17, 2026
@ItsMacto
ItsMacto force-pushed the fix-custom-metrics-registration branch from 3f294b8 to ce20918 Compare July 17, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support automatic custom metrics registration in LocalEvalSampler and 'adk optimize' CLI

3 participants