feat(harbor): add native Harbor evaluation plugin

[Abhijeet Prasad (AbhiPrasad)]

Resolves SDK-64.

Adds a native Braintrust plugin for Harbor.

Setup

Requires Python 3.12+ and a minimum Harbor version of 0.16.0. The plugin ships in the normal braintrust distribution and Harbor discovers it through the harbor.plugins entry point, so there is nothing to import or register.

pip install harbor braintrust

export BRAINTRUST_API_KEY=...

Enable it by selecting the braintrust plugin:

harbor run \
  -d terminal-bench/terminal-bench-2@latest \
  -a claude-code \
  -m anthropic/claude-sonnet-4-6 \
  -n 32 \
  --plugin braintrust \
  --plugin-kwarg project_name=agent-benchmarks

That is the whole setup. The API key stays in the host process and is never written into Harbor config, results, manifests, or the task container.

Configuration

Options are passed as Harbor kwargs, and every option also has a HARBOR_BRAINTRUST_* environment fallback:

harbor run ... \
  --plugin braintrust \
  --plugin-kwarg project_name=agent-benchmarks \
  --plugin-kwarg 'score_keys=["correctness","pass_*"]' \
  --plugin-kwarg 'reward_rules={"error_rate":{"type":"score","direction":"minimize","min":0,"max":1}}'

# Equivalent, so CI can configure it once:
export HARBOR_BRAINTRUST_PROJECT=agent-benchmarks
export HARBOR_BRAINTRUST_SCORE_KEYS='["correctness","pass_*"]'

Precedence is --plugin-kwarg > HARBOR_BRAINTRUST_* > default. Complex values are JSON because Harbor only accepts flat kwargs.

Option	Values	Purpose
project_name / project_id	string	Target project. Mutually exclusive. Env: HARBOR_BRAINTRUST_PROJECT.
dataset_mode	sync (default), none	Upsert a managed dataset and associate it with the experiment.
trajectory_mode	atif (default), summary, native	Full ATIF import, aggregates only, or skip (agent instrumented elsewhere).
content_mode	metadata, messages (default), full	How much payload to capture. full currently captures the same as messages.
score_keys / metric_keys	JSON glob arrays	Route reward keys to scores vs metrics. Overlap is rejected before any network I/O.
reward_rules	JSON object	Per-reward type, direction, and min/max normalization.
classifier_rules	JSON object	Map a classifier name to a documented result path.
invalid_score_policy	metric (default), drop, error	What to do when a configured score is out of range.
attachments	none, verifier-details (default), all	all also enables artifact_include globs.
max_content_bytes, max_attachment_bytes, max_total_attachment_bytes	ints	Size bounds.
strict	bool (default false)	Raise instead of isolating sync failures, where Harbor permits it.

What lands in Braintrust

Harbor job
├── resolved task selection ──────► Dataset (one per Harbor source)
├── dataset × semantic agent ─────► Experiment
└── final TrialResult ────────────► eval [root]
    ├── task
    │   ├── environment_setup
    │   ├── agent_setup
    │   ├── agent_execution
    │   │   ├── chat.completions.create  [llm]
    │   │   ├── read_file                [tool]
    │   │   └── chat.completions.create  [llm]
    │   └── verification
    ├── correctness  [score, purpose=scorer]
    └── category     [classifier, purpose=scorer]

Datasets. One managed dataset per Harbor source. Each resolved task is one record with a deterministic UUIDv5 ID, so reruns and backfill upsert instead of duplicating. Input holds task-authored semantics (identity, instruction, step instructions); expected stays null rather than treating solution or verifier code as expected output.

Experiments. Partitioned by (dataset identity, semantic agent config, skill digests). Agent, model, kwargs, MCP config, and skills split experiments; concurrency, retry policy, and output paths do not. Each retained final TrialResult is one root row — execution retries do not inflate counts, while intentional n_attempts stay separate rows.

Rewards. Harbor rewards stay authoritative and the raw dict is preserved at metadata.harbor.raw_rewards. Classification is semantic, not range-guessing: an exact reward_rules entry, then a score_keys/metric_keys glob, then conventional reward when it is in [0, 1]; every other numeric reward becomes a metric even if it happens to land in [0, 1]. A missing reward with no exception is flagged as unevaluated rather than scored zero.

ATIF traces are conformance-gated. A step becomes an llm span only with exactly one model call plus provider/model identity, canonical messages, token usage, and valid timing; a tool span needs arguments plus a correlated result or error. Anything deterministic, aggregated, truncated, or incomplete is downgraded to a task summary with a warning on the eval root instead of being mislabeled. Missing timestamps are interpolated inside the agent phase, outliers clamped, and every repair recorded. Subagents become nested task trees.

Privacy. Secret-like keys, credentials, and oversized payloads are redacted or bounded, and absolute paths are stripped from host-side metadata. Paths inside trajectory content are kept, because they name container files the agent actually operated on — redacting them would empty out filesystem tool calls.

Backfill

Backfill reuses the same identity, normalization, partitioning, reward, and ATIF core as live sync, so it reconciles an existing experiment rather than creating a second one. It never reruns trials.

import asyncio
from braintrust.integrations.harbor import backfill_job

asyncio.run(backfill_job("jobs/my-harbor-run", project_name="agent-benchmarks"))

Every job directory also gets a credential-free braintrust-sync.json with the dataset and experiment IDs, per-trial terminal state, retry counts, warnings, and completion state.