[Leaderboard] Stela - deepseek v4.1 flash max - 81.14% Pass@1

[ay27]

Stela — Leaderboard Submission

• Agent name: Stela, https://stelaapp.com/
• Submitter: ay27
• Backbone LLM: deepseek-flash (recorded endpoint model identifier, DeepSeek-V4.1-Flash)
• Reasoning effort: max
• Hints: Yes (db_description_withhint.txt)
• Tuned prompts: Yes (DAB-informed prompts and analysis guidance)
• Coverage: 54 queries across 12 datasets; five outcomes per query, 270 answer/trace pairs
• Local Pass@1: 81.14%, subject to official revalidation

Architecture

Stela is a local-first data analysis application. This submission uses its Agent harness with SQL and MongoDB query tools, a per-trial Python workspace, and model-backed semantic operations for text analysis. The agent explores schemas and data, performs joins, transformations, aggregation and ranking, and uses intermediate results to guide subsequent tool calls.

Analysis evidence contracts help track the intended result shape, coverage and supporting evidence. Strategy review supports reassessment during a query. Semantic operations use configured record, request and token budgets. These capabilities were enabled for this evaluation; they are available to the agent rather than guaranteed to run on every trial.

Evaluation

The submission contains the complete 54-query leaderboard suite with five trials per query. All 270 outcomes, including failed outcomes, are retained; this is a full run rather than a selected repair run.

Pass@1 is computed by averaging success across trials for each query, averaging the query scores within each dataset, and then averaging equally across the 12 datasets. This yields 81.14%. The separate trial-weighted success rate is 227/270 (84.07%).

Dataset	Passing trials	Total trials	Local Pass@1
music_brainz_20k	15	15	1.0000
stockindex	15	15	1.0000
yelp	35	35	1.0000
stockmarket	24	25	0.9600
bookreview	14	15	0.9333
googlelocal	18	20	0.9000
crmarenapro	57	65	0.8769
PANCANCER_ATLAS	10	15	0.6667
GITHUB_REPOS	13	20	0.6500
DEPS_DEV_V1	6	10	0.6000
PATENTS	9	15	0.6000
agnews	11	20	0.5500

Run configuration

• Stela base commit: 953e4574a9949f79c61bda12914493648fa97505
• DAB revision: 881bab89f69ac4f150e2586357ede8c5b95719d5
• Trial concurrency: 3; MongoDB concurrency: 2; Python concurrency: 2
• Bridge timeout: 1200000 ms; task timeout: 3600000 ms
• Semantic transmission, semantic optimization, analysis contracts and strategy review: enabled
• Per-trial semantic budgets: 1000 records, 200 requests, 200000 tokens

Concurrent trials used separate Python workspaces while sharing benchmark database fixtures. Original message timestamps are preserved in the traces.

The model name above is the endpoint identifier recorded by the runner; an exact backend model revision was not independently verified. The deployed source included one additional retained file, src/components/ai/prompt-input-keyboard.ts, beyond the base commit. The exact deployed source snapshot is retained separately for provenance review.

Execution traces

stela-leaderboard-submission.zip contains:

• submission.json: the submitted answers.
• traces.jsonl: 270 records, one per trial, keyed by dataset, query and run.
• README.md: trace format, verification details and recording limitations.

Each trace retains the original ordered application conversation: user messages, assistant reasoning and text, tool calls with their IDs and arguments, tool results and errors, and the exact submitted final answer. Failed tool calls and subsequent corrections are preserved.

All 270 final answers and 10,781 message contents were checked against the original records. All 6,393 tool calls have matching tool results. Internal scheduler logs, grading results and transport metadata are excluded from the reviewer archive.

Recording scope

These are application conversation traces. Per-request system prompts and tool schemas, and complete raw conversations for semantic subcalls, were not captured. Semantic tool inputs and returned results remain visible in the parent conversation. Original archives and the deployed source snapshot are retained separately for provenance review. Please let us know if additional recording is required for verification.

Submission artifacts

• Submission file: leaderboard_submissions/stela.json
• Execution traces: stela-leaderboard-submission.zip
• Trace archive SHA-256: bc1980b6121acf577004819597ceb5027eb96de9e7120b04d60741011d4382ce

stela-leaderboard-submission.zip