Skip to content

[Pre-review] datalens agent (DeepSeek-V4-Pro) historical 88.7753% submission - #106

Open
24kGarry wants to merge 2 commits into
ucbepic:mainfrom
24kGarry:main
Open

[Pre-review] datalens agent (DeepSeek-V4-Pro) historical 88.7753% submission#106
24kGarry wants to merge 2 commits into
ucbepic:mainfrom
24kGarry:main

Conversation

@24kGarry

Copy link
Copy Markdown

This is a transparent pre-review request for an unchanged historical run. We are not asserting final leaderboard eligibility.

Agent configuration

  • Agent: datalens agent (official-scaffold-v23)
  • Team: datalens
  • Backbone: deepseek-v4-pro via an OpenAI-compatible gateway
  • Hints: Yes
  • Tuned prompt: Yes
  • Max iterations: 100
  • Coverage: 12 datasets, 54 queries, 5 runs/query (270 trials)

Local result

  • Dataset-stratified Pass@1: 0.8877533577533577 (88.7753%)
  • Raw passing trials: 242/270
  • Frozen batch: frozen-20260906T230532Z-4745e3f14a
  • Results SHA-256: 95ebb22d8efcf60f84e809aa9e7c9dd903f733e999fb197f9aa6ac579bda53db

All 270 submitted answers match the corresponding final answers in the supplied traces. The score is local and subject to official re-scoring.

Disclosed limitations

  • A conservative local prompt screen flagged 55 historical trials; all were classified as requiring a fresh clean run.
  • 224 compact historical traces lack the newer full-result artifact manifests; missing payloads were not reconstructed or fabricated.
  • The historical aggregate dirty-tree/bytecode fingerprint is not reproducible; recovered prompt-producing source exact-matched 270/270 system and user messages.

Full details are in leaderboard_submissions/datalens_deepseek-v4-pro_submission.md and the trace bundle.

Could you advise whether this historical evidence is useful for pre-review and whether a fresh clean 270-run batch is required for leaderboard eligibility?

Historical 88.7753% DAB pre-review package for team datalens.
Add complete frozen-batch LLM/tool event logs and update verification metadata.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant