[Leaderboard] Qwen3.8-Flash-Next (DAB scaffold, hints): 70.87% Pass@1 - #99
Open
qiboyu301-crypto wants to merge 1 commit into
Open
[Leaderboard] Qwen3.8-Flash-Next (DAB scaffold, hints): 70.87% Pass@1#99qiboyu301-crypto wants to merge 1 commit into
qiboyu301-crypto wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This submits Qwen3.8-Flash-Next running the DAB DataAgent scaffold for leaderboard review.
Qwen/Qwen3.8-Flash-Next, self-hosted with vLLM on 8 NVIDIA L40S GPUs.The submission directory is
leaderboard_submissions/qwen38flash_parallel_v1/. It contains the 270 verbatim answers, per-trial scores, failure records, run configuration, deployment metadata, archive checksum, reproduction instructions, and an independent verification script.The complete execution traces, generated code/intermediates, batch/session records, and actual harness source overlay are available in the trace release. The archive includes all 270 final-agent, LLM-call and tool-call logs. Its manifest records original and published hashes; credentials and text-file deployment locations are redacted, with answer strings unchanged.
Validation
881bab89f69ac4f150e2586357ede8c5b95719d5.Failures and data-access disclosure
agnews/query4/run1–4andmusic_brainz_20k/query3/run2exhausted the 262144-token context budget while requesting 131072 output tokens. Their complete traces are retained and all five score zero.The audit scanned 5603 raw tool calls and inspected filesystem-related matches.
stockmarket/query2/run0andrun4enumerated repository filenames, including gold/grader filenames; run0 also counted records from its database configuration. Run2 enumerated directories and read/proc/self/cmdline. We did not identify reading gold/validator contents or obtaining an external answer, but DuckDB host filesystem access was not structurally restricted. All three of these stockmarket trials passed the validators. The calls are retained in the traces and described in the README for rubric review; the audit is not claimed to be exhaustive semantic verification.Please review the disclosed traces under the submission rubric and independently validate the reported score.