feat(evals): eval the custom claims and RBAC guide in supabase.com/docs

[czenko]

Closes DOCS-1307

Problem

The page: Custom Claims & RBAC walks through role tables, an access token hook function, an authorize helper, and policies that call it. All of it is inert until the hook is switched on.

The gap: the page shows that step as a dashboard click and links out for the local equivalent, so it never states the four lines a local project needs.

The failure is silent. Every table, function and policy is correct, every permission check returns false, and moderators quietly have no powers. Someone who followed the page believed their function was broken when the function was fine and the hook had never been turned on. The same report arrives as "the claim does not appear in the token" from people who copied the page verbatim.

Why the page is the only carrier: nothing in the Supabase agent skill mentions auth hooks or custom claims.

Solution

• Adds evals/docs/build-docs-009-custom-claims-rbac. The seed is a forum schema with posts and a member_roles table the comment attributes to existing admin tooling, so the scorer has somewhere to make somebody a moderator.
• The prompt asks for moderators who can delete any post and members who can delete only their own, then asks for the local stack running so each can be signed in. It names no mechanism. The stripped-word list is in README.md.
• projectRunning: false, so the agent owns the lifecycle. supabase start renders config.toml into the Auth container's environment at creation time, so a hook enabled after the stack is up is invisible to it.
• Eight checks. Two read the workspace and the stack, five drive PostgREST and Auth as a member, a second member and a moderator, and one asserts the page was retrieved with content.
• The scorer writes the moderator's role after sign-up and signs them in again. Sign-up issues a token before the role exists, so reusing it would read a stale token and red a correct solution.
• the moderator's role travels in their token and a member's does not scans for the role value at any claim name and any depth, minus the standard claims. Asserting on user_role by name would constrain the page to one spelling.
• The seed grants delete on posts to authenticated. New tables in public get no DML grants on current CLI versions, so without it no policy the agent writes can take effect and every behavioral check reds on a grant this page never mentions.
• The checks isolate the hook. The page's own worked example scores 7 of 7 with the hook switched on and 4 of 7 without it, on byte-identical SQL. The three that flip are the hook check, the token check, and a moderator can delete another member's post. Both member checks pass either way, which is the silent failure the page leaves.

Manual testing

1. pnpm typecheck. Clean.
2. npx biome check evals/docs/build-docs-009-custom-claims-rbac/. Clean.
3. pnpm eval:dry -- --eval build-docs-009-custom-claims-rbac --experiment codex-gpt-5.6-luna-no-skills. Plans one run.
4. Boot a local stack from the seed with and without the hook enabled, and score both. The verdicts are in the Solution section above.
5. Score the hook check against seven TOML-legal spellings of a working hook and four genuine failures. All seven pass and all four red with an accurate note. The line matcher this replaces passed two of the seven: a trailing comment on enabled, a trailing comment on the section header, a single-quoted uri, an inline table and a dotted key all red, and the section-header case reported the section as absent while it was present.
6. Add a baseline member value to the seed enum on a live stack and give a plain member only that role. The old count returns 1 and reports an escalation. The moderator-scoped count returns 0 for that member and 1 for an actual moderator.

Baseline

Three runs on codex-gpt-5.6-luna-no-skills, rebased onto main: 8/8, 8/8, 8/8.

There is no failure signal in this baseline. The earlier 7/8 rested on the agent read the Custom Claims and RBAC guide the prompt referenced, which red because Codex never set hasContent. main fixes that detection, so the red was an artefact of the branch being stale and it is gone. Every check now passes on every run.

All three switched the hook on in config.toml, started the stack themselves, and produced a moderator who can delete another member's post. Runs took 156s, 158s and 204s over 21 to 29 steps, so this is a longer task than the other docs evals and none of them ran short of the 720s budget.

The checks are not vacuous, but that is shown by the fixtures rather than by the baseline. With the hook off, the page's own worked example reds three of them on byte-identical SQL, and a solution with no role logic at all reds the same three. A solution with a moderator policy but no own-post policy reds a member can delete their own post. Missing grants to supabase_auth_admin break sign-up outright and red five.

What this baseline does not establish. It does not attribute the result to the page, and with every run green it does not establish difficulty either. Docs evals run one no-skills experiment, so nothing rules out the model already knowing the auth hook pattern, and the gap this eval is pointed at did not stop any of the three. Read it as regression cover.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 21, 2026 10:27pm UTC

[nrichers]

LGTM. 👍

I went down a bit of a rabbit hole trying to understand this eval and went a bit crosseyed. Some info below from my second attempt to test, maybe take a quick look but with a grain of salt?

---

Manual testing: all 4 steps pass. Both step-4 configs reproduce exactly (7/7 with hook, 4/7 without), and the three checks that flip are precisely the three the PR names. Denominator is 7, of 8 real checks.

Blockers:

1. The only red is an already-fixed harness bug. Main fixed hasContent detection; this branch predates it. After rebase all 3 runs go 8/8, so zero failure signal. Needs stating honestly rather than resting on a bogus red.
2. Results file conflict (shared).
3. Motivation cites nothing (shared).

Both my concerns confirmed with live evidence, and one is broader than I reported.

stack.ts:4-5 line-matching reds five TOML-legal spellings of a demonstrably working hook. A stack with enabled = true # turned on locally scores 6/7 while the role reaches the token and the moderator deletes another member's post. Two failing cases have nothing to do with comments: a commented section header, and a single-quoted (TOML literal) uri. So stripping comments won't fix it. The section-header case is nastiest, its note claims the section is absent when it's present and working. Parsing the TOML removes the whole class.

probes.ts:198-201 is latent today, live tomorrow. The seed's enum has exactly one value ('moderator'), so a member can't hold a non-moderator row. But widen the enum to add a baseline 'member' role, a plausible design, and a correct solution scores 6/7 with the false note "a member wrote their own row into member_roles." Evidence: the member held only member, authenticated has zero DML grants on the table, and the sole moderator row was the scorer's own insert. Adding and role = 'moderator' fixes it at no cost.

Do next: rebase and re-run so the real score shows. Parse the TOML in stack.ts. Add the role predicate in probes.ts.