Summary
Try it yourself
This runs entirely in your browser against policy-engine.js, the same code
that generated the answer key in tests.csv. No API call, no key required.
Pick some options above to see a decision.
161 access requests, each governed by one of three fictional companies' written IAM policies
(a general corporate policy, a HIPAA-flavored hospital policy, and a segregation-of-duties
trading-firm policy), all three given to the model in context at once so it has to pick the
right one, not just apply the only policy it's seen. This is v4 of the project, and the first
version that grades three separate dimensions per response instead of one: did the model apply
the right company's policy at all (policy_correct), cite the exact right rule
number within it (rule_correct), and reach the right final decision
(decision_correct). Model tested: claude-sonnet-5. Full write-up on
GitHub.
decision_correct fail even though the model's final, considered
answer was right, because it wrote ESCALATE, reconsidered out loud, then corrected to DENY,
and grading only reads the first line, per what the prompt instructs. rule_correct
and policy_correct both pass on that row, so this is specifically a first-line
formatting artifact, not a reasoning failure; the table below shows it exactly as graded,
nothing here is hidden.
This runs entirely in your browser against policy-engine.js, the same code
that generated the answer key in tests.csv. No API call, no key required.