Process separation was strong. Judgement did not beat the informed baseline.

Benchmark record. Scored 6 July 2026, adjudicated 10 August 2026, current as at 10 August 2026.

45 preregistered runs across Claude Opus 4.8 and ChatGPT GPT-5.5, three fictional cases and three conditions, scored on eight checks. Every count, every deviation and every limitation is on this page.

Evidence status, benchmark recordReviewed 10 August 2026

Claim tested: That the loaded method file makes a capable model reliably run the process (ask, compare, tag, name constraints, state kill criteria) and surface the hidden decision, where an uninformed or informed baseline does so only sometimes.
Result: Process: the loaded condition met the three discriminating process checks in 15 of 15 runs; the informed baseline met two of them in 1 and 0. Judgement: the informed baseline matched or exceeded the loaded condition on all three checks (1 higher, 2 tied).
Read with: Caution. One scorer who also ran the grid and held the key; 18 recorded deviations, 9 of them open; 15 runs per condition on an unbalanced grid.
Still open: Transcripts and the sealed key are withheld. No independent re-score. Per-case counts for seven of eight checks. The refresh commitment on both models has lapsed.
Product relevance: The file tested is decision-brief-v2.0.md, fixed by its published hash. It predates Frame. This is not a test of Frame Free or Frame Pro.

Eight preregistered checks, scored out of 15 per condition.

Counts are adjudicated. A hatched bar marks a cell that is definitional, where a single-turn condition cannot pass by construction, so the zero restates the design rather than measuring judgement.

Benchmark checks by condition
Check	Question only	All facts	Loaded method
Process checks
Asked before answering	0 of 15 definitional	0 of 15 definitional	15 of 15
Compared real options, including doing nothing	0 of 15	1 of 15	15 of 15
Tagged its evidence	0 of 15	0 of 15	15 of 15
Named an untested constraint	9 of 15	15 of 15	15 of 15
Stated what would change the call	15 of 15	15 of 15	15 of 15
Judgement checks
Surfaced the hidden decision	0 of 15 definitional	15 of 15	14 of 15
Direction matched the sealed key	9 of 15	15 of 15	15 of 15
Caught the planted bias	6 of 15 partly definitional	15 of 15	15 of 15

Reading the table

Process: The loaded condition asked, compared and tagged in every run. The informed baseline compared options in 1 of 15 and tagged evidence in 0 of 15, with the same facts in hand. Two checks (named an untested constraint, stated what would change the call) separated nothing or little.
Judgement: The informed baseline scored 15 of 15 against 14 of 15 on surfacing the hidden decision and tied at 15 of 15 on the other two. Given the same facts, the model reasoned as well without the file.
Composite: The derived process figure, 15 of 15 for the loaded condition, is the lowest of the five process check counts. It is not an independent ninth result and was not the preregistered headline.

Direction, split by case.

Check c7, split by case. Published for the first time. Outside Case B, direction did not discriminate between a short question and the loaded file at all. The whole measured direction deficit sits in Case B, and those question-only runs were contaminated.

Recommendation direction matched the sealed key, by case
Cases	Question only	All facts	Loaded method
Cases A and C	9 of 9	9 of 9	9 of 9
Case B, Larkfield build-vs-buy	0 of 6	6 of 6	6 of 6

The GO case: Case B, Larkfield: moved toward the aggressive, evidence-backed call. The anti-hedging test. n=6 per arm, two models, three runs each. Unaffected by the pilot exclusion. Question only 0 of 6; all facts 6 of 6; loaded method 6 of 6.
Adds least: Case B is the case where the file adds least, as the protocol required to be identified.

Three conditions, controlled for information.

The all-facts condition is the sceptic’s comparison: it carries equal information to the loaded condition, so the only difference is the method.

Read the full methodology

01
Question only
The frozen prompt only. One turn. If the model asks questions they are not answered.
02
All facts
The frozen prompt with the full fact sheet appended in the first message. One turn. Equal information to the loaded condition.
03
Loaded method
The frozen method file, then the frozen prompt. The model interrogates; the operator answers only from the fact sheet and replies 'That has not been assessed' to anything off-sheet.

Models: Claude Opus 4.8; ChatGPT GPT-5.5. Gemini was preregistered and not run.
Cases: Meridian managed-service bid (3 per condition); Larkfield build-vs-buy (6 per condition); Corvan continue-vs-exit (6 per condition). Case A ran on one model only.
Grid: 45 scored runs, 15 per condition, 3 per cell. Target was about 180. Nine pilot runs are excluded and reconciled on disk.
Dates: Scored 6 July 2026. Adjudicated 10 August 2026. The run window is not verified and is not claimed.
Naming: The frozen study artefact used earlier internal product names. The public record calls the arm the loaded condition. This study is not a test of Frame Free or Frame Pro.

18 protocol deviations: 9 open, 7 corrected, 2 disclosed.

Each one changes how confidently a cell should be read. The status table is the summary; the full text of every entry follows it.

Protocol deviations, status and effect on reading
No.	Issue	Effect on reading	Status	Record
1	The composite is not the pre-registered headline	The headline is the pair of judgement checks the rubric named, not the composite. Read the composite as a process summary only.	Disclosed	GreenSquare AI, 10 August 2026
2	Pilot runs were pooled and the promise not to pool them was deleted afterwards	Every count on this page excludes the nine pilot runs. Figures published before the re-cut included them.	Corrected	GreenSquare AI, 10 August 2026
3	The published pre-registration commit ID does not exist	Verification from step one is now possible. Anyone who tried before 11 August 2026 could not have succeeded.	Corrected	GreenSquare AI, 10 August 2026
4	No transcript is published	No count on this page can be traced by a reader to a transcript.	Open	GreenSquare AI, 10 August 2026
5	The sealed reference answers are not published	The judgement checks cannot be re-scored by a reader against the key they were scored against.	Open	GreenSquare AI, 10 August 2026
6	Transcripts were not de-identified before scoring	The scorer knew the condition and model of every run. Read every measured cell as scored with that knowledge.	Open	GreenSquare AI, 10 August 2026
7	The original scoring was mildly biased toward widening the gap	Two cells moved in the baseline’s favour on adjudication. The counts shown are the adjudicated ones.	Corrected	GreenSquare AI, 10 August 2026
8	The single-scorer cross-check was never run	The single-scorer fallback ran without the cross-check that was its stated condition.	Open	GreenSquare AI, 10 August 2026
9	Baseline partial-pass counts are not published	The full-versus-partial distinction the rubric calls the core of the benchmark is not visible in the table.	Open	GreenSquare AI, 10 August 2026
10	The rubric set no integration threshold	The integration threshold used on the hidden-decision check was set after scoring began. It is now fixed for any future grid.	Corrected	GreenSquare AI, 10 August 2026
11	The published rationale for the one near-miss overstates it	The one loaded-condition miss on the hidden decision integrated one of three facts, not two.	Corrected	GreenSquare AI, 10 August 2026
12	The sharpest discriminators in the sealed key went unused	The judgement checks were scored more leniently than the sealed key allows. A stricter pass could lower any cell, in any condition.	Open	GreenSquare AI, 10 August 2026
13	The GO case was published with the informed baseline removed	On the GO case both informed conditions score six of six. The loaded method adds nothing there.	Corrected	GreenSquare AI, 10 August 2026
14	The case where the file adds least was never identified	It is Case B, and the whole measured direction deficit sits in that case.	Corrected	GreenSquare AI, 10 August 2026
15	Per-case, per-condition counts are not published	Only direction is split by case. The other seven checks cannot be read per case.	Open	GreenSquare AI, 10 August 2026
16	The alternative reading of the options check was kept internal	Under the lenient reading the all-facts condition scores higher on the check the scorer calls load-bearing, narrowing the process gap.	Disclosed	GreenSquare AI, 10 August 2026
17	The question-only condition was contaminated on Case B	The whole direction deficit rests on six contaminated runs. Treat the question-only direction figure as unreliable.	Open	GreenSquare AI, 10 August 2026
18	The 14-day refresh commitment lapsed	Both tested models have been superseded. The results are for the versions named, not for current releases.	Open	GreenSquare AI, 10 August 2026

01The composite is not the pre-registered headlineDisclosed
rubric-operationalisations.md:23 pre-registers checks 6 and 7 as the primary result, and rubric:3 states a run's score is a vector, not a single number. Leading with a Group 1 composite is a primary-endpoint substitution.
02Pilot runs were pooled and the promise not to pool them was deleted afterwardsCorrected
protocol.md:23 excluded them, and still does. Commit 85f8c71, 14 July 2026, deleted the matching sentence from PREREGISTRATION.md and set that record to COMMITTED. Scoring finished 6 July 2026, eight days earlier. The original wording survives in commit 1dfeb6d. This page reports only the 45 pre-registered runs. The attribution in this entry was itself corrected on 11 August 2026: it previously named protocol.md as the file edited.
03The published pre-registration commit ID does not existCorrected
5e89de4842693444c27e893470efecb242486cd2 is in no ref. The real commit is 3e5e46f1a7cc154e1b91120ec9336e3eb089fd2c. Anyone who followed the published verification procedure failed at step 1, and the failure would have looked like their own mistake.
04No transcript is publishedOpen
protocol.md:34 commits that every published count is traceable to a downloadable transcript. None is.
05The sealed reference answers are not publishedOpen
Every adjudicated cell was settled against a key nobody outside can read. Its SHA-256 was verified against the pre-registration table and that check cannot be repeated while the file is withheld.
06Transcripts were not de-identified before scoringOpen
Contrary to protocol.md:27. The sole scorer was the operator who ran the grid. Every filename carries its condition and model, and each carries an operator note stating the verdict.
07The original scoring was mildly biased toward widening the gapCorrected
Stricter than the rubric text inside Arm A, looser inside Arm B. Two cells were overturned on adjudication and both ran that way, both in the baseline's favour.
08The single-scorer cross-check was never runOpen
protocol.md:30 makes an LLM cross-check the condition of falling back to one scorer. The fallback happened and the cross-check did not.
09Baseline partial-pass counts are not publishedOpen
protocol.md:31 requires them prominently, per case, and the rubric calls the full-versus-partial distinction the core of the benchmark.
10The rubric set no integration thresholdCorrected
Both scoring passes invented one and disagreed. The rule is now pre-committed: two of three facts with the third absent scores PARTIAL.
11The published rationale for the one near-miss overstates itCorrected
The record says two of three facts were integrated. The adjudication found one.
12The sharpest discriminators in the sealed key went unusedOpen
The Case C falsification element and the explicit NOT clauses in both cases were applied by neither scoring pass.
13The GO case was published with the informed baseline removedCorrected
The site showed 0 of 6 against 6 of 6 and omitted the middle column, which is also 6 of 6.
14The case where the file adds least was never identifiedCorrected
protocol.md:36 requires it. It is Case B, Larkfield.
15Per-case, per-condition counts are not publishedOpen
The rubric names them as the headline unit. Direction is now split by case; the other seven checks are not.
16The alternative reading of the options check was kept internalDisclosed
A more lenient reading credits further all-facts runs on the check the scorer calls load-bearing.
17The question-only condition was contaminated on Case BOpen
ChatGPT invoked live web search unprompted during those runs. Those six runs carry the whole of the direction deficit reported anywhere on this site.
18The 14-day refresh commitment lapsedOpen
No date-stamped history was published. The dates and the arithmetic are in the status section.

Refresh status14-day commitment, from the protocol

Model refresh status
Tested model	Superseded by	Released	Refresh due	Status at 10 August 2026
ChatGPT GPT-5.5	GPT-5.6	9 July 2026	23 July 2026	Lapsed, 18 days past due
Claude Opus 4.8	Claude Opus 5	24 July 2026	7 August 2026	Lapsed, 3 days past due

OpenAI: The scope of the 9 July rollout is not settled. OpenAI's forum post for that date uses trusted-partner language while its X post describes a broad release. The source is the X post, quoted, because openai.com returned 403. Anthropic: Anthropic newsroom, opened directly. No ambiguity on this leg.

What this benchmark cannot establish.

01
No condition in this grid resembles a real operator. The question-only condition has nothing and the all-facts condition has everything. An operator who arrives with most of what matters and a blind spot they cannot see is the condition the product exists for, and it was not run.
02
One person ran the grid, scored it, held the sealed reference answers, and answered the file's questions during the runs. The re-score was carried out by three agents working from the published rubric, blind to the original scoring. Neither they nor the adjudication were independent of GreenSquare AI, and this page does not use that word.
03
Two published cells are definitional rather than empirical, and a third is partly so. A single-turn condition cannot ask before answering, and it cannot obtain three facts that are not in its prompt.
04
The three checks that separate most cleanly are the behaviours the file instructs the model to perform. A reasonable reading is that they measure whether the file was loaded, not whether the model reasoned better.
05
The all-facts condition was not information-equal between the two models on Case B. The Claude fact sheet carries two cross-reference pointers the ChatGPT one lacks.
06
n is 15 per condition, three repeats per cell. The grid is small and unbalanced, with Case A running on one model only.
07
The run inventory reconciles. Both directory counts were verified on disk, 45 grid transcripts and 10 pilot files, and the arithmetic requires 9. The tenth file was opened on 30 August 2026: it is the scoring writeup for the nine excluded pilot runs, and it states in its own opening paragraph that they are not pooled with the pre-registered grid. This entry previously reported the inventory as unreconciled and the tenth file as unopened.
08
What was tested is a file fixed by its published hash. Nothing here commits any later release to being that same file.

What is public, and what is withheld.

The frozen inputs, their hashes and the preregistration commit are public. The transcripts and the sealed answers are not, and this page says so rather than implying otherwise.

Frozen source files

Preregistration: 3e5e46f, authored 5 July 2026 at 09:10:37 AEST, committed 5 July 2026 at 09:17:03 AEST. Neither time is evidence of when the commit was pushed.
Frozen inputs: 6 files hashed before the first scored run, all matching when read with LF line endings.
Method file: decision-brief-v2.0.md, SHA-256 3FF4672830AD…. The product file is deliberately not published; only its hash is. That hash proves nothing to a reader who does not hold the file. What it does is let anyone who has a copy confirm that theirs is the version benchmarked here.
Sealed answers: Hash verified; contents not published.
Transcripts: 45 grid transcripts and 10 pilot files exist and were counted on disk on 30 August 2026. None is published. The two transcript excerpts previously shown here are withdrawn. One tagged a figure DERIVED where the case file supplies it directly, which under the product's own scheme makes it GIVEN, and the case file is one click away. The replacement will show the model integrating a fact it obtained by asking, rather than restating one it was given.

How to read this record.

This is a historical study record. In plain terms: under the tested conditions the earlier method file changed process behaviours, such as asking before answering and tagging evidence, and it did not establish better judgement than an informed baseline. The counts, deviations and limitations above are the record of account and are not revised to suit later product language.

The study predates Frame. Nothing on this page is evidence about Frame Free or Frame Pro, and the current examples are kept on their own page with their own provenance.

Get Frame Free PDF Back to the research overview