Three words change the answer
With no location, the average model gave a complete New Zealand answer to 22% of questions. It hedged across several countries, or asked where you were, on 50%. On 13% it answered for another country without saying so. Add “In New Zealand:” and complete answers rise to 63%.
Each square is one answer in a hundred.
Data table: what the answers did, no location
| Model | Complete for New Zealand | For New Zealand, not complete | Hedged or asked | Another country |
|---|---|---|---|---|
| Claude Fable 5.2 | 35.6% | 12.6% | 42.5% | 9.3% |
| Gemini 4 Carbon | 33.5% | 12.5% | 44.0% | 10.0% |
| Claude Opus 5.5 | 28.6% | 13.6% | 52.1% | 5.6% |
| GPT-6.1 Sol | 17.2% | 10.3% | 69.2% | 3.3% |
| Muse Spark 1.1 | 22.7% | 14.8% | 36.2% | 26.4% |
| Claude Sonnet 5.5 | 19.1% | 16.5% | 53.9% | 10.5% |
| Grok 4.7 | 12.6% | 15.4% | 51.2% | 20.8% |
| Claude Haiku 5.5 | 10.0% | 18.8% | 54.5% | 16.7% |
| Average | 22.4% | 14.3% | 50.5% | 12.8% |
Every model improved. None got everything right.
Naming New Zealand raised every model’s score, by 28 to 51 points. With the location, complete answers ranged from 38% (Claude Haiku 5.5) to 79% (Claude Fable 5.2). Switch to the key figure, which a script checks with no AI in it, and the same model still leads.
Swipe sideways to see the whole chart.
Data table: results by model
| Model | Complete, with location | Complete, no location | Key figure, with location | Key figure, no location |
|---|---|---|---|---|
| Claude Fable 5.2 | 78.9% | 35.6% | 86.4% | 46.7% |
| Gemini 4 Carbon | 73.2% | 33.5% | 83.3% | 43.7% |
| Claude Opus 5.5 | 71.6% | 28.6% | 81.1% | 38.4% |
| GPT-6.1 Sol | 68.3% | 17.2% | 81.5% | 18.9% |
| Muse Spark 1.1 | 64.6% | 22.7% | 77.3% | 30.6% |
| Claude Sonnet 5.5 | 55.6% | 19.1% | 68.1% | 26.6% |
| Grok 4.7 | 50.6% | 12.6% | 65.2% | 18.3% |
| Claude Haiku 5.5 | 37.6% | 10.0% | 50.0% | 15.8% |
When a model guesses, it guesses America
When a model answered for one other country, it was the United States in 521 of 686 answers. You might expect New Zealand’s rules to be confused with Australia’s. The default was the United States.
Swipe sideways to see the whole chart.
Data table: which other country
| Country the answer was written for | Answers |
|---|---|
| United States | 521 |
| United Kingdom | 140 |
| Australia | 24 |
| Canada | 1 |
| More than one other country | 200 |
Right country, wrong year
Models were weakest on figures that change every year. With the location stated, New Zealand’s key figure appeared in 81% of answers about rules that rarely change, and in 28% of answers about rates reset every year. On the 45 questions where the old figure is on record, 23% of answers gave only the old one. A model can be right about the country and wrong about the year.
Swipe sideways to see the whole chart.
Data table: key figure by how often the rule changes
| Kind of rule | Figure questions | With location | No location |
|---|---|---|---|
| Rules that rarely change | 343 | 81.1% | 28.6% |
| Recently changed, or changes with Budgets | 129 | 68.2% | 36.7% |
| Rates reset every year | 36 | 28.1% | 17.5% |
Some subjects are much harder than others
The same topics were hard for every model. Immigration & visas, benefits and outdoors & fishing were the weakest. Wills & estates was the strongest. Select a model’s name to sort the topics by that model.
Swipe sideways to see the whole chart.
Data table: complete answers by topic and model, with location
| Topic | Questions | Average | Fable 5.2 | Gemini 4 | Opus 5.5 | GPT-6.1 | Muse 1.1 | Sonnet 5.5 | Grok 4.7 | Haiku 5.5 |
|---|---|---|---|---|---|---|---|---|---|---|
| ACC | 19 | 74% | 84% | 84% | 84% | 84% | 75% | 68% | 74% | 37% |
| Banking | 31 | 56% | 68% | 61% | 68% | 68% | 61% | 45% | 45% | 32% |
| Benefits | 54 | 45% | 63% | 63% | 48% | 56% | 43% | 37% | 30% | 19% |
| Citizenship & ID | 15 | 57% | 67% | 60% | 60% | 60% | 70% | 60% | 40% | 40% |
| Consumer | 53 | 67% | 79% | 74% | 81% | 75% | 62% | 60% | 55% | 51% |
| Disability support | 15 | 53% | 80% | 67% | 60% | 53% | 56% | 47% | 33% | 27% |
| Driving | 38 | 57% | 74% | 68% | 63% | 66% | 68% | 47% | 47% | 21% |
| Education | 25 | 64% | 84% | 76% | 76% | 64% | 86% | 60% | 44% | 24% |
| Emergencies | 29 | 83% | 90% | 90% | 79% | 86% | 95% | 83% | 72% | 72% |
| Employment rights | 53 | 80% | 94% | 87% | 87% | 85% | 84% | 74% | 75% | 51% |
| Everyday life | 27 | 67% | 85% | 89% | 78% | 67% | 73% | 59% | 48% | 37% |
| Family | 42 | 61% | 74% | 71% | 71% | 69% | 55% | 55% | 52% | 38% |
| Health | 61 | 56% | 72% | 57% | 61% | 59% | 71% | 46% | 44% | 34% |
| Health & safety | 30 | 77% | 87% | 83% | 90% | 73% | 86% | 80% | 67% | 53% |
| Holiday pay | 21 | 84% | 95% | 86% | 90% | 90% | 79% | 86% | 86% | 57% |
| Home & neighbours | 18 | 65% | 78% | 72% | 78% | 67% | 80% | 56% | 56% | 33% |
| Housing | 56 | 54% | 80% | 79% | 62% | 57% | 53% | 46% | 30% | 25% |
| Immigration & visas | 21 | 32% | 57% | 48% | 43% | 33% | 43% | 14% | 5% | 10% |
| KiwiSaver | 20 | 68% | 95% | 70% | 85% | 80% | 60% | 65% | 65% | 20% |
| Leave | 31 | 75% | 84% | 87% | 81% | 84% | 90% | 68% | 81% | 26% |
| NZ Super | 17 | 53% | 71% | 53% | 59% | 71% | 54% | 35% | 41% | 41% |
| Older people & aged care | 15 | 52% | 60% | 67% | 67% | 73% | 44% | 47% | 40% | 20% |
| Outdoors & fishing | 23 | 48% | 74% | 57% | 61% | 39% | 53% | 39% | 30% | 30% |
| Police & your rights | 45 | 67% | 82% | 76% | 84% | 67% | 70% | 64% | 47% | 49% |
| Small business | 30 | 59% | 70% | 70% | 73% | 67% | 62% | 50% | 37% | 40% |
| Student loans | 19 | 55% | 74% | 74% | 47% | 79% | 58% | 32% | 47% | 26% |
| Tax | 49 | 64% | 82% | 71% | 76% | 71% | 59% | 57% | 53% | 41% |
| Tenancy | 59 | 58% | 81% | 76% | 66% | 63% | 46% | 42% | 51% | 36% |
| Travel & border | 18 | 61% | 78% | 78% | 78% | 61% | 50% | 61% | 44% | 39% |
| Voting | 17 | 67% | 88% | 71% | 82% | 65% | 75% | 65% | 53% | 41% |
| Wages | 28 | 64% | 75% | 75% | 68% | 75% | 44% | 68% | 57% | 54% |
| Wills & estates | 21 | 88% | 100% | 95% | 95% | 90% | 91% | 81% | 76% | 71% |
We checked the judge. It was stricter than people.
An AI judge graded every answer, so we checked the judge. Two people each graded the same 120 answers without seeing its grades. Where the two agreed with each other, they agreed with the judge on 84.6% of answers, or 82.2% weighted to all graded answers. The pass mark we set in advance was 85%. The judge missed it.
Each square is one of the answers two people checked.
Data table: the judge against people
| Outcome | Answers |
|---|---|
| Judge and people both said complete | 56 |
| Both said not complete | 32 |
| Judge said no, people said yes | 16 |
| Judge said yes, people said no | 0 |
| The two people disagreed (set aside) | 16 |
Every disagreement ran the same way: the judge failed an answer that people passed. The author then read each of those 16 answers by hand. In 12 the judge was right: a required point was missing, wrong, or not given for New Zealand. In 3 the answer was good enough and the judge was too strict. In 1 the official answer said more than its source. Because the judge is strict, our complete answer scores are a minimum. The main findings also hold on the script, which uses no AI.
Swipe sideways to see the whole chart.
The 16 answers the judge failed and both people passed, as the author judged them on reading each one.
Data table: the author’s reading
| What the author found | Answers |
|---|---|
| Judge right: a required point was missing | 9 |
| Judge right: conflicts with official answer | 2 |
| Judge right: not answered for New Zealand | 1 |
| Good enough: judge too strict | 3 |
| Official answer overstates its source | 1 |
What to do about it
There are three simple ways to get better answers.
Say where you are
Start your question with “In New Zealand”. It is the cheapest fix there is, and it helped every model we tested.
Check anything with a date or a dollar sign
Rates, thresholds and deadlines are where answers go out of date. Confirm them on the official page. If you build with AI, give the model the current figure.
Test before the public uses it
If you are putting a chatbot in front of New Zealanders, run it against questions like these first, and again when the rules change.
Read the research
The paper sets out the study and its findings. The technical supplement has the full method, every table, the checks on the judge and the limits.
How this was done: 1,000 questions in 32 topics, each with an official answer quoted from its source. Eight models were asked through their programming interfaces, with no web search, on 14 and 15 November 2026. An AI judge, Claude Fable 5.2, graded every answer. It is also one of the models tested. A script with no AI in it checked the 508 questions whose answer is a single figure, and two people checked a random sample of the judge’s grades.
The public questions, answers, grades and code are released under CC BY 4.0 (data) and MIT (code) at https://doi.org/10.5281/zenodo.24364019 and on GitHub at github.com/RavenscarAI/NZBench. 76 questions are held back so that future models can be tested on questions they have never seen. This is information drawn from official pages, not legal advice.
Cite as: Carter-Scofield, B. (2026). Does AI get New Zealand right? NZBench, November 2026 edition. Ravenscar AI. https://doi.org/10.5281/zenodo.24364019