Skip to content
Contact us

NZBench · November 2026 edition

Does AI get New Zealand right?

People increasingly ask AI models everyday questions about wages, tenancy, health, benefits and immigration. Answers that are right for other countries can often be wrong for New Zealand. We asked eight widely used AI models 1,000 everyday questions, twice: once the way a person would type it, and once starting with “In New Zealand:”.

22%of answers were complete for New Zealand when the question gave no location
63%when the question began “In New Zealand:”
01

Three words change the answer

With no location, the average model gave a complete New Zealand answer to 22% of questions. It hedged across several countries, or asked where you were, on 50%. On 13% it answered for another country without saying so. Add “In New Zealand:” and complete answers rise to 63%.

Each square is one answer in a hundred.

Data table: what the answers did, no location
ModelComplete for New ZealandFor New Zealand, not completeHedged or askedAnother country
Claude Fable 5.235.6%12.6%42.5%9.3%
Gemini 4 Carbon33.5%12.5%44.0%10.0%
Claude Opus 5.528.6%13.6%52.1%5.6%
GPT-6.1 Sol17.2%10.3%69.2%3.3%
Muse Spark 1.122.7%14.8%36.2%26.4%
Claude Sonnet 5.519.1%16.5%53.9%10.5%
Grok 4.712.6%15.4%51.2%20.8%
Claude Haiku 5.510.0%18.8%54.5%16.7%
Average22.4%14.3%50.5%12.8%
02

Every model improved. None got everything right.

Naming New Zealand raised every model’s score, by 28 to 51 points. With the location, complete answers ranged from 38% (Claude Haiku 5.5) to 79% (Claude Fable 5.2). Switch to the key figure, which a script checks with no AI in it, and the same model still leads.

No location in the questionQuestion begins “In New Zealand:”

Swipe sideways to see the whole chart.

Data table: results by model
ModelComplete, with locationComplete, no locationKey figure, with locationKey figure, no location
Claude Fable 5.278.9%35.6%86.4%46.7%
Gemini 4 Carbon73.2%33.5%83.3%43.7%
Claude Opus 5.571.6%28.6%81.1%38.4%
GPT-6.1 Sol68.3%17.2%81.5%18.9%
Muse Spark 1.164.6%22.7%77.3%30.6%
Claude Sonnet 5.555.6%19.1%68.1%26.6%
Grok 4.750.6%12.6%65.2%18.3%
Claude Haiku 5.537.6%10.0%50.0%15.8%
03

When a model guesses, it guesses America

When a model answered for one other country, it was the United States in 521 of 686 answers. You might expect New Zealand’s rules to be confused with Australia’s. The default was the United States.

Swipe sideways to see the whole chart.

Data table: which other country
Country the answer was written forAnswers
United States521
United Kingdom140
Australia24
Canada1
More than one other country200
04

Right country, wrong year

Models were weakest on figures that change every year. With the location stated, New Zealand’s key figure appeared in 81% of answers about rules that rarely change, and in 28% of answers about rates reset every year. On the 45 questions where the old figure is on record, 23% of answers gave only the old one. A model can be right about the country and wrong about the year.

No location in the questionQuestion begins “In New Zealand:”

Swipe sideways to see the whole chart.

Data table: key figure by how often the rule changes
Kind of ruleFigure questionsWith locationNo location
Rules that rarely change34381.1%28.6%
Recently changed, or changes with Budgets12968.2%36.7%
Rates reset every year3628.1%17.5%
05

Some subjects are much harder than others

The same topics were hard for every model. Immigration & visas, benefits and outdoors & fishing were the weakest. Wills & estates was the strongest. Select a model’s name to sort the topics by that model.

Swipe sideways to see the whole chart.

Data table: complete answers by topic and model, with location
TopicQuestionsAverageFable 5.2Gemini 4Opus 5.5GPT-6.1Muse 1.1Sonnet 5.5Grok 4.7Haiku 5.5
ACC1974%84%84%84%84%75%68%74%37%
Banking3156%68%61%68%68%61%45%45%32%
Benefits5445%63%63%48%56%43%37%30%19%
Citizenship & ID1557%67%60%60%60%70%60%40%40%
Consumer5367%79%74%81%75%62%60%55%51%
Disability support1553%80%67%60%53%56%47%33%27%
Driving3857%74%68%63%66%68%47%47%21%
Education2564%84%76%76%64%86%60%44%24%
Emergencies2983%90%90%79%86%95%83%72%72%
Employment rights5380%94%87%87%85%84%74%75%51%
Everyday life2767%85%89%78%67%73%59%48%37%
Family4261%74%71%71%69%55%55%52%38%
Health6156%72%57%61%59%71%46%44%34%
Health & safety3077%87%83%90%73%86%80%67%53%
Holiday pay2184%95%86%90%90%79%86%86%57%
Home & neighbours1865%78%72%78%67%80%56%56%33%
Housing5654%80%79%62%57%53%46%30%25%
Immigration & visas2132%57%48%43%33%43%14%5%10%
KiwiSaver2068%95%70%85%80%60%65%65%20%
Leave3175%84%87%81%84%90%68%81%26%
NZ Super1753%71%53%59%71%54%35%41%41%
Older people & aged care1552%60%67%67%73%44%47%40%20%
Outdoors & fishing2348%74%57%61%39%53%39%30%30%
Police & your rights4567%82%76%84%67%70%64%47%49%
Small business3059%70%70%73%67%62%50%37%40%
Student loans1955%74%74%47%79%58%32%47%26%
Tax4964%82%71%76%71%59%57%53%41%
Tenancy5958%81%76%66%63%46%42%51%36%
Travel & border1861%78%78%78%61%50%61%44%39%
Voting1767%88%71%82%65%75%65%53%41%
Wages2864%75%75%68%75%44%68%57%54%
Wills & estates2188%100%95%95%90%91%81%76%71%
06

We checked the judge. It was stricter than people.

An AI judge graded every answer, so we checked the judge. Two people each graded the same 120 answers without seeing its grades. Where the two agreed with each other, they agreed with the judge on 84.6% of answers, or 82.2% weighted to all graded answers. The pass mark we set in advance was 85%. The judge missed it.

Judge and people both said completeBoth said not completeJudge said no, people said yesThe two people disagreed

Each square is one of the answers two people checked.

Data table: the judge against people
OutcomeAnswers
Judge and people both said complete56
Both said not complete32
Judge said no, people said yes16
Judge said yes, people said no0
The two people disagreed (set aside)16

Every disagreement ran the same way: the judge failed an answer that people passed. The author then read each of those 16 answers by hand. In 12 the judge was right: a required point was missing, wrong, or not given for New Zealand. In 3 the answer was good enough and the judge was too strict. In 1 the official answer said more than its source. Because the judge is strict, our complete answer scores are a minimum. The main findings also hold on the script, which uses no AI.

Swipe sideways to see the whole chart.

The 16 answers the judge failed and both people passed, as the author judged them on reading each one.

Data table: the author’s reading
What the author foundAnswers
Judge right: a required point was missing9
Judge right: conflicts with official answer2
Judge right: not answered for New Zealand1
Good enough: judge too strict3
Official answer overstates its source1
07

What to do about it

There are three simple ways to get better answers.

Say where you are

Start your question with “In New Zealand”. It is the cheapest fix there is, and it helped every model we tested.

Check anything with a date or a dollar sign

Rates, thresholds and deadlines are where answers go out of date. Confirm them on the official page. If you build with AI, give the model the current figure.

Test before the public uses it

If you are putting a chatbot in front of New Zealanders, run it against questions like these first, and again when the rules change.

08

Read the research

The paper sets out the study and its findings. The technical supplement has the full method, every table, the checks on the judge and the limits.

How this was done: 1,000 questions in 32 topics, each with an official answer quoted from its source. Eight models were asked through their programming interfaces, with no web search, on 14 and 15 November 2026. An AI judge, Claude Fable 5.2, graded every answer. It is also one of the models tested. A script with no AI in it checked the 508 questions whose answer is a single figure, and two people checked a random sample of the judge’s grades.

The public questions, answers, grades and code are released under CC BY 4.0 (data) and MIT (code) at https://doi.org/10.5281/zenodo.24364019 and on GitHub at github.com/RavenscarAI/NZBench. 76 questions are held back so that future models can be tested on questions they have never seen. This is information drawn from official pages, not legal advice.

Cite as: Carter-Scofield, B. (2026). Does AI get New Zealand right? NZBench, November 2026 edition. Ravenscar AI. https://doi.org/10.5281/zenodo.24364019

Get updates by email

New NZBench editions and Ravenscar research, as they come out.

We will only use your email to send you Ravenscar’s research updates. It is stored on servers in New Zealand. Unsubscribe any time.