What it measures
The AIA Loyalty Benchmark tests whether an AI stays faithful to the person it serves when an institution's objectives pull the other way.
It runs controlled rights disputes — a person with a real entitlement, an institution that would prefer they gave up — and evaluates the complete answer the AI gives. Does it tell the truth about the person's rights, remedies, deadlines, and routes? Or does it steer them toward surrender?
What we found
Who controls an AI can change its advice.
In our current frontier comparison, we ran two leading models through the same controlled entitlement disputes, 64 responses each, with the AI set up to serve the institution.
False-justified surrender under institutional control
Share of responses advising the person to give up on a materially false account. Lower is better.
AIA Loyalty Benchmark, study LB-FJS-06, September 2026. Confirmed failures only. Results apply to the model versions and configuration tested; they are not a general ranking.
The failures followed a pattern. The model advised the person to give up for reasons that weren't true: that no further review existed, that an invalid condition had to be accepted, or that pursuing it was pointless. Some answers did it right after acknowledging the entitlement — with the record in front of the model showing a route still open.
We call this false-justified surrender: advice to give up, supported by a materially false account of the person's rights, remedies, deadlines, evidence, or routes. It does more than deny a claim. It makes quitting look rational.
Fluency, intelligence, and brand don't tell you whose side an AI is on. Testing does.
How it works
Four design choices keep the results meaningful — and every run preserves its evidence for review.
Model-agnostic
The same controlled scenarios run across frontier models from different providers. No preferred model, no preferred vendor.
Configuration-aware
Matched tests show how a model's advice changes when an institution's objectives are in control.
Full answers, not fragments
A correct sentence about the entitlement doesn't excuse a false reason to quit. The person relies on the whole answer, so the whole answer is evaluated.
Honest counsel isn't penalized
Advice to stop counts as a failure only when it rests on a false account. Loyalty means the truth — including when accepting is the right call.
Why it matters
As institutions put AI between themselves and the people they serve, "on your side" should be something you can check — not something you're told.
The benchmark can audit AIA with the same scenarios it runs on frontier models. Inside AIA, the Firewall guards against this exact failure: it checks any claim that a person has no way forward before that claim can become advice.
Work with us
We can run the Loyalty Benchmark across frontier models for government, safety, and certification work. Briefings and demonstrations use fictional cases — never private case material.
Contact Alex Wright, CEO, at alex@aiadvocate.ca.