Every payer and every UM platform is being pitched the same idea right now: point a frontier model at the prior-auth packet and let it make the call. These models read messy clinical documents better than anything before them. The open question is whether they should own the final policy determination. So we ran the obvious experiment.
We took one synthetic prior-auth packet, a 26-year-old with chronic low back pain requesting single-level basivertebral nerve ablation, and the verbatim text of the payer's published medical policy for that procedure. No summaries, no retrieval tricks: the exact policy language a reviewer would use. Eight frontier models produced determinations, three runs each. Twenty-four determinations, identical inputs, every time.
The split
Twenty approvals. Four denials.
Six models approved on all three runs. One model denied on all three. One approved twice, then denied its own third run. Two of the most interesting columns came from the same vendor: Opus 4.8 approved every run; Sonnet 4.6 denied every run. Same company, same inputs, opposite verdicts.
Here's the uncomfortable part: Sonnet's denial wasn't stupid. Its rationale was that the documented conservative-therapy trial didn't meet the policy's requirement of being "optimal". Physical therapy, NSAIDs, duloxetine, and a single epidural injection were not enough by its reading. That is a defensible clinical interpretation. So is the opposite one. Which means the determination your member receives depends on which reasonable interpretation you happened to sample, a coin your intake pipeline flips silently at scale.
The agreements were worse
The 20 to 4 split is the headline, but the sharper finding was inside the agreements.
Models that reached the same verdict got there differently. One approver anchored on "chronic axial low back pain for more than 6 months." Another recited a different subset of criteria in a different structure, with different emphasis on the imaging findings. Rerunning the same model rewrote its own rationale between runs: same verdict, new reasoning.
For a demo, that's a curiosity. For a payer, it's a liability, because in healthcare the reason is the decision:
- A denial must ship with the specific criterion it failed. Appeals, ERISA disclosures, and CMS audits all attach to the stated reason. Defend a denial with a different reason later, and the process restarts with new exposure.
- Overturned appeals feed overturn rates; overturn rates feed Star Ratings; Star Ratings drive bonus payments and enrollment.
- A rationale that can't be reproduced can't be audited. If the answer to "why was this denied?" is "the model said so that day," every audit becomes a manual reconstruction project.
What we do instead
The models were genuinely good at reading the packet: extracting the diagnosis, the therapy timeline, and the imaging findings from faxed, scanned, partly handwritten documents. That's the hard 90% of the work, and they should keep doing it.
The conclusion is narrower: the decision itself can't come from a model. So we compile the payer's policy, the actual published document, into a versioned decision artifact. Every criterion carries its verbatim source excerpt and page anchor. The model proposes evidence; the policy rules decide. Same packet, same policy version, same answer, same reason, this run, next run, and in the audit two years from now. Weak or contradictory evidence doesn't become a silent verdict; it becomes a reviewer work item that says exactly what's missing.
You can inspect the full experiment yourself: the prior-auth workbench running on this site includes the recorded runs, the packet, and the policy, including the annotated handwriting the models had to read. The methodology note that matters: the packet is synthetic (no real patient), the policy text is real and published, and some frontier APIs don't expose temperature control, so we report observed variance without claiming anything about their sampling.
If you're deciding prior auths or fighting the denials that come out of someone else's model, the question to ask any AI vendor is simple: run it twice, and show me the reasons match.