Confidence scoring explained: how to make AI defensible in a SOC audit

In a SOC 2 audit, "the model is ninety-six percent accurate" is not an answer. It is a question with a follow-up the team is rarely prepared for: what happens to the other four percent?

Confidence scoring is the mechanism that converts "the model is good" into "the system is defensible." Without it, every AI deployment is one auditor's question away from a finding.

A defensible confidence framework has four components.

1. A documented threshold per workflow. Not "above eighty percent." A specific number — 0.87, 0.92, whatever the data supports — written into the spec, defended by the test set that produced it, and reviewed quarterly.
2. A binary action rule at the threshold. Above it, the system acts. Below it, the system stops. There is no "the system flags it for review" without a named queue, an SLA, and a documented owner. "Flagged for review" with no SLA is a polite way of saying nothing happens.
3. An exception logging schema. Every below-threshold event writes a structured row — input, model output, confidence score, timestamp, exception path taken, eventual resolution. The auditor will sample twenty-five of these. If you cannot reproduce them, the system is not auditable.
4. A quarterly threshold review. A thirty-minute meeting where the operations owner, the model owner, and the finance or compliance owner review exception logs, drift in confidence distribution, and any false positives that slipped above threshold. The review is documented. The threshold either holds or moves with a written reason.

In a healthcare receivables automation deployment I led — roughly fifty thousand transactions per month flowing into Sage Intacct via a custom integration — the confidence framework ran approximately sixty-five lines of specification across all transaction types. Every single line had to survive both internal compliance review and the external audit. None of it was about the model. All of it was about what happens when the model is uncertain.

If a model is uncertain and the system has no documented response, the system has no controls. "No controls" is the language an auditor uses to mean we cannot rely on the financial statements.

The model's accuracy is a question for your ML team. The confidence framework is a question for your CFO. They are not the same question, and confusing them is how AI deployments that worked perfectly in the demo fail the first real review.