Proctoring flags need a human review queue, not an automatic accusation

Most flags are the room, not the candidate

A candidate reading a question aloud, thinking with their eyes on the desk, or sitting in a home with other people in it will collect flags in any session. Siblings, a television through a wall, a pet crossing the camera: the software cannot rank these by intent, and it is not designed to. It is designed for sensitivity, because a missed cheat costs the vendor a contract complaint, while a false flag costs the vendor nothing. That asymmetry belongs in every conversation about flag thresholds.

Review overturn rates tell the truth about a programs settings. If human review clears nearly everything, the threshold is set to generate work, not signal, and reviewers drowning in innocent flags start clearing quickly, which is when a real case slides through inside the noise. Tune the threshold using your own overturn data, not the vendors default and not another programs settings.

Build the queue like a triage system

Order the queue by consequence and clarity, not by arrival. Sessions with a second person interacting with the candidate, a device appearing in frame, or a voice reading answers go first. Single low level flags, one gaze event in three hours, should be batch cleared under a documented rule, signed off in aggregate, so reviewer hours go where judgement is actually needed.

Two reviewers for any case that will end a candidacy. The second reviewer watches without the first reviewers conclusion in front of them, and disagreements are recorded rather than averaged away. That single practice does more for appeal survival than any other, because it converts one persons impression into a process the program can describe and defend, step by step.

A review note written from memory a week later is worth little in an appeal. Notes should be timestamped observations, at 41 minutes a second face appears in the doorway for six seconds and does not interact, not character judgements. Ban conclusion words from first pass notes. The finding is a separate step, taken after the observations are complete, and the separation is what makes both credible.

Write the record while you watch

Keep the recording, the flag log, the notes, and the decision together under the candidate record, with retention matched to the appeal window plus a margin. Programs lose appeals most often not on the merits but on the file: a recording overwritten, a flag log nobody exported, a decision letter that describes evidence the program can no longer produce.

Publish, in plain language, what is flagged, that flags are reviewed by a person, and that no result is voided on software output alone, if that is true. If it is not true, change the process until it is, because candidates will ask, regulators increasingly ask, and an honest answer is also the programs best defence. Secrecy about review does not deter the determined cheat, who assumes review anyway. It frightens the honest candidate, who does not.

Tell candidates what the system is

Give every adverse decision a specific appeal route with a deadline and a named reviewer who was not the original reviewer. Appeals that land on the desk of the person who decided are theatre, and candidates, institutions, and courts can all tell the difference.

Keep reading: How do testing programs spot collusion when two candidates answer alike?, How should exam programs handle AI assistants a candidate can run on a second screen?.

Frequently asked questions