Judging a segmentation mask when you already have the correct one is a solved exercise. You compute an overlap and you are done. The case people actually have is the other one: a model producing thousands of masks in production, an outsourced annotation batch that has to be signed off, a training set nobody has ever audited. There is no answer key, and a perfect judge would have to be a perfect segmenter.
So the tool we built is not an oracle. It is triage. It scores every mask, sorts the queue worst first, shows where and why each one looks wrong, and lets a reviewer open only the suspicious ones. Everything below that line in the queue is time the reviewer gets back.
This report describes how it decides, and the design choices that make its number mean something.
The number
The score is a 0 to 100 estimate of the grade the mask would receive against an expert’s. The grade is surface Dice: the share of the mask’s outline lying within a tolerance of the expert’s outline.
The choice of surface Dice over volume Dice is deliberate. Volume overlap flatters large organs (a sloppy liver still scores 95) and punishes small ones (a one-voxel error on an adrenal gland reads as a disaster). A tool meant to span body parts cannot use a metric that means different things on each.
The tolerance is size-aware: a small fraction of the structure’s size, with a floor of one voxel and a ceiling of a few millimetres. A small structure is graded tightly and a large one more loosely, in proportion, so the same number means the same thing whether the structure is small or large.
The band is load-bearing
Every score comes with a band, written as 82 (74–89), and the band decides the colour.
- Green requires a high score and a narrow band. A confident 88 passes. An 88 that might be a 62 does not.
- Red is a low score with a band tight enough to mean it, or any hard finding.
- Amber is everything else, including a wide band in either direction.
The last rule is the one that matters. A score of 30 with a band of 19 to 97 is not a bad mask. It is an unknown one, and the tool says so rather than claiming knowledge it has just disclaimed. Reviewers quickly learn that amber means “look”, red means “fix” and green means “move on”, and that the tool never pretends to a certainty it does not have.
Hard findings outrank the model
Some things are not judgement calls. An empty label. A mask that does not sit on its image’s grid. Blank slices inside a structure that should be continuous. None of these wait for a model to agree. A case can be red at a score of 92, and the note explains why.
This is the same principle we apply in every system we build: deterministic checks first, and the model’s estimate only where arithmetic cannot reach.
Families of checks
Several families of checks turn an image and a mask into features. Each feature is designed so that its value means the same thing on a hippocampus as on a spleen.
- Integrity. Does the mask exist, does it align with the image, are there holes, islands or blank slices where there should not be.
- Shape. Compactness, sphericity, surface roughness, and how far the structure departs from what its own class usually looks like, measured in size-normalised terms.
- Boundary agreement. Does the mask’s edge sit on an intensity edge in the image, and how often does it cut across uniform tissue instead.
- Intensity. Is the tissue inside the mask homogeneous in the way the structure should be, and does the tissue just outside look different.
- Slice consistency. For stacks and volumes, how much does the outline change between neighbouring slices, and where.
- Context. Is the structure where structures of this kind are, relative to the image, and is it the size such structures are.
A model trained on cases with expert references combines these into the surface Dice estimate and the band. Because the inputs are normalised, the model trained on one set of structures transfers to others, which is what “any body part” has to mean if it is to mean anything.
What the reviewer sees
For each case: the score and band, the colour, a list of findings in plain words, a heatmap of where the evidence is weakest, a per-slice strip for stacks and volumes, and a written QA note that a reviewer could paste into a sign-off. The queue is sorted worst first, and a reviewer can stop partway down.
The heatmap deserves emphasis. A score tells a reviewer whether to open a case. The heatmap tells them where to look once it is open, which is most of the time saved.
2D, stacks and 3D
The tool works on a single 2D image, on a stack segmented slice by slice, and on a full 3D volume. The check families are the same; the slice-consistency family is simply absent for a single image. The score means the same thing in each mode because the tolerance is derived from the structure, not the modality.
What it is not
It does not replace a reviewer, and it is not evidence that a mask is correct. It is evidence about where review time should go. Used that way, it turns an unauditable batch into a queue with a line in it, and that line is the product.
Open questions
Two questions decide whether a deployment is engineering or research. First, whether the client’s structures resemble those the model was trained on closely enough for the band to be calibrated, which we test on a held-out sample before committing. Second, whether the client has a few hundred expert-reviewed cases to recalibrate against, which turns a research question into a fitting exercise. We ask both before quoting.
About the authors. Neugent Consulting designs and builds AI systems for operators. The examples in this piece come from client work; clients are anonymised and details are simplified.
