Some tasks don’t have one honest answer. Soft labels keep expert disagreement as signal instead of forcing false certainty.
Severity ratings, policy calls, and many engineering judgments admit more than one defensible answer. Collapsing them to a majority vote throws away information your model could learn from — and can make eval scores look cleaner than they are.
Soft labels fit when qualified experts can honestly disagree, and when you care about calibration — not only accuracy. When disagreement is just a labeling error, we still adjudicate. Soft labels are for honest ambiguity, not sloppy review.
Each item can include a label distribution, reviewer count, disagreement class, per-reviewer rationales, and whether adjudication was applied — shipped as JSONL with a short QA summary.
{
"item_id": "mri-2031",
"distribution": { "mild": 0.25, "moderate": 0.50, "severe": 0.25 },
"reviewer_count": 4,
"disagreement_class": "major",
"adjudicated_label": null
}A small pilot can return distributions on the items that actually split.