How to Audit Sports Annotation Labels: A Checklist (2026)

To audit sports annotation labels, check a sample of the delivered labels against a written specification, using reviewers who did not make them. Report error rates by error type, each with a confidence interval, and locate every error by file and frame. In practice that means seven steps: fix the spec, plan the sample, review in two passes, classify each error, check the sports-specific failure points, write the report, and rank the fixes.

An annotation audit is an independent check of labels that someone else produced: a vendor, an in-house team or a dataset seller. It answers one question before you spend compute on training: what is this data about to teach the model?

This guide is for ML leads who already have labels and don't fully trust them. It is the companion to our guide on how to tell if an annotation vendor's labels are good, which covers what to ask before you buy. This one covers what to do once the labels are in your hands.

Key Takeaways

• Audit against a written spec. If none exists, write the rules down first. Without them, every finding is an opinion.
• Plan the sample before you look at anything. With 100 random units and 5 errors, the 95% interval for the true error rate runs from about 2% to 11% (Wilson method, NIST).
• Have a second reviewer confirm every flag before counting it. Even when researchers flagged label errors with an algorithm and had humans check them, only 51% of candidates turned out to be real errors on average (Northcutt et al., 2021).
• In sports video, the expensive errors cluster in identity across cuts and occlusion, event timing, and players at the edge of the frame. Check those on purpose.
• Report random-sample error rates and hard-case findings separately. Mixing them makes the data look worse than it is.

---

When do you need an annotation audit?

Audit before the data becomes expensive to be wrong. The common moments:

• Before a training run. Find out what the model is about to learn before you spend the compute.
• When a model plateaus. Check whether the ceiling is in the architecture or in the labels.
• Before switching vendors. Get evidence instead of a feeling, on the same footage and the same spec.
• Before you buy a dataset. Know its error rate before it becomes your problem.
• When labels come from several vendors. Audit each against the same spec and sample design, so the results can be compared.

Benchmark datasets are not exempt. A study of 10 widely used test sets estimated an average of at least 3.3% label errors, and at least 6% in the ImageNet validation set (Northcutt, Athalye & Mueller, NeurIPS 2021). If carefully built public datasets carry errors at that rate, assume a commercial delivery does too until you have measured it.

Step 1: What spec are you auditing against?

An error only exists relative to a rule. Before reviewing a single frame, collect:

• the annotation guidelines the labels were made to, with their version number;
• the class list and the definition of every class and event;
• the geometry rules: how tight a box must be, which keypoints count as visible, how masks treat occlusion;
• the identity rules: how long a track ID must persist, and what happens at camera cuts;
• the timing rules: which frame an event belongs to (ball strike, ball landing, first contact).

If the vendor worked without written guidelines, write down the rules you will review against, from the model's needs and a first look at the data, and agree them with the team before the audit starts. This step often turns out to be as useful as the audit itself. Our player re-ID labeling guide covers how to write identity rules, and our football event data schema shows timing and QA fields written into the file itself.

Check the spec version against the delivery. If the guidelines changed halfway through a project, matches tagged before the change are not wrong under the old rules. Audit each batch against the version it was tagged to, and report drift between versions separately.

Step 2: How big should the audit sample be?

Decide the unit of review first, then the sample size, then which units go in.

The unit of review is the thing you count as right or wrong: a frame, a track, an event, or a match segment. Choose the one closest to how the model uses the data. A tracking model cares about tracks. An event-spotting model cares about events.

Then split the sample in two:

1. A random sample, drawn with a recorded seed across all matches, cameras and annotators. Only this sample gives you an error rate that describes the whole delivery.
2. A targeted sample of hard cases: occlusions, camera cuts, crowded penalty boxes, set pieces, rare classes, night games, low-resolution feeds. This is where systematic errors show up first. Report it separately, because hard cases are over-represented on purpose.

Missing labels need a different sample. A sample drawn from the delivered labels only contains labels that exist, so it can find wrong and false labels but never a missed one. To measure missing labels, sample the footage instead: pick frames or match segments at random and check everything in them that the spec says should be labeled.

Count units, not errors. The rate for each error type is the number of units with at least one error of that type, divided by the units reviewed. A single frame can hold several errors, so keep the total error count as a separate figure.

Sample size sets how precise your error rate is. The table shows the 95% Wilson interval around three observed error rates. NIST's engineering statistics handbook notes that Agresti and Coull recommend this method for virtually all combinations of sample size and proportion:

Intervals computed with the Wilson formula from the NIST/SEMATECH e-Handbook of Statistical Methods.

Two practical consequences:

• Zero errors does not mean zero. If a random sample of n units has no errors, the 95% upper bound on the true rate is about 3/n: 3% after 100 units, 1% after 300, 0.3% after 1,000. This is the "rule of three" (Hanley & Lippman-Hand, JAMA 1983).
• Labels are clustered, so the real interval is wider. Errors bunch by match, camera and annotator. Twenty units from one match tell you less than twenty units from twenty matches. Spread the random sample across as many matches and annotators as you can, and treat the intervals above as the best case.

Step 3: How should the review work?

Use two passes, and count only what both reviewers agree is an error.

• First pass. A reviewer who knows the sport checks each sampled unit against the spec and flags every suspected error, with file, frame and the rule it breaks.
• Second pass. A second reviewer confirms or rejects each flag. Only confirmed flags count toward the error rate.

The second pass matters because a first-pass flag is a hypothesis, not a finding. In the Northcutt study, only about half of the algorithmically flagged candidates were confirmed as errors by human reviewers. First-pass human flags can over-flag too, especially when the spec is ambiguous. Rejected flags are still useful: a cluster of them on the same rule usually means the rule itself is unclear.

Review or re-label blind? The two methods answer different questions:

Reviewing is faster but anchors the reviewer to what is already there. A blind re-label of a small subset shows whether disagreements come from the vendor or from the spec. For class and event labels, agreement is usually reported as Cohen's kappa; for boxes and keypoints it is usually measured by overlap or distance. Our guide to vendor onboarding and calibration explains the kappa bands and why they are conventions, not pass marks.

Step 4: How do you classify the errors?

Give every confirmed error one type and one severity. Six types cover almost everything in sports video:

Then give each error a severity. Severity reflects what the error does to the model, not how hard it was to spot:

• Critical: systematic, or corrupts many downstream labels. A persistent identity swap, a class confused across a whole match.
• Major: a wrong label the model will learn from, but isolated.
• Minor: inside the noise your model can tolerate, or a borderline call the spec doesn't settle.

Schema drift is the one type to check on every file, not just on a sample. It is cheap to automate: validate each file against the schema and list every class value in use. If your labels come as JSON, a JSON Schema does most of this work.

Step 5: Which sports-specific failure points should you check?

Sports footage fails in predictable places. Put each of these in the targeted sample.

Identity through occlusion and camera cuts

The MOT16 tracking benchmark counts an identity switch whenever a ground-truth target is matched to a different track than in its last known assignment, and adds identity switches to missed and false detections in its headline MOTA score (Milan et al., 2016). Its authors also note that the raw count of switches is not always expressive on its own, and report it relative to recall.

For an audit, the count matters less than the duration. A swap that is corrected two frames later is a minor error. A swap that persists for 40 frames credits one player's sprint, touches and position to another, and every statistic built on that track inherits the error. Record the first and last frame of every swap, and weight severity by how long it lasted.

Event timing

Decide your timing tolerance in frames before the review, per event type. Benchmarks are a poor guide here. SoccerNet's action spotting evaluation, for example, uses tolerance windows of 1 to 5 seconds for its "tight" metric and 5 to 60 seconds for its "loose" one (SoccerNet evaluation code). That suits judging a model that finds events in a long broadcast. It is far too loose for training data where a shot must sit on the frame of the strike. Report timing errors as a distribution of frame offsets, not as pass or fail. A vendor that is consistently 7 frames late has one fixable rule problem. A vendor that is randomly ±7 frames has a process problem.

The edges of the frame

Players at the far touchline, partly out of shot, or blurred during a fast pan are where missing labels concentrate. Check that the spec says when a player counts as visible enough to label, and audit against that rule, not against what a reviewer can make out on a paused frame.

Set pieces and crowded areas

Corners, free kicks, rebounds under the basket and goalmouth scrambles combine occlusion, fast contact and ambiguous event definitions. They produce identity, class and timing errors in the same few seconds. Sample them deliberately.

Automatic pre-labels

If the vendor used a model to pre-label and humans to correct it, audit the pre-label failure modes too. In our view, automated labels tend to fail the same way every time, which turns one blind spot into a systematic error. Our guide on whether AI auto-tagging still needs human QA covers where automatic tags miss.

Step 6: What goes in the audit report?

A report the vendor can act on and you can defend. It should contain:

1. Scope: dataset, label types, spec version, footage.
2. Sample plan: population, unit of review, random and targeted sample sizes, seed.
3. Method: who reviewed, timing and geometry tolerances, how flags were confirmed.
4. Results from the random sample: errors, units, rate and 95% interval for each error type, plus the severity split.
5. Targeted findings: patterns from the hard cases, with file and frame references.
6. Root causes: spec gap, annotator training, tool behavior or vendor process, each with evidence.
7. A ranked fix list: what to fix first, which files it affects, and who owns it.
8. The full error log, one row per confirmed error, so anyone can check any finding.

Rank fixes by effect, not by count. Ten minor geometry errors matter less than one rule that puts every shot 7 frames late. A systematic fix applied across the whole delivery, followed by a re-audit of a fresh random sample, is usually worth more than correcting the sampled errors one by one.

What error rate is acceptable?

There is no universal threshold. Set one per error type, from what the model can tolerate. A detection model trained on thousands of frames may absorb a few percent of loose boxes. An event model trained on a few hundred examples of a rare class may not survive a handful of class errors in that class. Identity errors in tracking data compound, because every downstream statistic inherits them.

Write the thresholds down before the audit, per error type and severity, so the result is a decision rather than a debate. If you are agreeing them with a vendor, put them in the contract along with the sample plan that will measure them.

Download the templates

Both are free to copy and adapt:

• Audit report template (Markdown): the eight sections above, with a results table that already has a Wilson interval column.
• Error log template (CSV): one row per error, with file, frame range, object or event ID, error type, severity, spec rule, both reviewers and the found and expected values. The four example rows are synthetic, and one shows a flag rejected by the second reviewer.

The sports annotation audit checklist

1. Collect the spec and its version history. If there is none, write the rules down first.
2. Choose the unit of review: frame, track, event or segment.
3. Draw a random sample across matches, cameras and annotators, and record the seed. Sample footage, not labels, to measure missing labels.
4. Draw a separate targeted sample of hard cases.
5. Set timing and geometry tolerances per class and event before reviewing.
6. Review every sampled unit against the spec and flag suspected errors with file, frame and rule.
7. Have a second reviewer confirm or reject every flag.
8. Give each confirmed error one type and one severity.
9. Run automated schema checks on every file, not just the sample.
10. Report random-sample rates with 95% intervals, and targeted findings separately.
11. Trace each pattern to a root cause and rank the fixes by effect.
12. Re-audit a fresh random sample after the fixes.

Frequently asked questions

How many labels should an annotation audit review?

At an observed 5% error rate, 100 random units give a 95% interval of about 2% to 11%, and 400 units narrow it to about 3% to 8%. Choose the sample size whose interval is narrow enough that both ends lead to the same decision. Spread the sample across matches and annotators, because errors cluster.

Can one audit compare two annotation vendors?

Yes, if both are reviewed against the same spec, the same unit of review and the same sample design, ideally on the same footage. Otherwise differences in the error rates may reflect differences in the audits, not the vendors.

Should auditors see the vendor's labels?

For the main error rate, yes: reviewers judge the delivered labels against the spec. Add a small blind re-label, where reviewers label from scratch without seeing the vendor's work, to check whether the spec itself can be applied consistently.

Can a tool audit labels automatically?

Partly. Schema checks can run on every file, and methods such as confident learning can rank likely label errors. But the Northcutt study found that only about half of algorithmically flagged candidates were real errors, so a human still has to confirm each one before it counts.

What if the vendor disputes the findings?

That is what the error log is for. Every confirmed error has a file, a frame, the rule it breaks and two reviewers. Disputes then become questions about the spec, which can be settled and written down, rather than arguments about quality in general.

Get your labels reviewed independently

If you have labels from another vendor, or from several, and need to know what they will teach your model, Train Matricx runs this audit as a service. Sport-trained reviewers check a sample against your spec in two passes and deliver a report with every error located by file and frame. New clients can start with a free pilot review on a small sample of their labels. See the sports annotation quality review service.

Interval values were computed with the NIST Wilson formula, and benchmark definitions were checked against the cited papers and code in November 2026. Recommendations are our own. Spot an error? Email info@trainmatricx.com.