Bad annotation doesn't announce itself. It looks fine in a spot check, passes a quick review, and then quietly caps your model's accuracy in production — because a model can never be more correct than the ground truth it learned from. By the time you notice, you've trained on it. The teams that avoid this measure annotation quality before it reaches the model, and they know exactly what to look for.
This guide covers how to measure annotation quality, the metrics that matter, and the questions that separate a serious vendor from one hoping you won't check.
High-end technical visualization of a QA quality assurance dashboard showing inter-annotator agreement metrics, first-pass acceptance rates, and ID-persistence audits.
How do you measure annotation quality?
Annotation quality is measured with a few concrete signals: inter-annotator agreement (do independent labelers produce the same labels on the same footage), first-pass acceptance rate (what share of deliveries pass QA without rework), and task-specific accuracy checks like ID-persistence audits for tracking or keypoint tolerance for pose. A trustworthy vendor measures and reports these numbers rather than asking you to trust a spot check. If a provider can't state its first-pass acceptance or rework rate, it isn't measuring quality — it's hoping.
Quality isn't a vibe you get from eyeballing a few frames. It's a set of measurable properties, and the whole point of measuring them is to catch problems before they're baked into your weights.
Why spot-checking a few frames fails
The instinct is to open the delivery, look at ten frames, see clean boxes, and approve it. This misses the errors that actually hurt sports CV models, because those errors are temporal and statistical, not visible in a single frame:
- A bounding box can look perfect in frame 1 and belong to the wrong player by frame 40, after two occlusions.
- Event labels can be individually plausible but inconsistent in how "tackle" vs "interception" was applied across a whole match.
- Pose keypoints can look right at a glance but drift out of tolerance during fast motion.
Single-frame spot checks can't see any of this. You need metrics that measure consistency across labelers and across time.
The metrics that actually matter
| Metric | What it tells you | Where it matters most |
|---|---|---|
| Inter-annotator agreement (IAA) | Whether independent labelers, given the same footage and schema, produce the same labels. Low IAA means the schema is ambiguous or labelers are untrained | Every task — it's the foundation |
| First-pass acceptance rate | The share of deliveries that pass QA with no rework. The single most honest summary of a vendor's real quality | Ongoing engagements |
| ID-persistence / track accuracy | Whether player IDs stay attached to the correct player across occlusions and camera cuts | Tracking datasets |
| Keypoint tolerance / PCK | Whether skeletal keypoints fall within an acceptable pixel distance of ground truth | Pose and biomechanics |
| Event timing accuracy | Whether event labels land on the correct frame | Event recognition, DRS-style, broadcast |
| Class confusion rate | How often one label type is mistaken for another (tackle vs interception) | Any sport-specific event taxonomy |
You don't need all of these on every project. Pick the two or three that match your model's task and hold the vendor to them.
Inter-annotator agreement, in plain terms
Give two trained annotators the same clip and the same schema, independently. If they produce nearly identical labels, your schema is clear and your labelers understand the sport. If they diverge, you've found a problem before it reached your model — either the schema is ambiguous or the labelers aren't sport-trained. High agreement is the leading indicator; most other quality problems trace back to low agreement.
First-pass acceptance: the number that's hardest to fake
Of everything above, first-pass acceptance is the most useful single number because it's outcome-based: what fraction of work is accepted without needing rework. It captures schema clarity, labeler skill, and QA in one figure. It's why we report our quality as 98% accepted first-pass — roughly a 2% rework rate, measured across 6,000+ matches for our flagship client, rather than an unmeasured claim of "high accuracy."
A two-layer QA pipeline: labels are reviewed for keypoint coordinate coordinates and audited for temporal/motion coherence before delivery.
Quality comes from process, not luck
Good numbers are the output of a real QA process. When you evaluate a vendor, look for structure behind the metrics:
- A written schema and taxonomy with explicit edge-case rules, so labelers aren't guessing — the foundation of high agreement.
- A multi-layer review pipeline, where a second, more senior pass audits the first rather than the labeler grading their own work. Our post on human QA for auto-tagged sports video shows how a three-layer review is structured.
- Dedicated tracking audits that follow IDs across occlusions instead of checking isolated frames.
- Sport-trained annotators, because agreement collapses when labelers don't understand what they're looking at — this is why domain-expert annotators produce measurably better ground truth than generalists.
A vendor with a strong process can tell you their numbers off the top of their head. A vendor without one changes the subject.
Ten questions to ask any annotation vendor
Use these to separate real quality from marketing:
- What's your first-pass acceptance (or rework) rate, and how do you measure it?
- How do you measure inter-annotator agreement, and what's typical for my sport?
- Walk me through your QA pipeline — how many review layers?
- How do you audit player-ID persistence across occlusions?
- How are your annotators trained on my specific sport's edge cases?
- Who reviews the reviewers?
- How do you handle a labeling disagreement or an ambiguous edge case?
- Can you deliver a sample with a quality report on my own footage?
- What keypoint tolerance do you hold pose annotation to?
- What happens, at your cost, if a delivery fails my acceptance bar?
The answer to #8 is the most revealing — because it turns every claim into something you can verify.
Frequently asked questions
What is a good inter-annotator agreement score? It depends on the task, but for well-defined sports annotation you want high agreement — independent labelers producing near-identical labels on the same footage. What matters more than a single threshold is that the vendor measures it at all and can explain what drives their number up or down.
What's a good first-pass acceptance rate? The higher the better, and the honesty of the number matters as much as its size. A vendor reporting a realistic, measured rate (with a small, disclosed rework percentage) is more trustworthy than one claiming a suspiciously round "99%+ accuracy" with no method behind it.
How do I check quality without training a model first? Measure it on a sample before you commit: run a pilot on your own footage, then check agreement, ID persistence, and event accuracy against your bar. You catch problems before they reach your weights.
Isn't spot-checking a few frames enough? No. The errors that hurt sports models — ID switches, temporal drift, inconsistent event labeling — are invisible in single frames. You need metrics that measure consistency across labelers and across time.
See the quality on your own footage
Every quality claim is theoretical until it's measured on your data. The most direct way to evaluate us — or any vendor — is a pilot: send your hardest clip, the occlusion-heavy sequence your current pipeline mislabels, and we'll annotate it with full QA and return it with a quality report inside 48 hours. Hold it to the metrics above before you commit to anything.
Client results
“Train Matricx didn't just provide labeling — they became an extension of our team. With 6K+ football matches annotated and counting at 98% accuracy, they are our premier data partner.”
Billy Brush
Founder & CEO, Dark Horse AI
Don't take our word for it. Test the labels.
Send us the footage your current vendor gets wrong — the occluded players, the motion blur, the edge cases. We annotate it and return a QA report you can check frame by frame.
- Your hardest clip, annotated in 48 hours
- Full QA report with per-frame accuracy
- No commitment, no contract, no card
