Almost every failed sports computer vision project we've seen shares a root cause, and it isn't the model. It's a taxonomy that was designed in an afternoon, before anyone knew what the model would actually be asked to do, and then locked in place by a hundred thousand labels that are now expensive to change.
Taxonomy is the schema that defines what your annotators label and what your model learns to predict. Routing is the system that decides which content goes to which annotator, which model, or which review queue. Together they determine your ceiling — a model cannot learn a distinction its taxonomy never encoded, and it cannot be trained efficiently if every clip is treated as equally difficult.
This guide covers how to design a sports content taxonomy that holds up under production load, and how to build routing logic that puts the right footage in front of the right resource.
Technical visualization of a hierarchical sports event taxonomy: entity, event, attribute and context layers resolving into structured annotation output.
What is a sports content taxonomy?
A sports content taxonomy is the structured, hierarchical definition of every entity, event, attribute and contextual state that annotators label in sports footage and that a computer vision model is trained to predict — including the exact boundary rules that determine when each label applies, when it does not, and how ambiguous cases are resolved.
It is not a list of labels. A list of labels is what most teams start with, and it's the reason their inter-annotator agreement collapses in month three. A taxonomy is a list of labels plus the decision rules that make those labels reproducible across different annotators, different matches, and different competitions.
The difference is concrete. "Pass" is a label. "A pass is recorded when a player makes deliberate contact with the ball that transfers possession toward a teammate, regardless of whether the teammate successfully receives it; deflections off an opponent are not passes; a deliberate contact where the intended recipient cannot be determined is labelled as a clearance" is a taxonomy entry.
Why taxonomy is the highest-leverage decision in the project
Taxonomy decisions compound. Every other decision in a sports AI pipeline can be revised at moderate cost. Taxonomy cannot, because it is retroactive — changing it invalidates work already done.
| Decision | Cost to change after 100k labels |
|---|---|
| Model architecture | Low — retrain on existing data |
| Annotation tooling | Low — migrate, labels are portable |
| Annotation vendor | Moderate — onboarding and calibration |
| Frame sampling rate | Moderate — re-extract, re-annotate the delta |
| Taxonomy | Severe — every existing label may need review or re-annotation |
When a taxonomy changes mid-project, teams face three unattractive options: re-annotate historical data at full cost, accept a dataset that is internally inconsistent across time, or maintain two schema versions and the mapping logic between them indefinitely. All three are expensive. The second is the most dangerous, because the inconsistency is invisible until a model trained on the merged data underperforms for reasons nobody can isolate.
This is why taxonomy design deserves a disproportionate share of project time up front, relative to how trivial it looks on a whiteboard.
The four layers of a sports taxonomy
A well-formed sports taxonomy separates concerns into distinct layers rather than flattening everything into one label set. Flattening is the most common structural mistake, and it produces combinatorial explosion — if "shot" and "left foot" and "inside the box" are all sibling labels, you eventually need a label for every combination.
Layer 1 — Entities
The objects that exist in the frame and persist across frames. Players, ball, officials, goal, boundary lines, equipment. Entities carry identity — this is the same player across the whole match — and their annotation is spatial: bounding boxes, keypoints, segmentation masks.
Entity design questions that matter: Do you track officials as a distinct class or as generic persons? Do substitutes on the sideline count as entities? Is the ball one class, or does it have states — in play, out of play, held?
Layer 2 — Events
Things that happen at a moment or across an interval. Pass, shot, tackle, foul, substitution, restart. Events have temporal boundaries and this is where most taxonomy failures originate, because the boundaries are genuinely ambiguous in real footage.
Every event needs an explicit answer to: When exactly does it start? When does it end? Is it instantaneous or does it have duration? Can it overlap with other events? A tackle that becomes a foul that becomes a card is either one event with attributes or three linked events, and choosing wrongly makes downstream analysis awkward forever.
Layer 3 — Attributes
Properties that modify an entity or event rather than existing independently. Body part used, outcome (successful or not), direction, pressure level, technique classification. Attributes are what prevent combinatorial explosion — instead of thirty shot variants as separate labels, you have one shot event with attributes for body part, outcome and location.
The design rule: if a property can apply to more than one event type, it belongs in the attribute layer, not the event layer.
Layer 4 — Context
State that is true for a span of time rather than attached to a specific event. Score line, phase of play, formation, in-possession team, weather, period of the match. Context is frequently omitted from early taxonomies and frequently needed later, because it is what makes events comparable — a pass under high press in the 89th minute at one goal down is a different data point from the same pass at 3-0 up in the 20th.
| Layer | Answers | Annotation type | Changes |
|---|---|---|---|
| Entity | What is present? | Spatial (boxes, keypoints, masks) | Per frame |
| Event | What happened? | Temporal (timestamps, intervals) | At moments |
| Attribute | What kind? | Categorical, attached to entity or event | With its parent |
| Context | What was the situation? | State intervals | Between transitions |
Four-layer sports taxonomy breakdown: separating entities, events, attributes, and contextual states into structured schema layers.
Seven taxonomy design failures we see repeatedly
1. Designing the taxonomy before defining the model's job. The taxonomy must be derived from the questions the product needs to answer. If the product is a tactical dashboard, the taxonomy needs pressure and space attributes. If it is broadcast graphics, it needs precise event timing. If it is officiating support, it needs millimetre spatial precision on specific body parts. A taxonomy built without a stated downstream question ends up mediocre for all of them.
2. Labels that cannot be judged from the footage alone. "Intentional foul" requires knowing intent. "Poor first touch" requires a quality judgement. Any label that requires information not present in the video, or that depends on subjective assessment, will produce low inter-annotator agreement no matter how skilled the annotators are. If a label cannot be operationalised into observable criteria, it does not belong in the taxonomy.
3. No defined resolution for ambiguous cases. Real footage contains genuinely unclear moments — was that a shot or a cross, a tackle or a collision? A taxonomy without an explicit tie-breaking rule delegates that decision to each individual annotator, and the result is systematic noise that caps model accuracy.
4. Unbounded label sets. Allowing free-text or open-ended categories feels flexible and destroys consistency. Every category should be closed, with an explicitly defined "other" bucket that is monitored — if "other" exceeds a few percent of labels, the taxonomy is missing a category and needs revision before the dataset grows further.
5. Ignoring class imbalance at design time. Some events are rare by nature — red cards, penalties, specific injury mechanisms. A taxonomy that treats a red card as one of forty equally weighted event types will produce a dataset where the model sees a handful of examples. Rare classes need a deliberate sampling strategy defined alongside the taxonomy, not discovered later. This is the same rare-event problem covered in our guide to building sports CV training datasets.
6. Sport-agnostic taxonomies applied across sports. A generic "shot" definition that spans football, basketball and hockey will be wrong for all three. Events, boundaries and edge cases are sport-specific, which is why annotation quality depends so heavily on domain-expert annotators working from a sport-specific schema.
7. No versioning. Taxonomies evolve. Without an explicit version attached to every annotation record, you lose the ability to know which rules were in force when a given label was created — and with it, the ability to migrate cleanly.
What is AI routing in sports annotation?
AI routing is the logic that determines where each piece of content goes within an annotation pipeline — which annotator, which automated model, which review tier — based on properties of the content itself, such as predicted difficulty, sport, event rarity, model confidence, or client priority.
Without routing, every clip is treated identically: same annotator pool, same automation attempt, same review depth. That is simple, and it is expensive, because it spends senior annotator time on trivially easy footage and lets genuinely hard footage through with the same review as easy footage.
Routing is what makes a pipeline economically viable at volume.
Content routing architecture: incoming footage classified by difficulty and confidence, then directed to automated pre-labelling, standard annotation, or specialist expert review.
Five routing strategies that actually earn their complexity
Confidence-based routing
Run an automated pre-labelling model first. Where model confidence is high, route to light human verification. Where confidence is low, route to full manual annotation by a senior annotator. This is the highest-return routing strategy for most pipelines, because it concentrates expensive human attention exactly where automation is weakest.
The critical detail is calibration: model confidence scores must actually correlate with correctness. An overconfident model routes hard cases to light review, which is worse than no routing at all.
Difficulty-based routing
Classify footage on observable difficulty signals before annotation begins — occlusion density, camera distance, motion blur, number of entities in frame, lighting quality. High-difficulty footage goes to experienced annotators; clean footage goes to the general pool. Unlike confidence routing, this requires no model inference and can run on raw footage properties.
Event-rarity routing
Route clips containing rare events to specialist review regardless of other signals. A clip flagged as containing a possible red card, a specific injury mechanism, or any class where your dataset is thin deserves disproportionate attention, because each example carries disproportionate training weight.
Sport and competition routing
Route by sport, league and competition to annotator pools with matching domain expertise. A cricket specialist should not be labelling ice hockey, and a Premier League specialist may still need calibration for a different competition's tactical conventions and rule interpretations.
Disagreement routing
Where two annotators independently label the same content and disagree, route automatically to a senior adjudicator. This costs a second annotation pass on a sampled subset, and it is the single most reliable mechanism for catching taxonomy ambiguity before it contaminates the dataset at scale.
| Strategy | Requires | Best for | Main risk |
|---|---|---|---|
| Confidence-based | Calibrated pre-labelling model | High-volume, mature pipelines | Miscalibrated confidence |
| Difficulty-based | Footage property analysis | Mixed-quality source footage | Difficulty signals that don't predict error |
| Event-rarity | Preliminary event detection | Imbalanced datasets | Missing rare events at the detection step |
| Sport/competition | Segmented annotator pools | Multi-sport operations | Pool fragmentation reducing utilisation |
| Disagreement | Overlap sampling budget | Taxonomy validation | Cost of duplicate annotation |
How routing and taxonomy interact
These two systems are usually designed by different people at different times, and that separation causes problems. They are coupled.
Routing depends on taxonomy granularity. You cannot route by event rarity if the taxonomy does not distinguish rare events from common ones. You cannot route by difficulty if occlusion state is not part of your annotation schema.
Taxonomy ambiguity surfaces as routing load. If disagreement routing is sending an unexpectedly high share of a particular event type to adjudication, that is not an annotator quality problem. It is a taxonomy problem — the boundary rules for that event are underspecified, and the routing system is telling you exactly which entry to fix.
This makes routing telemetry one of the most useful taxonomy diagnostics available. A well-instrumented pipeline reports disagreement rate per label, and that ranking is a prioritised list of which taxonomy definitions need tightening.
Versioning and migration
Every annotation record should carry the taxonomy version under which it was created. This is unglamorous and it is what makes taxonomy evolution survivable.
Additive changes are safe. Adding a new event type, a new attribute value, or a new context state does not invalidate existing labels — it means older records simply lack the new field. Prefer additive changes wherever possible.
Splitting a label is a migration. If "shot" becomes "shot on target" and "shot off target", every historical shot label needs reclassification. This is tractable if the distinguishing information is recoverable from existing annotation or footage; it is expensive if it is not.
Merging labels is usually safe. Collapsing two categories into one can generally be done programmatically without revisiting footage.
Changing a boundary rule is the dangerous one, because it silently invalidates historical labels without changing their names. A label called "pass" created under version 1 rules may not mean the same thing as a "pass" created under version 2. Without version stamps, these are indistinguishable in the dataset, and the resulting inconsistency is very hard to detect and very damaging to model performance.
A practical design sequence
- State the downstream question first. Write, in one sentence, what decision the model output will support. Every taxonomy entry should trace back to it.
- Draft the four layers separately. Entities, events, attributes, context — do not flatten them.
- Write boundary rules before labelling anything. For each event, define start, end, overlap behaviour and the tie-break for the most likely ambiguous case.
- Pilot on genuinely hard footage. Annotate a small set of deliberately difficult clips with two independent annotators. Measure agreement per label.
- Fix the lowest-agreement entries. Low agreement is a definition problem, not a people problem. Rewrite those rules and re-pilot. For a worked example in one sport, see how we layer definitions in tennis match tagging for coaches.
- Define routing before scaling. Decide difficulty signals and review tiers while volume is still small enough to change them.
- Version and freeze. Stamp the version, then treat changes as deliberate migrations rather than edits.
Steps 4 and 5 are the ones most often skipped, and they are the cheapest possible place to discover a taxonomy flaw. Finding it after 100,000 labels costs several orders of magnitude more than finding it after 200. Our guide to measuring annotation quality covers the agreement metrics worth tracking through this process.
Frequently asked questions
What is a sports content taxonomy? A sports content taxonomy is the structured definition of every entity, event, attribute and context state that annotators label in sports footage and that a model is trained to predict, including the explicit boundary rules determining when each label applies and how ambiguous cases are resolved. It differs from a simple label list precisely because of those rules — they are what make labels reproducible across different annotators and matches.
What is AI routing in an annotation pipeline? AI routing is the logic that decides where each piece of content goes within the pipeline — which annotator, which automated pre-labelling model, which review tier — based on properties like predicted difficulty, model confidence, event rarity, or sport. Its purpose is to concentrate expensive human expertise on the footage that actually needs it rather than spending it uniformly.
Why is taxonomy the most expensive thing to get wrong? Because taxonomy changes are retroactive. Model architecture, tooling and even vendors can be changed while keeping existing labels intact. Changing a taxonomy can invalidate every label already collected under the old rules, forcing a choice between costly re-annotation, an internally inconsistent dataset, or permanently maintaining version-mapping logic.
How many layers should a sports taxonomy have? Four is the practical standard: entities (what is present), events (what happened), attributes (what kind), and context (what was the surrounding situation). Separating these prevents combinatorial explosion — without an attribute layer, every combination of event and modifier becomes its own label, and the label set grows unmanageably.
What is the most common taxonomy design mistake? Designing the taxonomy before defining what question the model output needs to answer. A taxonomy derived from a stated downstream use case is coherent; one designed in the abstract tends to be simultaneously over-specified in areas that do not matter and under-specified in the areas that do.
How do you handle rare events in a sports taxonomy? By identifying them at design time and defining a deliberate sampling and routing strategy alongside the schema, rather than discovering the imbalance after the dataset is built. Rare events also warrant dedicated routing to specialist review, since each individual example carries far more training weight than a common event does.
What is confidence-based routing and when does it work? Confidence-based routing runs an automated pre-labelling model first, sends high-confidence output to light human verification, and sends low-confidence output to full manual annotation. It works when model confidence scores are genuinely calibrated against correctness. If the model is overconfident on hard cases, routing actively harms quality by sending exactly the wrong content to light review.
How should a taxonomy be versioned? Every annotation record should store the taxonomy version under which it was created. Additive changes — new events, new attribute values — are generally safe. Splitting a label requires migration of historical records. Changing a boundary rule without a version stamp is the most dangerous change, because label names stay identical while their meaning shifts, making the inconsistency effectively invisible in the dataset.
Can one taxonomy be reused across multiple sports? Only at the structural level. The four-layer architecture transfers across sports, but the actual entities, events, boundary rules and edge cases are sport-specific. A generic event definition applied across football, basketball and hockey will be subtly wrong in each, which is why sport-specific schema design is a distinct piece of work at the start of every project.
How do you know if your taxonomy has a problem? Track inter-annotator disagreement per label. A label with disproportionately high disagreement almost always indicates an underspecified definition rather than poor annotator performance. Routing telemetry gives you this ranking automatically, making it a prioritised list of which taxonomy entries to tighten first.
The takeaway
Taxonomy and routing are the two pieces of a sports AI pipeline that are cheapest to design well and most expensive to fix later. A taxonomy built from a clearly stated downstream question, separated into entity, event, attribute and context layers, with explicit boundary rules validated against genuinely hard footage, sets the ceiling for everything the model can eventually learn. Routing determines whether reaching that ceiling is economically viable at volume.
Neither is a modelling problem, which is why both are so consistently under-invested in by engineering teams whose instincts point toward architecture. The teams that get this right spend an uncomfortable amount of time on schema design before annotating anything at scale, and it is almost always the reason their datasets hold up.
If you are designing a sports annotation schema or building a routing pipeline and want it pressure-tested by people who have built them across nine sports, see how Train Matricx works or review our annotated dataset results. We start every engagement with schema design and annotate a free pilot clip against it, so you can evaluate both the taxonomy and the output quality before committing to volume.
Client results
“Train Matricx didn't just provide labeling — they became an extension of our team. With 6K+ football matches annotated and counting at 98% accuracy, they are our premier data partner.”
Billy Brush
Founder & CEO, Dark Horse AI
Don't take our word for it. Test the labels.
Send us the footage your current vendor gets wrong — the occluded players, the motion blur, the edge cases. We annotate it and return a QA report you can check frame by frame.
- Your hardest clip, annotated in 48 hours
- Full QA report with per-frame accuracy
- No commitment, no contract, no card
