An annotation vendor is productive when its annotators clear a calibration gate on your schema, not when the contract is signed. That gate has four things in front of it: a frozen schema, a gold-standard set built from your own footage, training on the edge cases rather than the volume, and a measured agreement check. Teams that skip straight to production footage appear faster for about a week, then spend longer fixing the first deliveries than the gate would have cost.
Ramp is the part of an annotation engagement nobody quotes for and everybody pays. It decides whether your first delivery is usable, and it is the reason two vendors quoting the same price per match can differ by a month on time-to-usable-data.
This guide sets out what the ramp actually consists of, what published research says it costs in labour, where it fails, and the questions that tell you whether a vendor has a real onboarding process or is improvising on your footage.
Key Takeaways
- "Productive" is a measurable state, not a date. Define it as annotators passing an agreement check against gold-standard clips you control.
- Annotation is labour-bound. SoccerNet-v2 reports roughly 105 minutes per game to annotate actions, 140 minutes for camera changes and 70 minutes for replay grounding — about 1,600 hours across 500 games (Deliège et al., 2021).
- Freeze the schema before anyone is trained. Changing class definitions later invalidates everything labelled under the old ones.
- Errors introduced during ramp surface late. 92% of AI practitioners in a Google study had watched a data problem appear downstream in a live system (Sambasivan et al., CHI 2021).
- Our own gate: agreement of κ ≥ 0.80 against held-back gold clips cut from the client's footage, two attempts, before an annotator touches production work. Ask any vendor for theirs.
What does "productive" mean for an annotation vendor?
Productive means annotators are producing labels that pass your acceptance bar without rework, on your schema, at volume. It does not mean people have been assigned, a kickoff call has happened, or the first batch has been delivered. Those are all compatible with output you will have to throw away.
The distinction matters because the two definitions behave differently under pressure. A vendor optimising for "started" puts annotators on client footage in the first days and lets the reviewers absorb the error rate. A vendor optimising for "passing" spends those days on definitions and calibration, delivers nothing in week one, and then delivers work that does not come back.
The second is faster to usable data in every engagement long enough to matter. It only looks slower if you measure the wrong thing, which is why the metric to agree in advance is first-pass acceptance rather than delivery date. We cover how to measure that, and the rest of the quality signals, in our guide to telling whether an annotation vendor's labels are good.
There is also a commercial question hiding here. Calibration work consumes annotator hours, and somebody pays for them. Ask directly whether the ramp is billed, absorbed, or folded into the per-match rate, because a vendor that bills it has an incentive to make it long and a vendor that hides it has an incentive to make it short.
Why ramp costs more than it looks
Annotation is labour-bound, so every hour spent training is an hour not spent producing — and the production hours themselves are larger than most buyers assume. The clearest public evidence comes from SoccerNet-v2, whose authors documented their own annotation effort.
The paper states that "annotating all the actions of a single game takes ~105 minutes; annotating all the camera changes requires ~140 minutes per game, while only associating each replay shot of a game with its action takes ~70 minutes" (Deliège et al., SoccerNet-v2, arXiv:2011.13367, §7.2). Across the full dataset they "hired 33 annotators for the annotation process, all frequent observers of soccer, for a total of ~1600 hours of annotations" covering 500 games.
Three things in that are worth sitting with.
The tasks are additive. Those three figures are separate passes over the same match. Together they come to roughly five and a quarter hours of annotator time per game, and that is before any review layer.
The annotators were pre-selected for domain familiarity. "All frequent observers of soccer" is a recruitment filter, applied by a research team building a benchmark. They did not train football knowledge; they hired for it and trained the schema. That is the same reason domain-expert annotators reach a usable standard faster than generalists on sport-specific work.
This was a research dataset with a fixed, published schema. A client engagement adds a schema that did not exist last month, footage conditions nobody has annotated before, and an acceptance bar set by someone else. Ramp is where those three get resolved.
For what the same arithmetic does to a pipeline at peak volume, see our breakdown of annotation capacity planning.
Step 1: Freeze the schema before anyone is trained
A schema that is still moving cannot be trained against, and every change made after labelling starts invalidates the work done under the previous definition. This is the single most expensive sequencing error in annotation projects, and it is almost always made for a sympathetic reason: the team wants to start producing data while the taxonomy is "nearly final".
Nearly final is not final. If the boundary between two event classes moves in week three, every match labelled in weeks one and two carries the old boundary, and you now hold a dataset with two incompatible definitions inside it and no flag distinguishing them.
Freezing means writing the class list down with explicit rules for the cases that sit on a boundary, and agreeing that changes after the freeze are a versioned schema change with a re-labelling cost attached, not an email. Our guide to sports event schema design covers how to structure that taxonomy so the boundaries are decidable in the first place.
Our standard: any change after the freeze becomes a new schema version, and annotators re-run the agreement gate on the changed classes before they label anything under it.
The freeze is also what makes a fast ramp possible at all. On the Dark Horse AI engagement we recruited and trained a dedicated team of 30+ annotators within one week, on the client's own 30+ class taxonomy — which was only achievable because the class definitions were settled before the team existed rather than negotiated while they worked.
Step 2: Build the gold-standard set from your own footage
A gold-standard set is a small collection of clips, labelled to the agreed schema and treated as correct, that every annotator is measured against before and during production. It is the instrument the whole ramp is calibrated with, and its quality caps the quality of everything downstream.
Three decisions determine whether it works.
Who authors it. The gold set has to be labelled by whoever holds final authority over the schema — typically the client's own lead, or the vendor's senior reviewer working from the client's rules, with the client signing off. A gold set authored by the same population it will be used to test measures nothing.
What is in it. The instinct is to pick representative footage. The better choice is to over-weight the hard cases: the occlusions, the boundary events, the conditions your existing pipeline already gets wrong. Clips where everybody agrees carry no information about whether an annotator is calibrated.
Whether it stays hidden. Annotators can be trained on part of it, but a portion has to be held back, or the measurement becomes a memory test. The held-back portion is what the gate in step 4 uses.
Our standard: at least 20 clips cut from the client's own footage, most of them chosen from the hard cases, with a quarter held back from training entirely. The client signs off the gold labels before training starts, so the instrument is theirs, not ours.
Example at the 20-clip minimum: hold back five clips before training begins so the gate measures calibration rather than recall.
Step 3: Train on the edge cases, not the volume
Annotator training should spend most of its time on the decisions that are genuinely ambiguous, because the easy labels do not need teaching and the ambiguous ones are where agreement collapses. A schema with 30 classes does not have 30 difficult decisions in it. It has perhaps six boundaries that actually generate disagreement, and those six deserve the session.
The pattern that works is adversarial rather than instructional: show the clip, have annotators commit to a label independently, then reveal the disagreement and resolve it against the written rule. Where the rule does not resolve it, the rule is incomplete, and that is a schema defect found in training rather than in delivery.
This is also the cheapest moment in the entire engagement to discover that a class definition is undecidable. Finding it in week one costs a training session. Finding it in month two costs a re-labelling programme.
Step 4: The agreement gate before client footage
The gate is a measured check: annotators independently label the held-back gold clips, and their agreement with the gold labels and with each other has to clear an agreed bar before they are released onto production footage. This is the step that converts "we trained them" into something you can inspect.
Inter-annotator agreement is usually reported as a kappa statistic, and the interpretation bands in general use come from Landis and Koch, who set out slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80) and almost perfect (0.81–1.00) (Landis & Koch, Biometrics 33(1):159–174, 1977).
Two cautions belong with that scale, and a vendor that states them is a vendor that has used it.
The bands are conventions, not thresholds. Landis and Koch offered them as useful benchmarks rather than as a statistical test, and they have been criticised ever since for being applied as though they were one. A kappa of 0.62 is not a pass and 0.59 is not a failure; they are two readings either side of an arbitrary line.
Kappa is schema-sensitive. Agreement falls when classes are ambiguous, even with perfectly trained annotators, so a low score is as likely to be evidence about your taxonomy as about the people. That diagnostic value is the main reason to run the gate at all.
The practical form of the gate is simpler than the statistics: set the bar with the vendor in advance, in writing, and require that the measurement is run on clips the annotators have not seen. What the number should be depends on the task and the schema. That it exists, and that somebody checked it before your footage was touched, does not.
Our bar is a kappa of 0.80 or higher against the gold labels on the held-back clips, and no uncorrected identity switch on the held-back tracking clips. We set it at the top of the substantial band rather than deep inside almost perfect on purpose. On a schema with 30 or more classes, some of the remaining disagreement above that line reflects boundaries that are genuinely ambiguous, and a higher bar would start grading the taxonomy instead of the annotator.
An annotator gets two attempts, with a calibration session on their specific disagreements in between. Missing the bar twice means no client footage.
Step 5: Staged release onto client footage
Passing the gate earns an annotator access to production work under heavier review, not unsupervised production. The review ratio starts high and comes down as the evidence accumulates, which means the first weeks of "production" are still partly ramp and should be planned as such.
What makes this affordable is that the review layers are looking for different things. On the Dark Horse engagement every match runs through three: a primary annotator applying the labels, a specialist reviewer cross-checking complex multi-tag events and verifying that player IDs survive occlusions, and a senior auditor confirming the delivery matches the client's schema exactly. Identity errors in particular are invisible in any single frame and only appear when a reviewer follows a player across a passage of play.
Our standard: every event in a new annotator's first 10 production matches is re-checked at the second layer rather than sampled, with one reviewer for every five annotators during that period. An annotator moves to the normal review scope once 95% of their matches pass the second layer first time across 10 consecutive matches.
That exit bar sits deliberately below our long-run rate of 98% of deliveries accepted by the client first time. It marks the point where an annotator no longer needs full re-checking, not the point where they have finished improving.
The staged release is also where the first real acceptance data appears. Until client footage has been delivered and reviewed by the client, every quality number is an internal one.
What skipping the gate actually costs
Errors introduced during ramp do not announce themselves at delivery. They surface downstream, after training, when the cost of correction is highest. The Google research team that studied this called the pattern data cascades: "compounding events causing negative, downstream effects from data issues".
Their finding is blunt. From semi-structured interviews with 53 AI practitioners working on high-stakes applications in India, East and West African countries and the USA, they report that "data cascades are pervasive (92% prevalence), invisible, delayed, but often avoidable" (Sambasivan et al., CHI 2021).
Invisible and delayed are the operative words. An uncalibrated annotator does not produce obviously broken labels. They produce plausible labels applied under a slightly different interpretation of the schema than their colleagues, which reads as noise in your training set and caps model accuracy without ever failing a review.
Avoidable is the third word, and the gate is how.
What a one-week onboarding actually required
Our fastest documented ramp is the Dark Horse AI engagement, where a dedicated team of 30+ sport-domain annotators was recruited, onboarded and trained within one week of kickoff, and the client's accumulated backlog cleared in 15 days. The engagement has since passed 6,000 matches at 98% accepted first pass on a 24-hour turnaround.
The week was not the achievement. The conditions that made the week possible were:
- The taxonomy was the client's, and it was settled. Annotators trained on Dark Horse AI's own 30+ class definitions, not on generic football labelling, before touching client footage.
- The annotators already knew the sport. Training covered the schema, not the game — the same filter SoccerNet's team applied when they hired frequent observers of soccer.
- The review architecture existed before the volume arrived. Three layers, each looking for a different class of error, staffed from day one rather than assembled after the first rejected delivery.
The full account, including what the previous vendors' labels looked like on arrival, is in the Dark Horse AI case study.
Our ramp standards at a glance
These are the bars we set for every new engagement. They are commitments, not averages, and a client can ask to see any of them measured on their own project.
| Gate | Our standard |
|---|---|
| Schema freeze | Changes after the freeze are versioned; changed classes re-run the agreement gate |
| Gold-standard set | At least 20 clips from the client's footage, mostly hard cases, a quarter held back; client signs off the labels |
| Training | Built around the class boundaries that generate disagreement, not around volume |
| Agreement gate | κ ≥ 0.80 against gold on held-back clips; no uncorrected ID switch on tracking clips; two attempts |
| Staged release | First 10 matches fully re-checked; one reviewer per five annotators; exit at 95% first-time pass over 10 consecutive matches |
| Settled production | Three review layers on every match; 98% of deliveries accepted first time (Dark Horse AI, 6,000+ matches) |
Seven questions to ask about a vendor's ramp
- What has to be true before your annotators touch our footage?
- Who authors the gold-standard set, and do we sign off on it?
- What agreement bar do annotators have to clear, and on how many held-back clips?
- What happens to an annotator who does not clear it?
- What is the review ratio in the first two weeks, and when does it come down?
- Is calibration time billed to us, absorbed, or priced into the per-match rate?
- If our schema changes in month two, what is the re-labelling cost and who carries it?
Question 2 is the most revealing. A vendor with a real process will have an opinion about who should author the gold set and why, and will want your sign-off on it. A vendor without one will tell you they use experienced annotators.
For the wider vendor evaluation this sits inside, see our buyer's checklist for sports annotation companies.
Frequently asked questions
How long does annotator onboarding take? It depends on whether the schema is frozen and whether annotators already know the sport. With both settled, a trained team can reach production in around a week, as on our Dark Horse AI engagement. Building an in-house team from scratch is a different exercise: our in-house versus outsourced breakdown puts recruiting and ramp at four to eight weeks before a new hire produces reliable sport-specific labels, and that estimate assumes the schema is already written.
What is a gold-standard set in annotation? A small set of clips labelled to the agreed schema and treated as ground truth, used to train annotators and then to measure them. Part of it is held back so the measurement tests calibration rather than recall.
What inter-annotator agreement score should I require? There is no universal number, and any vendor quoting one without asking about your schema is quoting a convention. Landis and Koch's bands — substantial at 0.61–0.80, almost perfect above 0.81 — are widely used as reference points but were offered as benchmarks rather than thresholds. Set the bar with the vendor against your own gold set and task. Ours is κ ≥ 0.80 on held-back clips from the client's footage.
Should we pay for the ramp period? That is a commercial decision, but it should be an explicit one. The failure mode is a ramp that is neither billed nor planned, which is the arrangement most likely to be cut short.
Can a vendor start on our footage immediately? They can. What they cannot do is tell you whether the output is calibrated, because without a gold set and an agreement check there is no measurement — only the first client review, which happens after the work is done and paid for.
See the ramp on your own footage
The only reliable way to evaluate a vendor's onboarding is to watch it happen on data you can check. Send the clip your current pipeline gets wrong — the occlusion-heavy passage, the boundary events your schema argues about — and we will annotate it under full QA and return it with a quality report.
You will see the schema questions we ask before starting, which is the part of the ramp that predicts everything after it.
Client results
“Train Matricx didn't just provide labeling — they became an extension of our team. With 6K+ football matches annotated and counting at 98% accuracy, they are our premier data partner.”
Billy Brush
Founder & CEO, Dark Horse AI
Don't take our word for it. Test the labels.
Send us the footage your current vendor gets wrong — the occluded players, the motion blur, the edge cases. We annotate it and return a QA report you can check frame by frame.
- Your hardest clip, annotated in 48 hours
- Full QA report with per-frame accuracy
- No commitment, no contract, no card
