Vision-language models can now watch match video and answer questions about it, but sports products built on them still need labeled data for three jobs: fine-tuning the model to the sport and your definitions, evaluating it on matches it has never seen, and grounding its answers to exact frames and players. Two strong published specialist systems, MatchVision and the SoccerNet question-answering entry MSUE, lean on annotated or fine-tuned sports data.
A vision-language model (VLM) is an AI model that takes images or video together with text, and answers in text. Ask it "who made the last pass before the goal?" and it looks at the clip and replies. General-purpose models can do this out of the box, and in 2026 FIFA put a video-and-data assistant in front of every World Cup team.
That raises a question many sports-tech founders are now asking: if a model can watch the match, do we still need annotation? This guide looks at what has happened this year, where general models still fall short, and which labeled data still matters.
Key Takeaways
- Sports foundation models are arriving. Carnegie Mellon researchers are building one on official NFL game film and Next Gen Stats tracking data (CMU, November 2026).
- General models still struggle with hard sports reasoning. On the SPORTU benchmark's hard video tasks, the best model tested scored 52.6% (Xia et al., ICLR 2025).
- Specialist systems lean on labeled sports data. MatchVision was built on annotations from 1,988 complete matches (Rao et al., CVPR 2025).
- Labeled data shifts from "label everything" to three targeted jobs: fine-tuning sets, held-out test sets, and frame-level grounding.
What happened with sports AI models in 2026?
Sports AI is moving toward models that combine video, data and language. Three examples from this year, plus the research they build on:
- Carnegie Mellon, using NFL data. Researchers Chenyan Xiong and Ron Yurko are building a football foundation model on official NFL game film and Next Gen Stats player-tracking data. They first built separate pretrained systems for film and for tracking data, and note that combining the two could require more data and more time. Goals include predicting plays, anticipating pre-snap decisions and automating tedious film review (CMU News, 5 November 2026).
- The World Cup. FIFA AI Pro, developed with Lenovo, brought official event and tracking data together with match video in a conversational assistant. FIFA reports that all 48 teams at the 2026 World Cup used it (CGTN, 5 November 2026).
- SoccerNet. The 2026 SoccerNet challenges included visual question answering about football video. One entry, MSUE, routes questions between specialist models, including a fine-tuned Qwen3-VL vision-language model. It reached 0.95 accuracy on the challenge benchmark and placed third, according to its authors' preprint (Li et al., 2026).
- The research behind them. MatchVision, presented at CVPR 2025, is a soccer video model handling event classification, commentary generation and multi-view foul recognition. It was built on SoccerReplay-1988, a dataset with videos and detailed annotations from 1,988 complete matches (Rao et al., 2025).
Both of the last two lean on labeled sports data: one is fine-tuned, the other built on annotated matches.
Where do general-purpose models still fall short?
On hard reasoning about rules and fine detail, the gap is still large. SPORTU, a benchmark presented at ICLR 2025, tested multimodal models on 1,701 slow-motion clips across 7 sports, with 12,048 question-answer pairs ranging from recognising the sport to detecting fouls and applying rules. On the hard video tasks, the best of the 13 multimodal models tested, Claude 3.5 Sonnet, scored 52.6%. On the text-only rules questions, GPT-4o led with 71%, still short of human level. The authors conclude that models fall short on hard tasks that need deep reasoning and rule-based understanding (Xia et al., 2025).
Those were the models available when the benchmark was built, and newer models have arrived since. The direction of the finding still matches what reporting on the 2026 World Cup concluded. CGTN's analysis states that general-purpose AI models are not yet dependable technical coaches, and cites benchmarks showing that even the best multimodal models struggle to identify precise form errors reliably (CGTN).
For a product, three gaps matter most:
- Your definitions. A general model has its own idea of what a pass, a duel or pressure is. Your product needs your definition, applied the same way every time. Our football event data schema shows how definitions and versions live inside the data.
- Precision. "There was a shot around the 30th minute" is not the same as a shot stamped on the frame of the strike, by a named player, from a pitch position.
- Proof. A demo answer that looks right is not a measured accuracy on your footage.
What does labeled data still do for a sports VLM?
Three jobs: teach the sport, prove the model works, and tie every answer to a moment.
Job 1: Fine-tuning teaches the sport and your rules
General models are trained on the open web. Fine-tuning shows them your sport, your camera angles and your definitions. The data looks like clips paired with questions and answers, plus event labels with time, player and outcome. MSUE's visual expert is a fine-tuned Qwen3-VL, and MatchVision was trained on annotations from 1,988 matches. Neither relied on a general model alone.
Fine-tuning data doesn't need to cover every match you will ever process. It needs to cover the situations your product cares about, labeled consistently, including the hard cases: crowded boxes, camera cuts, unreadable shirt numbers.
Job 2: Evaluation proves the model works on your footage
A benchmark score is measured on someone else's footage, questions and definitions. Before a model goes in front of a coach or a customer, you need a held-out test set: matches the model never saw during fine-tuning, labeled to your spec and checked by two reviewers.
Score it per event type and per sport, not as one average, because a model can be strong on goals and weak on duels. A small, carefully labeled test set is worth more than a large, noisy one. Our annotation audit checklist covers how many examples you need for an error rate you can trust, and how to report it.
Job 3: Grounding ties answers to frames and players
An answer you can't trace to a frame can't be checked. Grounding labels give every event a frame index, a stable player ID that survives camera cuts, and a position on the pitch or court. With them, the model can say not just "the left-back lost the ball" but which frame, which player and where, and a reviewer can confirm it.
Grounding needs the precise labels that are hardest to produce: player re-ID that holds through cuts and occlusion, and timestamps tied to a specific video file. Some benchmarks score answers within a window of seconds: SoccerNet's action spotting evaluation, for example, uses tolerances of 1 to 5 seconds for its tight metric (SoccerNet evaluation code). A product that links an answer to a clip needs the frame.
Does this mean less annotation work?
Our view: less labeling of everything, and more labeling of the right things. The shift we expect looks like this:
- From full coverage to targeted sets. Fewer teams will label every event of every match from scratch. More will label focused fine-tuning sets and held-out test sets.
- From labeling to reviewing. As models pre-label more, human work moves toward checking and correcting model output. Our guide on whether AI auto-tagging still needs human QA covers where automatic tags miss.
- From volume to precision. Frame-accurate timing, consistent identities and written definitions become the scarce part.
The CMU project is a useful example, though this is our reading of it. Film study has traditionally meant people manually annotating footage, and automating that is one of the project's goals. But the models in development start from official game film plus Next Gen Stats tracking data, which is itself structured data. That suggests automation builds on structured data rather than replacing the need for it.
What labeled data should you build first?
If you are building a sports product on a vision-language model, start here:
- A written definition set for every event your product reports, versioned.
- A held-out test set from your own footage, labeled by trained annotators and checked by a second reviewer.
- A fine-tuning set focused on the situations where the general model fails your test set.
- Grounding labels for the events your product shows to users: frame index, player ID and position.
- A review loop that sends low-confidence model answers to a human and feeds corrections back into the fine-tuning set.
Build the test set before the fine-tuning set. Without it, you can't tell whether fine-tuning helped.
Frequently asked questions
Can ChatGPT or Gemini analyse a football match on its own?
They can describe clips and answer general questions. Published benchmarks of earlier versions showed them falling short on hard rule-based reasoning and precise detail, and no general model knows your event definitions. For a product, you need to measure them on your own footage first.
What is a sports foundation model?
A large model pretrained on a broad set of sports data, such as game film and tracking data, that can then be adapted to many tasks: predicting plays, tagging events, answering questions. The Carnegie Mellon project on NFL data, announced in November 2026, is one example.
How much labeled data does fine-tuning need?
There is no fixed number. Start with a held-out test set, measure where the general model fails, and label fine-tuning examples for those situations. Coverage of hard cases matters more than raw volume.
Is benchmark accuracy enough to choose a model?
No. Benchmarks use their own footage, questions and definitions. Accuracy on your footage, measured per event type against labels checked by two reviewers, is the number that predicts how the product will behave.
Will vision-language models replace sports annotators?
In our view, they change the work rather than remove it. Labeling shifts toward test sets, targeted fine-tuning data, precise grounding and reviewing model output. Those are the parts general models can't produce for themselves.
Get labeled data your model can learn from
Train Matricx produces the labeled data sports AI teams use to fine-tune, test and ground their models: events to your definitions, precise timing, and player identities that survive camera cuts, checked through three QA layers. Start with a free pilot on your own footage, or see how we handle football event tagging and basketball computer vision annotation.
News items and benchmark figures were checked against the cited articles and papers in November 2026. Views on how annotation work will change are our own. Spot an error? Email info@trainmatricx.com.

Client results
“Train Matricx didn't just provide labeling — they became an extension of our team. With 6K+ football matches annotated and counting at 98% accuracy, they are our premier data partner.”

Billy Brush
Founder & CEO, Dark Horse AI
Don't take our word for it. Test the labels.
Send us the footage your current vendor gets wrong — the occluded players, the motion blur, the edge cases. We annotate it and return a QA report you can check frame by frame.
- Your hardest clip, annotated in 3 business days
- Full QA report with per-frame accuracy
- No commitment, no contract, no card


