The NFL has collected player tracking data from every snap in every regular season and playoff game since 2017. Every player wears a RFID chip in their shoulder pads. Every ball is chipped. Position data is captured ten times per second. The resulting dataset — marketed as Next Gen Stats — powers broadcast graphics, coaching analysis and betting products across the league.
But RFID tracking has limits. It measures where a player's shoulder pad is. It does not know who is blocking whom, which receiver ran which route, whether the quarterback's release mechanics changed after an injury, or what the defensive formation was at the snap. Answering those questions requires computer vision — models that watch the footage and extract information that sensors cannot.
This guide covers how computer vision works in American football, why the sport creates challenges that other sports do not, and what training data requirements look like for teams building NFL-grade models.
Premium sports analytics visualization showing 22-player tracking, offensive formation classification, and receiver route paths on an American football field.
What is American football computer vision?
American football computer vision is the use of AI to interpret football footage — tracking all 22 players simultaneously, classifying offensive and defensive formations, recognising route combinations, detecting blocking assignments, analysing quarterback mechanics, and converting match video into structured data for coaching, scouting, broadcast and AI model training.
American football is unique among major sports in that every play is a discrete, structured event — a formation sets, the ball is snapped, a sequence of coordinated movements executes, the play ends with a tackle or out-of-bounds. This structure makes some CV problems more tractable than in continuous-flow sports. It makes others significantly harder, particularly around the simultaneity of 22 players executing specific assignments at the same instant.
Why American football is uniquely challenging for computer vision
American football presents a configuration of challenges that does not exist in any other mainstream sport.
Twenty-two players in simultaneous coordinated motion. At the snap, every player on the field begins moving simultaneously — eleven players on offence executing a designed play, eleven on defence executing a designed scheme. A detection and tracking model must maintain identity for all 22 players across the first few seconds of the play, when players are at maximum density near the line of scrimmage and collisions, overlaps and pileups are densest.
Helmets remove face-based identity. In other sports, jersey numbers and player appearance support re-identification. In American football, helmets cover the head and face entirely. Re-identification relies on jersey number, body type, and positional context — and jersey numbers may not be readable when players are viewed from above, from behind, or when they are in physical contact. A player on the ground in a pile may have no readable jersey number and no distinguishing visual feature beyond body size and approximate position.
The line of scrimmage pile. The offensive and defensive lines collide at the snap and form a dense physical cluster of large players engaged in hand-fighting, pushing and grappling. From any camera angle above field level, this cluster is partially or fully opaque — players inside the pile are hidden, their positions ambiguous, their identities unverifiable by visual means. This matters for offensive and defensive line analysis, which is one of the least-analysed and most strategically important parts of the game.
Play clock and dead ball complexity. American football alternates between live play — typically four to seven seconds — and extended dead ball periods between snaps. The model must handle rapid transitions between a static formation state (pre-snap) and a maximum-density tracking state (post-snap) and back to a static state (post-play), while maintaining player identity across both states and through the physical contact of the play.
Camera geometry at the broadcast level. Standard broadcast cameras in American football follow the ball, which means the wide camera view that captures all 22 players is frequently an end zone or press box angle that compresses the field into a flat perspective. Position-map analysis that converts pixel positions to real field coordinates requires either multiple camera calibration or a model trained specifically to handle the perspective distortion of the standard broadcast frame.
Route and assignment classification without pre-play knowledge. In analysing a passing play after the fact, the question is not just "where did each player go?" but "what route was each receiver running?" and "was that player in their assigned block?" These questions require understanding both the movement trajectory and the pre-snap formation context — linking what happened to what was designed to happen.
The five applications of American football computer vision
| Application | What it produces | Primary buyers |
|---|---|---|
| Player tracking | All-22 positions, speed, distance per play | NFL clubs, coaching platforms, Next Gen Stats |
| Formation recognition | Pre-snap offensive and defensive formation classification | Coaching, scouting, opponent preparation |
| Route tracking | Receiver route trees, separation metrics, target depth | Offensive analytics, quarterback evaluation |
| Blocking assignment analysis | Pass protection mapping, run block success, gap assignment | Offensive line coaching, defensive analytics |
| Quarterback mechanics | Release time, arm angle, drop back mechanics | Player development, injury risk, scouting |
Each application requires different training data. Formation recognition needs pre-snap frame labels with formation classifications. Route tracking needs frame-accurate movement labels across the full route tree taxonomy. Quarterback mechanics analysis needs dense skeletal keypoints at specific phases of the throwing motion.
Player tracking in American football: the 22-player problem
Tracking all 22 players simultaneously across a play is the foundational challenge in American football computer vision. The solution that the NFL uses — RFID chips in shoulder pads — sidesteps this challenge. Computer vision systems that work from video footage alone must solve it directly.
Pre-snap identification
Before the ball is snapped, players line up in formation. In this static or near-static state, player identification is most reliable — jersey numbers are visible, body positions are stable, and players are separated enough that detection is relatively clean. The pre-snap frame is the best opportunity to establish ground-truth identity for all 22 players before the chaos of the play.
Training data for pre-snap identification should include labeled formations where every player identity is confirmed from jersey number and cross-referenced with roster data. This pre-snap identity serves as the anchor for tracking through the play.
Through-the-snap tracking
At the snap, all 22 players begin moving. The offensive line and defensive line immediately collide, creating the densest occlusion scenario in sport. Behind the line, receivers, running backs and linebackers spread across different parts of the field at varying speeds and directions.
Tracking through the snap requires models trained on the specific sequence of events: players transitioning from the pre-snap formation state, through the contact and collision of the snap and the first two seconds of the play, to the post-contact dispersal phase. Training data needs to represent the full range of play types — running plays, passing plays, special teams — because each creates a different post-snap tracking scenario.
Identity through contact
On running plays, multiple players may be stacked on top of each other — ball carrier, lead blocker, multiple defenders — for extended frames. From a coach's-eye-view camera, a running play into a crowded box may have eight to twelve players within a five-yard radius, with several in full physical contact. Re-identifying each player as they separate from the pile requires training data that explicitly handles this stacked-player occlusion, not generic occlusion handling designed for players briefly crossing paths.
Formation and route recognition
Formation recognition is one of the highest-value applications of American football computer vision because it directly addresses the question coaching staffs spend most time on: what did the opponent run, and what tendencies does it reveal?
Pre-snap offensive formation classification (11 Personnel / Shotgun) and receiver route mapping showing trajectory paths in real-time.
Pre-snap formation classification
Offensive formations in American football follow a taxonomy with significant depth. A "21 personnel, I-formation" is different from a "21 personnel, split-backs" despite having the same number of running backs and tight ends. A "11 personnel, trips right" is different from "11 personnel, trips left, motion to slot right." Defensive formations have equal complexity — Cover 2, Cover 3, Tampa 2, Cover 4, quarters, man-under, and their countless variations and disguises.
Classifying these formations from a single frame requires a model that understands the rules of formation — where the tight end is relative to the line, how many receivers are split wide, where the extra defensive back is aligned. This is not purely a visual task. A correctly-annotated training dataset for formation recognition must include labels assigned by someone who understands football formation taxonomy, not just someone who can count players in zones.
Route tree annotation
After the snap, receivers run routes from a defined taxonomy: slant, out, curl, comeback, dig, post, corner, go, flat, wheel, screen. Coaching and scouting platforms want to know which receiver ran which route on every passing play — both for understanding what the offence designed and for analysing whether receivers created separation on their routes.
Annotating route trees from video requires:
- Frame-accurate start points (the snap) and end points (the completion, incompletion, sack or scramble)
- Movement trajectory for each receiver throughout the route
- Route type classification from the standard taxonomy
- Separation measurement — how far from the nearest defender the receiver was at the target point
Route annotation is one of the most knowledge-intensive sports annotation tasks. The difference between a dig and a crossing route, or between a comeback and a curl, is in the receiver's mechanics and depth — distinctions that require football knowledge, not just observation of player movement.
The line of scrimmage: the unsolved problem
The most physically contested and strategically complex part of American football — the offensive and defensive line collision — is also the part that computer vision handles least well.
Offensive and defensive linemen are the largest players on the field. They operate in a space no wider than five yards across the line of scrimmage. At the snap, they engage in hand-fighting and leverage battles where success is measured in inches. The outcome — did the offensive tackle successfully block the defensive end? Did the guard pull correctly to lead the running play? — determines the result of the play more often than any other position group.
From any overhead or broadcast camera angle, this battle is partially or fully hidden. The players are too close together, too large, and too physically intertwined for standard detection approaches to separate cleanly.
Solving line-of-scrimmage tracking requires:
- Multi-camera setups including end zone cameras that provide a different angle on the line
- Training data specifically focused on lineman pre-snap alignments and post-snap movements
- Labeling strategies that handle the dense-occlusion scenario with explicit annotations for players who are partially or fully hidden
- Domain experts who understand blocking technique and can interpret what is happening inside the pile from available visual information
This is an active research area. No commercial system has fully solved it yet, which makes it one of the highest-value unsolved problems in sports AI.
How NFL Next Gen Stats works
NFL Next Gen Stats is the official player tracking data product of the NFL. It is produced by Zebra Technologies through RFID chips embedded in every player's shoulder pads and in the football. The system produces:
- Player position in field coordinates at 10 Hz per player
- Speed and acceleration per player
- Ball position at 10 Hz
- Basic derived metrics: distance run, top speed, separation at the catch point
What Next Gen Stats does not produce directly from sensors: what formation the team was in, what route each receiver ran, what blocking assignment each lineman had, or why a play succeeded or failed. These questions require video analysis — either human film study or computer vision.
The NFL's computer vision work, through partnerships with companies including AWS, attempts to extract these tactical dimensions from the combination of RFID tracking data and video footage. The RFID data provides clean positional input. Computer vision provides the classification layer — formation, route, blocking assignment, outcome — on top of the positional data.
For teams and analytics companies working outside the official Next Gen Stats system, the equivalent problem is building computer vision models that can extract both the positional and the tactical layer from video alone, without access to RFID data.
Who uses American football computer vision?
NFL clubs use computer vision for opponent film analysis, formation tendency scouting, route tree statistics, quarterback evaluation and draft scouting. Every NFL team employs analysts who conduct video-based formation and play analysis; the long-term direction is automating this analysis through computer vision.
The NFL and broadcast partners use Next Gen Stats-derived products for broadcast graphics — receiver separation at the catch point, defensive back closing speed, quarterback release time — as standard on-screen elements during game broadcasts.
Sports analytics platforms build products for NFL teams that codify and automate the film study process. Formation recognition, route charting, coverage classification and pressure charting are common products. These require computer vision models trained on annotated NFL footage.
Sports betting and fantasy operators consume game-state data, target share information, air yards statistics and next-play predictive models. Real-time computer vision systems that can extract this from broadcast footage faster than manual data entry provide a commercial advantage.
AI research labs use NFL footage as a benchmark for multi-agent tracking, coordinated action recognition and formation prediction research. The structured nature of the game — fixed teams of eleven, discrete plays, well-defined rules — makes it a useful research environment.
What American football computer vision training data requires
For player tracking:
- Bounding boxes for all 22 players on every frame of every play
- Persistent player IDs maintained through the snap, contact phases and play resolution
- Jersey number annotations confirmed against roster data for ground-truth identity
- Field coordinate positions where camera calibration allows
- Play phase labels: pre-snap, snap, live play, dead ball
For formation recognition:
- Pre-snap formation labels for every play using a standardised taxonomy
- Player position labels in formation-relative coordinates (left tackle, strong-side linebacker, etc.)
- Motion and shift labels where players move pre-snap
For route tracking:
- Frame-accurate route trajectories for every eligible receiver on every passing play
- Route type classifications from a documented route tree taxonomy
- Separation measurements at the target point
- Route outcome labels: targeted, not targeted, touchdown, first down, incomplete, interception
For quarterback analysis:
- Skeletal keypoints at defined phases of the throwing motion: setup, wind-up, release, follow-through
- Release time measurements from snap to ball release
- Pressure classification: clean pocket, pressured, scramble
- Drop back type: under centre, shotgun, pistol
For blocking analysis:
- Assignment labels linking each lineman to their blocking target pre-snap
- Outcome labels: successful block, beaten, penalty
- Technique classification where annotator knowledge allows
Frequently asked questions
What is American football computer vision? American football computer vision uses AI to interpret football footage — tracking all 22 players simultaneously, classifying offensive and defensive formations, recognising receiver routes, analysing quarterback mechanics and producing structured data for coaching, scouting, broadcast and AI training. It extracts the tactical and biomechanical information that RFID sensors cannot provide.
How does NFL player tracking work? The NFL's official player tracking system uses RFID chips embedded in every player's shoulder pads and in the football. The chips emit signals captured by receivers installed in stadium infrastructure, producing positional data at 10 Hz per player. This data powers Next Gen Stats metrics. Separately, computer vision systems analyse game footage to add tactical layers — formation, route, blocking assignment — that sensors alone cannot classify.
What is Next Gen Stats in the NFL? Next Gen Stats is the NFL's official player tracking data product, produced in partnership with Zebra Technologies and distributed by the league and its broadcast partners. It provides player positions, speed, acceleration and derived metrics for every player in every game. Computer vision systems sit on top of this positional data to add formation recognition, route classification and tactical analysis.
Why is American football harder than other sports for computer vision? American football has 22 players executing simultaneous coordinated movements from a standing start at every snap. The offensive and defensive lines collide immediately, creating the densest occlusion scenario in mainstream sport. Helmets prevent face-based re-identification. Route and formation classification require understanding football strategy, not just observing movement. No other sport combines this player density, this occlusion severity and this level of positional assignment complexity in a single tracking problem.
What is formation recognition in American football AI? Formation recognition is the automated classification of offensive and defensive alignments from a pre-snap frame. The system identifies how many running backs, tight ends and wide receivers are on the field and where they are aligned, then classifies the formation against a defined taxonomy. This tells coaching and scouting staffs what personnel packages and formation tendencies opponents favour in specific game situations.
What is route tracking in sports AI? Route tracking is the automated labelling of the movement path each eligible receiver runs after the snap. The system tracks the receiver's trajectory, classifies the route type from a standard taxonomy (slant, curl, out, post, go, flat, etc.) and measures separation from the nearest defender at the target point. Route tracking data feeds receiver efficiency analysis, quarterback decision-making evaluation and coverage scheme analysis.
How do NFL teams use computer vision for scouting? NFL teams use computer vision to automate and scale the film study process. Formation tendency analysis — what formations does an opponent run in third-and-long from their own 30? — previously required analysts to manually log every play in an opponent's film. Computer vision systems that recognise formations and routes automatically can process a full season of opponent footage in hours rather than weeks, giving analysts more time to interpret data and less time collecting it.
What training data does an American football AI model need? It depends on the use case. Formation recognition models need pre-snap formation labels across a taxonomy for every play. Route tracking models need frame-accurate receiver trajectories and route type classifications. Player tracking models need 22-player bounding boxes with persistent IDs across the full play sequence. Quarterback analysis models need skeletal keypoints at defined throwing motion phases. All American football CV models need training data annotated by people who understand football — the formation classifications and route labels require football knowledge, not just annotation tooling.
What makes American football annotation different from other sports? American football annotation requires football knowledge at every level. Formation classification requires understanding offensive and defensive alignment rules. Route labeling requires understanding the route tree taxonomy. Blocking assignment analysis requires understanding blocking scheme and technique. Lineman tracking requires annotation strategies for the dense-occlusion scenario at the line of scrimmage. Generic annotators who can label visible players by position cannot reliably produce any of these higher-level labels.
The takeaway
American football is the most strategically complex team sport for computer vision. The 22-player tracking problem, the formation and route classification requirements, the line-of-scrimmage occlusion challenge and the football knowledge required to label any of it correctly make it one of the highest-bar annotation environments in professional sports.
The NFL market is also one of the most commercially significant in sports AI — the league generates over $20 billion in annual revenue, and the demand for AI-driven coaching, scouting, broadcast and betting products continues to grow every season.
If you are building American football computer vision models and need expert-annotated training data — player tracking, formation recognition, route trees, quarterback mechanics or blocking analysis — see how Train Matricx works or review annotated dataset results in our case studies. We annotate a free pilot clip so you can evaluate quality before committing to any volume.
Written by
Train Matricx Team
