football-cvRead the film.
Call the play.
football-cv turns one play of film into a position label for all 22 players, then reads the pre-snap look to predict run or pass and which way the run goes.
The labelled frame is real pipeline output. The play-call card uses made-up example values.
Who is where, and what they are.
Detect and track everyone, split the teams, calibrate to field yards, find the line and the snap, and classify every player.
What the offense is likely to do.
Turn the alignment at the snap into pre-snap features, add the game situation, and predict run or pass with a model trained on your own snaps.
Boxes are easy.
Football is the hard part.
Most player tracking stops at boxes. Turning them into football means knowing which body is a defensive lineman and which is a safety, who is on offense, and where the ball is. That takes camera calibration to field yards, a jersey-colour team split, snap detection, and filtering out everyone who isn't in the play: officials, benches, the chain crew.
Detect and track
YOLO finds every person in each frame; a multi-object tracker turns them into stable ids, and a stitch step merges fragments of one player back into one track.
Split the teams
Jersey colours are clustered into team A, team B and officials, so the officials drop out and offense and defense are known.
Map to the field
Mark at least four field points on one frame and every track is projected to real yards. Other plays from the same game reuse that calibration by feature matching.
Line, snap, positions
It finds the line of scrimmage and the snap, then classifies each player from their field-relative alignment into plain position groups.
| Defense | Read |
|---|---|
DL | on the line of scrimmage |
LB | off the line, inside |
CB | pressed wide, near the boundary |
S | deep, toward the middle |
| Offense | Read |
|---|---|
OL | on the line |
WR | split wide |
QB | deep, centered |
RB | in the backfield |
The labels are standard position groups, never team- or playbook-specific names. It is a toolkit: a one-call Python API, a football-cv CLI, and a zero-build browser viewer for calibration and overlays. Every stage writes its own files per play, so runs are inspectable and resumable.
From the pre-snap look
to run or pass.
An optional add-on. Once a play is classified, football-cv reads the offense's alignment at the snap into one row of features. You add the game situation and your own tags, and an XGBoost model trained on your snaps predicts run or pass, and for runs, left or right.
The look
- Formation as receivers outside the tackles per side, like
3x1or2x2 - Strength: the side with more players outside the tackles
- Backfield: how many running backs
- QB set: gun or under center
- Box count: defenders near the ball
- Pre-snap motion, the hash and field position
The situation
- Down and distance, usually the strongest features
- Quarter, the previous play and its gain
- The game and its date, to split by game
- The labels to learn: run or pass, and run direction
Every column but the game and the label is optional, so you can start with whatever you already tag.
Split by game, never by snap
Games in date order: the latest two are the test set, the one before them is for early stopping, the rest train. A snap never sees its own game in training.
Beat the baselines
Every result sits next to "always the most common play" and a down-and-distance pass-rate table, roughly what a coach does by hand. Not beating the table means the model adds nothing.
When it's sure, is it right?
It reports how often the model is at least 70% or 80% one way, and how often those confident calls are right. That is the number a play-caller actually uses.
Film features inherit every error upstream. They lean on the offense labels, so backfield and QB set are the noisiest; a tight end lands in OL or WR, so formations are generic rather than playbook names; and motion needs the motion man tracked through the last two seconds before the snap.
Results.
Position labels are scored against 21 hand-labelled plays, 162 players. The labels are private; the evaluation tool and the numbers are not.
~70% overall
Across every player, offense and defense, after the learned offense model.
~77% on defense
The strength: full alignment and role logic for DL, LB, CB and S.
~65% without the line
Leaving out OL and DL, the easiest groups, so the headline isn't flattered by them.
| Defense | Precision | Recall | F1 | n |
|---|---|---|---|---|
CB | 1.00 | 0.89 | 0.94 | 18 |
DL | 0.87 | 0.87 | 0.87 | 23 |
LB | 0.83 | 0.69 | 0.75 | 35 |
S | 0.68 | 0.71 | 0.70 | 21 |
| Offense | OL | QB | RB | WR | All |
|---|---|---|---|---|---|
| Geometric rules | 82% | 12% | 47% | 63% | 57% |
| Learned (random forest) | 82% | 50% | 36% | 69% | 61% |
The offense backfield is the hardest part: a QB and an RB look almost the same in field-relative alignment, and the geometric rules found the quarterback once in eight. A small random forest on alignment features, scored leave-one-play-out so no play leaks into its own test, gets him half the time. It is used only for offense, where it helps; an all-position model tied the defense logic and did worse on QB.
Run it on your own film.
Python 3.11+. Detection (torch) and the play-call model (XGBoost) are optional extras, so the core install stays light.
# install pip install -e . # pipeline pip install -e ".[detect]" # + YOLO pip install -e ".[playcall]" # + XGBoost # read the film football-cv ingest myplay --video play.mp4 --data ./data football-cv all myplay --data ./data
# call the play football-cv playcall features play1 play2 \ --data ./data --out snaps.csv # add down, dist, game, run_pass… then football-cv playcall train --data snaps.csv --out model/ football-cv playcall predict --model model/ --data next.csv
As a library
footballcv.analyze(video, calibration, snap) returns a position group for every track id.
The viewer
A browser tool to scrub to a frame, click field points, mark the snap and see the labels drawn over the film.
License
MIT. Use it, fork it, build on it. Issues and pull requests are welcome.