AI should skim a video before trying to explain it

A fifteen-minute game recording can contain tens of thousands of frames. Sending an AI every one of them would be wasteful, but selecting only the important moments first seems to require already knowing the story. The expensive analysis needs an index, while the index risks erasing the exact detail that makes a moment important.

That circular problem appeared while reconsidering LeagueLoop, a post-match coaching product for League of Legends. The current system reads official match records and event timelines, then produces a small set of evidence-backed lessons for the player. Video had been treated as an optional supplement: structured data would find an uncertain moment, and a few images could clarify what the logs could not see.

I proposed reversing that relationship. A screen recording from the player's point of view should be able to produce a useful review on its own, even when official game logs are unavailable. One low-quality frame per second could reconstruct the broad match story—but if that sparse pass became the final evidence, the product would confuse knowing where to look with knowing what happened.

The source of truth moved

The existing architecture starts with metadata because it is cheap, structured, and auditable. It can establish deaths, purchases, positions at sampled times, objectives, and broad changes in advantage. However, it cannot reliably establish the exact wave shape, the player’s camera, momentary ability availability, cursor movement, or which enemies were actually visible when a decision was made.

The earlier video plan preserved that hierarchy. Metadata would select a handful of windows, while visual analysis answered one narrow question inside each window. That kept the work bounded, though it also made official data a prerequisite for finding the story.

The user correction changed the product premise: the recording was not merely extra evidence because it was the player's actual experience of the match. It showed what appeared on the screen, where the camera moved, what the interface exposed, and what unfolded between the periodic positions retained by the logs. Official records could still corroborate exact events, but they should not overrule what the recording established about the player’s visible information.

This was not another input adapter. Moving the source of truth changed the intake, privacy boundary, evidence model, failure behavior, and the first experiment worth running.

A skim is useful because it is incomplete

One frame per second is enough to notice broad transitions: the player left the lane, opened the shop, moved toward another area, entered a fight, died, or returned to base. Across the first fifteen minutes, that creates 900 timestamped observations instead of asking a model to treat the entire recording as one undifferentiated object.

It is also guaranteed to miss things. A quick trade can begin and end between samples. A brief enemy appearance may never reach the ledger. Ability order, cursor precision, reaction timing, and momentary interface changes cannot be reconstructed honestly from a one-second interval.

That limitation is not a reason to discard the sparse pass because it defines its proper job. The first pass should build an inspectable index, mark transitions, and record uncertainty; it should not produce the final coaching claim.

1. 01Sample one low-detail frame per second
2. 02Build a timestamped observation ledger
3. 03Detect transitions and unresolved questions
4. 04Return to the original recording
5. 05Re-sample selected windows at higher quality and density
6. 06Validate visual observations
7. 07Produce coaching from the combined evidence

The second pass would return to the original video around a small number of candidate moments, perhaps using several frames per second and focused interface crops. Instead of asking for a general account of the match, it could ask whether the wave was prepared before a roam, which threats were visible before a commitment, or what resources remained before a fight.

The AI is allowed to skim only because the product retains the source and knows when to look again.

Spend detail where it can change the answer

Coarse-to-fine analysis is attractive because it reduces cost, though the more important benefit is epistemic: every level of detail has a named responsibility.

The cheap pass answers, “Where did the situation change?” The detailed pass answers, “What can this evidence actually support?” Deterministic code joins the timeline, preserves timestamps, separates observation from interpretation, and prevents an unreadable frame from quietly becoming a confident fact.

That division also creates a useful product test. The first prototype does not need an upload interface or a complete coaching pipeline. It can process one owned recording locally, produce only a timeline with thumbnails and confidence, and ask whether a human recognizes the story of the early game. A second experiment can revisit three detected moments at higher density and measure whether the additional frames correct or materially enrich the sparse account.

If detailed inspection does not improve the explanation, the expensive second pass has not earned its place. If the sparse index cannot identify the broad story, building the rest of the product would only wrap an unreliable foundation in more interface.

Rich evidence brings richer privacy problems

Official game records are narrow compared with a screen recording. Video may contain player names, chat, voice, notifications, or even another application if the capture is wrong. Making the recording primary therefore expands the product's responsibility before a single coaching result exists.

The proposed boundary would discard audio, isolate the game viewport, redact names and chat, show the user which derived frames will be analyzed, and delete the original automatically unless retention is explicitly requested. Cancellation and failures would also need to remove temporary media, while every visual claim would retain the timestamp, frame provenance, readability, and confidence that support it.

These are not finishing touches because the same recording that makes the product useful also makes careless processing unacceptable. A cheaper inference path is not a successful architecture if it keeps unnecessary personal material or hides what crossed the model boundary.

As of this Note's publication date, LeagueLoop remains a metadata-only product and the recording-primary spike has not been implemented or validated. The revised design is evidence of a sharper hypothesis, not evidence that one-frame-per-second reconstruction works, that users beyond the initial owner will upload recordings, or that visual coaching will improve the current review.

The principle extends beyond game recordings. Support calls, factory footage, security video, long documents, and operational logs all tempt an AI product to either process everything equally or summarize too aggressively. A better system can let a cheap pass find structure, but it must preserve the original evidence, expose uncertainty, and spend detail where another look can still change the decision.