A live AI demo can be less honest than a replay

“Try it live” sounds like the strongest possible proof for an AI product. A visitor enters something, a model responds, and the product appears to work in real time.

But what did the interaction actually prove?

It may prove that an API answered once. It does not necessarily show which evidence the answer used, whether the important claims were checked, how a failure is handled, what the call costs, or what would happen if several visitors arrived at once. The apparent openness of a live demo can hide the system around the model.

I ran into this question while shaping LeagueLoop, a post-match coaching product. Its useful job is narrow: turn one completed match into one adjustment a player can apply next game. A model helps write the review, but deterministic code owns the match evidence, candidate selection, structured validation, and cost boundary.

Making the portfolio version call a model for every visitor would add spectacle. It would not automatically make those surrounding decisions easier to inspect.

What the replay preserves

The portfolio surface therefore uses a curated, zero-network report. It shows the accepted output, the evidence supporting it, the product flow, and the limits of the demonstration without exposing a paid endpoint or pretending to be a production coaching service.

1. 01Normalize match facts
2. 02Select candidate moments
3. 03Generate one bounded review
4. 04Validate factual claims
5. 05Present the accepted report

The replay is not presented as proof of live scale or universal coaching quality. It is proof of a specific product and architecture decision: generated advice is surrounded by deterministic evidence and an explicit human acceptance point.

Live and honest are different dimensions

A live product can be honest. A replay can be misleading. The point is not that recorded demonstrations are inherently better; it is that interactivity and evidence answer different questions.

For early portfolio work, I care more about whether a visitor can inspect the whole decision than whether they can make the system improvise. A live textbox often pulls attention toward the novelty of the response. A replay can direct attention toward the product question: what was the input, what did the system decide, what did it reject, and why should anyone trust the result?

There are limits. A replay cannot establish that the system generalizes. It cannot reveal true production latency, concurrency, provider incidents, or the quality distribution across unfamiliar inputs. Those require controlled evaluation and, eventually, carefully operated live use.

That limitation should be visible rather than patched with ambitious language.

The boundary also protects the visitor from becoming an involuntary tester. If a public prototype accepts arbitrary input, someone has to decide what is stored, what is moderated, what may be sent to a provider, and how unsafe or malformed results are explained. Those responsibilities exist even when the interface looks like a harmless experiment.

The stronger sequence

My preferred sequence now is:

1. Build the deterministic product workflow.
2. Record a small set of representative successes and failures.
3. Publish a replay that makes the evidence inspectable.
4. Add live inference only when variability itself is part of what needs to be evaluated—and when cost, privacy, abuse, and recovery are real product requirements.

This is not an argument for avoiding real integrations. It is an argument for earning them. A live endpoint should exist because it answers a question the replay cannot, not because every AI product is expected to have an empty input box and a glowing submit button.

For LeagueLoop, the current replay says something modest but defensible: here is one complete coaching experience, here is the evidence path behind it, and here is where the demonstration stops. That boundary is less dramatic than unrestricted inference. It is also easier to trust.