Video demos hide the costliest part of generative production: getting from a plausible clip to a usable shot. A tool can create one impressive frame and still fail when the subject turns, the camera moves or an edit must preserve continuity.
The fair comparison unit is a shot with explicit constraints, not a highlight reel.
Updated
The protocol is provider-neutral and avoids claims about a permanent winner. Model access, duration limits, audio support, pricing and safety controls change frequently.
Tested with
OpenAI video-generation resources · Sora 2 system card · C2PA 2.x
Choose a six-to-eight-second shot with one subject, one action and one camera instruction. For example:
A small service robot crosses a quiet workshop from left to right, stops at a blue workbench and places a red component on the surface. Locked medium-wide camera, soft morning light, no text.
The brief exposes useful failure modes: object identity, left-to-right motion, contact with the workbench, color preservation and unwanted camera drift.
Prepare a reference frame when the products support image conditioning. Use the same source file and document when a provider transforms it before generation.
Before running anything, set:
- target duration, resolution and aspect ratio;
- maximum number of generations;
- maximum number of edit or extension attempts;
- a delivery deadline;
- a policy for counting failed or moderated requests;
- the exchange rate or credit price used for cost calculations.
Report both paid cost and operator time. Cheap generations can become expensive when every acceptable clip requires manual repair.
Use three score groups.
- subject and environment match the brief;
- colors and proportions remain stable;
- frames do not contain accidental text or brand marks;
- the first and last frames are usable for an edit.
- the action follows the requested order;
- limbs, tools and surfaces interact coherently;
- identity survives occlusion and camera motion;
- motion does not jump, melt or reverse without cause.
- the tool exposes enough control to correct one failure;
- versions and source inputs remain traceable;
- exports match delivery requirements;
- content policy and provenance information are understandable;
- queue time and retries fit the deadline.
Key takeaway
Judge the minimum acceptable shot, not the maximum spectacle. Editorial and game-production pipelines need repeatability, clean handoffs and predictable repair more often than they need one exceptional sample.
The first pass measures generation from the frozen brief. The second measures recovery from one common failure.
- Submit the same shot brief and reference frame to every tool.
- Keep all attempts and stop at the predetermined generation limit.
- Select the highest-scoring candidate with reviewer names hidden from the provider label.
- Issue one equivalent repair request, such as preserving the robot while fixing the object placement.
- Export the best deliverable clip without external compositing.
- Record elapsed time, billed units, rejected requests and operator interventions.
- Repeat the scoring after randomizing filenames, then reconcile reviewer disagreements.
Do not quietly use a different production feature for one tool. If one provider offers a storyboard, keyframe controls or a region editor, include that capability in the workflow result and say so. The product is part of what you are evaluating.
A single score cannot explain a temporal error. Add timestamps:
00:00–00:02 subject and direction correct
00:03 right wheel changes shape behind the table leg
00:04 red component becomes orange during contact
00:06 camera begins an unrequested push-inThis log turns “the motion felt wrong” into evidence. It also shows whether the repair feature corrected the original defect or merely generated a different shot.
The conclusion should name the shot, date, tool version where visible, number of attempts and review rubric. It should not generalize from one stylized workshop clip to dialogue scenes, photorealistic humans or long sequences.
Also disclose what the test does not measure: audio synchronization, character reuse across scenes, legal rights to training material, regional availability and performance under load may each require a separate evaluation.
Warning
Never use an evaluation clip to imply that a real person performed an action they did not perform. For publishable work, document consent, disclosure and provenance separately from visual quality.
- OpenAI video-generation resources — current first-party guides and examples for video generation.
- Sora 2 system card — capability, evaluation and safety context for a current video model.
- C2PA specifications — technical standards for recording and validating media provenance.
- OpenAI evaluation best practices — guidance on explicit objectives, datasets, metrics and continuous evaluation.
