Context Learning
Agentic, fine-grained evaluation of physical and semantic consistency in generated videos
Overview
Generated videos can look fluent while still violating the objects, events, spatial relations, timing, or physical dynamics requested by a prompt. Most evaluators compress these failures into a single score, which cannot explain what failed, when it failed, or what evidence supports the decision.
Context Learning presents an auditable, physically grounded video-evaluation agent. Rather than asking a learned judge for the final label, it coordinates semantic verification with specialist tools that return measurements; deterministic rules then produce supported, contradicted, or unknown decisions together with localized evidence and provenance.
- 01 · Plan Translate exact prompt spans into typed, testable obligations.
- 02 · Measure Run the relevant video checks—such as counting, tracking, depth, OCR, or audio—over grounded time windows.
- 03 · Adjudicate Combine the measurements with fixed rules into an evidence-backed verdict.
The report is anchored by a corpus of 1,500 generated clips with 2,582 human-written flaw records. On the current 149-clip development core, plan-based execution recovered 197 of 286 localized flaws, compared with 179 for isolated per-claim verification, at roughly one additional call per clip. These are non-held-out development results, not a final generalization claim.
Examples
Four jets, not three
Prompt: Three F-22 jets fly in formation under the Golden Gate Bridge.
Four fighter jets appear, not the requested three.
Four fighter jets are visible, not three.
Evidence & decision trace
-
Prompt claims
Count: exactly three jets. Spatial relation: the jets fly beneath the bridge. Scene action: tourists gather at the railing.
-
Observed video evidence
Four distinct jets are visible in formation. Their trajectories stay above the bridge structure, and the bridge deck contains vehicles but no visible tourists.
-
First-pass verdict
Pass — incorrect. The initial judge described “three fighter jets flying in tight formation under a bridge,” accepting the prompt claim without a reliable recount or spatial check.
-
Reviewer correction
The reviewer explicitly recounts the visible jets and grounds their position against the bridge, rejecting the first-pass description as a false pass.
-
Final finding
Contradicted: four jets are visible, not three. Additional contradictions—flying above the bridge and missing tourists—are retained as separate findings.
-
Human-reference check
The final count finding matches the independent human annotation: four jets appear instead of the requested three.
Skin moves before contact
Prompt: A ring catches at the knuckle, pushes the skin, then pops into place.
The skin moves before the ring touches it, so the event order is physically inconsistent.
The ring never contacts the skin or pushes a roll of skin forward.
Evidence & decision trace
-
Prompt claims
The required sequence is: ring reaches the knuckle → physical contact pushes a roll of skin forward → ring pops into place.
-
Observed video evidence
The apparent skin deformation begins before the ring reaches the finger. No frame establishes ring-to-skin contact followed by the requested push.
-
First-pass verdict
Pass — incorrect. The initial judge treated visual proximity between the ring and finger as evidence that the ring was pushing the skin.
-
Reviewer correction
The reviewer separates proximity from contact and checks the temporal order. The required cause must occur before the deformation it produces.
-
Final finding
Contradicted: the ring never visibly contacts the skin or pushes a roll of skin forward in the required sequence.
-
Human-reference check
The final finding agrees with the human annotation that the skin moves before contact, making the event order physically inconsistent.
The dog already has the stick
Prompt: A golden retriever jumps into a lake after a thrown stick and retrieves it.
The dog enters the water with the stick already in its mouth instead of chasing it.
The dog already holds the stick rather than chasing a thrown one.
Evidence & decision trace
-
Prompt claims
The required causal sequence is: a stick is thrown → the dog launches into the water after it → the dog retrieves and emerges with it.
-
Observed video evidence
The dog already holds the stick in its mouth when it enters the water. The video never establishes a thrown stick or a chase that precedes possession.
-
First-pass verdict
Pass — incorrect. The initial judge saw a dog swimming with a stick and treated that endpoint as sufficient evidence for the full chase-and-retrieve sequence.
-
Reviewer correction
The reviewer distinguishes the final state—dog possesses stick—from the causal process requested by the prompt—dog pursues a previously thrown stick.
-
Final finding
Contradicted: the dog already has the stick rather than chasing and retrieving a thrown one.
-
Human-reference check
The final finding matches the human annotation that the dog enters the water with the stick already in its mouth.
The sun never rises
Prompt: The golden sun slowly rises behind tall mountains.
The sun remains stationary instead of slowly rising.
Tracking measures only a −2 px net shift over 192 frames, contradicting the required upward motion.
Evidence & decision trace
-
Prompt claim
The sun must move upward over time and visibly rise from behind the mountains.
-
Measurement setup
Context Learning tracks the sun's silhouette across 192 frames and measures vertical displacement relative to the object's own radius.
-
Tool evidence
The furthest measured displacement is only 0.71 sun radii, with a net end-to-end shift of −2 pixels. The measured direction does not support sustained upward travel.
-
Claim comparison
Changing mist and illumination can create an impression of sunrise, but the tracked object itself remains effectively stationary.
-
Final finding
Contradicted: the sun does not perform the required rising motion.
-
Human-reference check
The tool-grounded finding matches the human annotation that the sun remains stationary.
The annotation contradicts the prompt
Prompt: A fluffy Ragdoll cat sits on a windowsill and watches autumn leaves fall.
The video shows a Ragdoll rather than a LaPerm cat, so it does not match the instruction.
No corresponding defect: the prompt explicitly requests a Ragdoll cat.
Annotation audit
-
Prompt ground truth
The prompt explicitly asks for a Ragdoll cat. It never requests a LaPerm cat.
-
Human annotation
The annotation identifies the cat as a Ragdoll but incorrectly claims that being a Ragdoll violates a supposed LaPerm requirement.
-
Consistency check
The alleged defect is compared with the literal prompt. “Video contains a Ragdoll” is consistent with “Prompt requests a Ragdoll,” so there is no contradiction to report.
-
Context Learning outcome
No match — correct. Context Learning does not reproduce the invalid breed-mismatch claim.
-
Adjudication
The disagreement is caused by an erroneous human reference, not by a missed system finding.
-
Final status
Correct non-match: the human annotation should be corrected or excluded before computing system recall.
The technical report is in progress. The auditable critic and its evidence traces are implemented; closing the critic-to-generator refinement loop remains future work.