AEbench: A Human Aligned Motion Amplitude Benchmark for Video Generation Evaluation
Project Status: Discontinued
- Following the rapid advancement of video generation models in late 2024–2026, we reassessed the urgency of this benchmark. State-of-the-art models have significantly narrowed the motion amplitude gap that originally motivated this work, reducing the practical demand for a dedicated evaluation framework in this direction. Development has been suspended accordingly.
Motivation
The rapid advancement of video generation models has outpaced the development of rigorous evaluation frameworks. In particular, motion amplitude — the degree to which generated videos exhibit dynamic, realistic movement — remains poorly captured by existing metrics.
The absence of an explicit motion amplitude signal introduces a systematic bias: benchmarks tend to assign high scores to slow-motion or near-static videos, effectively disincentivizing models from generating truly dynamic content. This misalignment between metric behavior and human expectation is the core problem AEbench aims to solve.
What's Wrong with Existing Approaches?
Prior work on motion evaluation falls into two categories, each with distinct failure modes:
| Approach | Examples | Problem |
|---|---|---|
| Optical flow (RAFT-based) | Traditional metrics | Highly sensitive to camera movement; noise inflates scores artificially |
| LLM + MLLM | Video-Bench, ChronoMagic-Bench | High variance — the same video can receive drastically different scores across runs |
| LLM + Traditional CV | VMBench | Focuses narrowly on detected object displacement; discards rich motion context |
AEbench is designed to address all three failure modes simultaneously.
Pipeline Overview
AEbench evaluates generated videos through a four-stage pipeline:
Stage 1 — Disentangle Pipeline
Given a generated video and its source prompt, two parallel disentanglement processes are run:
- Text Disentangler (LLM): Parses the prompt to extract each subject and generates per-object Motion Evaluation Criteria — precise, semantically grounded descriptions of expected motion behavior.
- Video Disentangler (CV): Slices the video into prompt-based and video-based segments using object detection (PAS), optical flow (RAFT), and saliency estimation (UNISAL).
Stage 2 — Dual-Perspective Evaluation Pipeline
The disentangled outputs are evaluated from two complementary perspectives in parallel:
- Feature-View (Feature Evaluator): Extracts quantitative motion features — segmentation features and basic optical flow statistics — grounded in the detected object regions.
- Semantic-View (Semantic Evaluator / MLLM): Uses the per-object Motion Evaluation Criteria from Stage 1 to produce rich natural-language descriptions of observed motion for each video segment.
Stage 3 — Hybrid Integration Pipeline
A Comprehensive Evaluator (LLM) integrates the feature table and semantic descriptions to produce multiple independent scored assessments, each with an explicit score and reasoning chain. A Final Evaluator then assesses the reliability of each scoring round and produces confidence weights.
Stage 4 — Final Scoring Pipeline
The final motion amplitude score is computed as a confidence-weighted mean across all evaluation rounds, combining quantitative rigor with semantic alignment.
Key Design Principles
- Human Alignment: Motion criteria are derived from the generation prompt itself, ensuring evaluation reflects human intent rather than proxy statistics.
- Dual-Perspective Robustness: Combining feature-level and semantic-level signals reduces the variance inherent in pure LLM/MLLM evaluation.
- Object-Centric Granularity: Per-object evaluation surfaces fine-grained motion quality issues rather than collapsing everything into a single global score.
- Uncertainty-Aware Scoring: Confidence weighting makes the final score robust to inconsistent reasoning in individual evaluation rounds.
System Implementation
The current implementation includes:
- LLM/MLLM module: 4 specialized agents (
SemanticDisentanglerAgent,SegmentationEvaluatorAgent,SummarizeEvaluatorAgent,FinalEvaluatorAgent) with multi-provider support (OpenAI, Anthropic, Google, HuggingFace, Ollama) - CV module: Wrappers for RAFT (optical flow), UNISAL (visual saliency), PAS (object detection), and EfficientLoFTR (feature matching)
- Status: LLM and CV modules complete; end-to-end pipeline integration in progress