The research group of Professor Jin Xiaogang at the State Key Laboratory of CAD&CG, Zhejiang University, in collaboration with Tencent Timi Studio Group, presented a paper titled “Unifying Precise Keyframes and Semantic Control via Multi level Diffusion” at CVPR 2026. This work addresses text driven human motion generation and interactive animation authoring by proposing a unified multi level diffusion framework that synergistically models textual semantics and spatiotemporal keyframe constraints. The method achieves significant improvements in high precision motion interpolation and semantic preserving editing tasks.
The core challenge tackled is how to reconcile semantic controllability with precise spatiotemporal constraints within a single generative framework. The proposed solution employs a multi level conditional guidance mechanism that decouples and then seamlessly integrates high level semantic information from text with local spatiotemporal constraints from keyframes, thereby producing high quality human motions that are both semantically faithful and strictly keyframe compliant.
Background
Text driven human motion generation has emerged as a key research direction in computer animation and virtual humans. Compared to traditional keyframe based animation, text offers a more natural and flexible semantic control, allowing users to describe actions using plain language. However, text itself lacks precise temporal and spatial expressiveness – it cannot easily specify exact moments, trajectory changes, or detailed local poses. As a result, generated motions often suffer from semantic drift or spatial instability in complex scenarios.
Keyframes, on the other hand, serve as a core control primitive in traditional animation, providing accurate pose and spatial constraints at specific timestamps, which significantly improves controllability and physical plausibility. Yet keyframes convey no semantic meaning; relying solely on them for generation often requires extensive manual annotation to describe a full action sequence, substantially raising production costs. Thus, a key challenge is to harness both the semantic power of text and the precision of keyframes in a unified model, enabling low cost, high accuracy motion generation and editing.
Existing approaches have attempted to incorporate keyframes or spatial constraints into diffusion models, but they typically face two major issues: (1) ineffective alignment between textual semantics and keyframe constraints, leading to redundant motions or temporal misalignment; and (2) keyframes are often treated as soft constraints, making it difficult to enforce strict spatial consistency during generation, which compromises physical realism and visual stability.
Innovations and Contributions
To overcome these limitations, this work introduces a multi level diffusion framework that embeds a dual control structure – global semantic guidance and local keyframe guidance – within the diffusion model, enabling unified modelling of textual semantics and spatiotemporal keyframe constraints. During generation, the global guidance fuses textual semantics with the overall structural information of the keyframe sequence and injects them via cross attention into the diffusion U Net backbone, modulating the overall action category and temporal structure. Meanwhile, the local guidance applies fine grained constraints around each keyframe, ensuring strict pose alignment at specified timestamps while preserving natural transitions and motion continuity in unconstrained regions.
To further enforce strict keyframe adherence, we incorporate a refinement strategy during inference that combines trajectory correction with diffusion based completion. This includes adaptive velocity based root node trajectory correction to eliminate global spatial drift, and a mask driven pose back filling mechanism for hard keyframe alignment. For interactive motion editing, we further design a semantic preserving editing strategy based on DDIM inversion: the original motion is mapped into the diffusion latent space, and then fixed point iteration with keyframe re sampling is applied to maximally retain the original semantic structure while accommodating new keyframe constraints. This supports high fidelity, controllable human motion generation and editing.
Our main contributions are:
A multi level diffusion generation framework that jointly models textual semantics and spatiotemporal keyframe constraints, achieving synergistic motion generation with both high level semantic control and fine grained constraint satisfaction, thereby improving controllability and consistency.
A global and local multi level conditional guidance mechanism, where global guidance fuses text semantics with keyframe structure to modulate overall motion dynamics, and local guidance enforces fine grained constraints around keyframes, ensuring both semantic coherence and spatial precision.
A training free semantic preserving motion editing method that leverages DDIM inversion and fixed point iteration to satisfy new keyframe constraints while preserving the original motion semantics to the greatest extent, enabling high fidelity interactive motion editing.
overview of the proposed framework
Results
We present qualitative and quantitative comparisons with state of the art methods, including OmniControl, CondMDI, and MaskControl.
Qualitative comparison on text conditioned motion interpolation tasks shows that existing methods frequently suffer from temporal misalignment, unstable poses, and keyframe deviations in multi stage actions (e.g., “run – decelerate – walk”) and object interaction scenarios. In contrast, our method strictly satisfies keyframe constraints while maintaining semantic consistency and natural transitions, demonstrating superior spatiotemporal control and higher generation quality in complex settings.
Quantitative evaluation on the HumanML3D dataset (table) shows that our method achieves the best overall performance across all metrics. It attains strict keyframe alignment, higher semantic consistency scores, and outperforms all baselines in motion quality and physical plausibility (Skating Ratio and Jitter). These results indicate that our approach preserves a natural motion distribution and stable temporal structure while enforcing high precision spatial constraints.
Acknowledgements
This work was co authored by Professor Jin Xiaogang (corresponding author) from the State Key Laboratory of CAD&CG, Zhejiang University, and Ph.D. candidate Wu Linjun as the first author. The research was supported by the Zhejiang Provincial Key R&D Program and the National Natural Science Foundation of China, and was conducted in collaboration with Tencent Timi Studio Group.
