DreamLoop: Controllable Cinemagraph Generation from a Single Photograph

Aniruddha Mahapatra, Long Mai, Cusuh Ham, Feng Liu

ADOBE RESEARCH

Controllable General Domain Cinemagraphs - Our Results More Results

Cinemagraph Creation from Single Photograph with Motion Control

DreamLoop enables users to generates cinemagraphs from a single photograph with intuitive and easy-to-use, but precise motion control.

Prompt: A close-up shot of a person holding a DSLR camera. The person is trying to adjust the lens of the camera by rotating it. Everything else remains static in place. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Prompt: A cat wiggling its tail while sitting still on a window counter top. The cat does not move its body and remains still in one place. The camera is completely static. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

A bird is sitting on a branch, with faded flowers in the background. The branch and the bird are oscillating up and down. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt: A bird is sitting on a branch, with faded flowers in the background. The branch and the bird are oscillating up and down. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt: A close-up shot of a table with a burger and a girl holding her phone. She remains in the same posture, static, only moving her thumb to swipe on her phone. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

A pendulum is oscillating in a car hanging from the rear-view mirror with a thread. Everything else in the scene is completely static. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt: A pendulum is oscillating in a car hanging from the rear-view mirror with a thread. Everything else in the scene is completely static. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt: A girl is sitting in a cafe, stirring her coffee in a glass. She remains stationary in her place and does not move. The camera is also stationary. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end."

Controllable General Domain Cinemagraphs - Comparisons More Results

Comparison of Our method with Baselines for Controllable General Domain Cinemagraphs

We compare our method (both with internal model and Wan2.2-5B) with baselines which include: I) Image-to-Video Models - CogVideoX and Wan2.2-5B (pretrained), II) Motion Controllable Image-to-Video Models - DragAnything, Motion-I2V, Time-to-Move, MotionPro, and Diffusion as Shader, III) Our method with only first-last-frame conditioning, and Generative Inbetweening. For I) and II), we use post-processing to make the videos looping.

Prompt: A makeup artist is applying makeup with a brush on the face of a female celebrity. The scene is static, the makeup artist and celebrity do not move, they remain still in their place. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Input

CogVideoX

Wan2.2-5B (Pretrained)

Generative Inbetweening

Motion-I2V

Time-to-Move

MotionPro

Diffusion as Shader

DragAnything

Ours (Wan2.2-5B) [first-last]

Ours (internal) [first-last]

Ours (Wan2.2-5B)

Ours (internal)

Comparison with Commercial Models

Comparison with Commercial Video Generation Models

None of the commercially available models can generate controllable cinemagraphs. Veo 3.1 and Luma Ray 3 are conditioned on first and last frames to encourage looping, yet they cannot follow precise motion control. Runway Gen 4.5 is prompted for looping motion without first-last frame conditioning. DreamLoop is the only method that produces temporally consistent, loop-closed cinemagraphs with user-defined motion control.

Input

Veo 3.1 (first-last)

Luma Ray 3 (first-last)

Runway Gen 4.5

Ours

Input

Veo 3.1 (first-last)

Luma Ray 3 (first-last)

Runway Gen 4.5

Ours

Timing Control for Cinemagraphs More Results

Importance of Timing Control over Object Motion

DreamLoop provides additional control over the timing of object motion in cinemagraphs, precisely letting users define the duration of the motion for each action.

Prompt: “A metallic ball is oscillating in a car hanging from the rear-view mirror with a thread. Everything else in the scene is completely static. The shot is taken with a static camera..”

Explanation: The metallic ball is oscillating in a Simple Harmonic Motion (SHM). The speed of motion near the equilibrium position is faster than the speed of motion near the extreme positions. We can achieve this effect by controlling the timing of the motion.

Reference frame for timed motion control

w/o Timing Control

w/ Timing Control

Full vs Partial Control over Object Motion More Results

Full vs Partial Control over Object Motion

DreamLoop also works in cases where user does to provide the full motion path for all objects in the scene, but just the intial motion hint. The rest is inferred by the model.

Prompt: A scene of two empty swings oscillating due to the winds blowing. The background is covered in snow, and it is a morning scene. The shot is with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Full Input Reference frame for partial control

Generated (Full)

Partial Input Reference frame for partial control

Generated (Partial)

Mask-Based Post-Processing

Correcting Residual Motion with User-Drawn Masks

Even though we provide static point tracks for regions with no user-guided motion, certain generations can still have in-place small motion, like eyes blinking. Users can correct this by drawing a rough mask around the regions they want to additionally keep static. We then perform blending with the input image and generated cinemagraph as a post-processing step, suppressing any unintended motion within the masked region.

Input Image Input image for mask editing example 1

Generated

User Mask

Stabilized

Input Image Input image for mask editing example 2

Generated

User Mask

Stabilized

Limitations of DreamLoop

Despite the capabilities of DreamLoop and its ability to generate cinemagraphs from a single photograph with intuitive and easy-to-use, but precise motion control, it still has some limitations.

Limitation: The hand of the girl gets deformed in the video. This can happen because of limitations of the base model involving human motion.

Limitation: Only part of the petal moves for which motion path was defined. But the results is not as expected intuitively.

Limitation: The buttons on the blazer do not move in the video with the hand. This is likely the limitation of the base model involving object motion.

Teaser

Teaser Figure

Video results for the teaser figure in the main paper (1835).

Ptompt (top): A girl holding a rabbit in her lap. Both are rabbit and the girl are completely stationary. Only the earrnig of the girl is oscillating very slowly. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Prompt (bottom): A cute rabbit is sitting in the lap of a girl and slowly rotating its head while nibbling. The girl is completely stationary. The shot is taken with a static camera. Camera remains stationary. Static Camera. Camera locked down at the same place. The video loops back to the beginning of the video after it reaches the end.

Reference frame for timing control scene 1

Quantitative Comparison - General Domain Cinemagraphs

Performance on VBench

DreamLoop outperforms baselines on custom domain cinemagraphs on controllable generation for general domain. across all VBench metrics.

Method	Aesthetic Quality ↑	Imaging Quality ↑	Temporal Flickering ↑	Motion Smoothness ↑	Background Consistency ↑	Subject Consistency ↑
CogVideoX	0.5353	0.6419	0.9888	0.9904	0.9741	0.9787
Wan2.2-5B (pretrained)	0.5442	0.6018	0.9881	0.9922	0.9598	0.9605
DragAnything	0.4986	0.5535	0.9674	0.9789	0.9613	0.9474
Motion-I2V	0.4979	0.5510	0.9658	0.9775	0.9598	0.9459
Time-to-Move	0.5050	0.6005	0.9720	0.9795	0.9650	0.9550
MotionPro	0.5320	0.6030	0.9885	0.9720	0.9705	0.9625
Diffusion as Shader	0.5085	0.5980	0.9725	0.9845	0.9665	0.9520
Generative Inbetweening	0.5650	0.6205	0.9940	0.9911	0.9688	0.9785
Ours (Wan2.2-5B) [only first-last]	0.5897	0.6756	0.9950	0.9959	0.9746	0.9941
Ours (internal) [only first-last]	0.5872	0.6615	0.9899	0.9929	0.9685	0.9566
Ours (Wan2.2-5B)	0.5998	0.6762	0.9955	0.9964	0.9800	0.9868
Ours (internal)	0.5997	0.6771	0.9965	0.9959	0.9849	0.9858

User Study - General Domain Cinemagraphs

Human Preference Study

User Study with Amazon Mechanical Turk involving 50 participants confirms DreamLoop achieves higher user preference for controllable cinemagraph generation compared with contemporary open-source methods. Each comparison is done by 3 participants.