AnimateDiff: Video Generation

Introduction

AnimateDiff is an animation module that integrates with Stable Diffusion to generate short videos (2-5 seconds). It produces coherent frame sequences with fluid motion.

Installation

Install "AnimateDiff Evolved" via ComfyUI Manager. Download motion modules to models/animatediff_models/.

Basic Workflow

  1. AnimateDiff Loader: loads motion module
  2. Set Empty Latent Image batch_size to 16 (= 16 frames)
  3. AnimateDiff Combine: assembles frames into GIF/MP4

Context Options

  • context_length: 16 (frames processed together)
  • context_overlap: 4 (overlap between windows)
  • closed_loop: true for GIF loops

Advanced Techniques

  • Motion LoRAs: predefined camera movements (ZoomIn, PanLeft, etc.)
  • ControlNet per frame: control movement with OpenPose sequences
  • AnimateDiff + IP-Adapter: animate a consistent character
  • Prompt Scheduling: change prompt across frames

Long Videos with Context Sliding

Process frames in sliding windows of 16 with overlap for longer videos (batch_size: 48+).

img2vid and vid2vid

  • img2vid: animate a still image (VAE Encode + Repeat Latent Batch + denoise < 1.0)
  • vid2vid: restyle existing video with ControlNet for temporal coherence

Optimization

  • Use --lowvram, reduce resolution for testing
  • 30-40 steps, cfg 7-9
  • Frame interpolation with RIFE for smoother results