How to Keep a Character Consistent Across AnimateDiff Scenes

November 20, 2026

You keep a character consistent across separate AnimateDiff renders by combining three things: a fixed seed, a character LoRA trained on 15-25 images, and a fixed reference frame used as the first frame of every clip. Any one alone drifts within two scenes.

Seed alone controls noise, not identity. The model still reinterprets the face each time it sees a new prompt. LoRA alone gives the same face but varies clothing, hair length and age with the phrasing. You need all three locked down.

Why does the same prompt produce a different face each render?

Because the text encoder maps \”a woman in a red jacket\” to a region of latent space, not a person. Small differences in sampler, window overlap and frame count move you to a different point in that region.

AnimateDiff makes this worse than still-image SD, because each 16-frame window is denoised with slightly different attention, and the motion module adds texture the still model never produced. Faces rendered in window three can differ from window one even inside a single clip.

That is the real reason multi-scene consistency is hard. You are not fighting the text encoder once; you are fighting it per window.

Step 1: Train a character LoRA and stop at 15-25 images

Collect 15-25 images of the character from different angles, with varied lighting and neutral expressions. More is not better past 30 — LoRAs over ~30 images start encoding the training background instead of the person.

Train at 512 or 768 resolution depending on your base model, 10-15 repeats, 8 epochs as a starting point, learning rate around 1e-4 with a cosine schedule. Inspect every epoch checkpoint at epoch 4, 6 and 8 on a fixed test prompt. The right checkpoint is almost never the last one — typically epoch 6 stops flexing and epoch 8 starts copying training photos too literally.

Name the trigger token something that is not an English word, like ohwx or chrfem. A real word gets diluted by the model’s existing associations.

Step 2: Lock the seed and the frame count

For a multi-scene project, pick one seed and use it for every clip. Write it down alongside the prompt, sampler, steps and CFG. If the seed works for scene one, changing it in scene three guarantees a different face.

Also freeze the frame count. A 48-frame clip and a 40-frame clip with the same seed and prompt do not produce the same character, because the window layout differs and the final window’s padding changes the last frames.

A versioned settings sheet is unglamorous and the single highest-value habit here:

  • seed: 4422179
  • sampler: DPM++ 2M Karras, 25 steps, CFG 7.5
  • context: 16/4
  • frames: 48
  • LoRA weight: 0.75

LoRA weight below 1.0 matters. At 1.0 the face is exact but the model loses flexibility and poses get stiff. Around 0.7 to 0.8 keeps the identity while letting the scene change.

Step 3: Use image-to-video for continuity, not text-to-video

For anything that must match a previous clip, generate the first frame as a still image, then run AnimateDiff from that frame. The first frame becomes ground truth and every window is conditioned on it to some degree.

This shifts the problem to producing a good still. Generate 20 candidates with the LoRA and seed locked, pick one face you like, and reuse that face image across all scenes by editing pose with img2img at low denoise (0.35-0.45). Low denoise preserves identity; above 0.6 you are redrawing the person.

What breaks consistency even when everything is locked?

Four things, in order of how often they bite:

  1. Checkpoint changes. Swapping the base model mid-project moves every face. Pick one checkpoint for the whole project.
  2. Prompt ordering. \”red jacket, woman\” and \”woman, red jacket\” weight tokens differently. Keep prompt structure identical across scenes and vary only the scene description.
  3. Negative prompt drift. A different negative prompt changes what the model suppresses. Copy negatives verbatim between scenes.
  4. Resolution changes. 512×512 and 512×768 produce different faces from the same seed. Render the whole project at one aspect ratio, crop in post.

What about VAE and upscaling?

Use a fixed VAE across the project. Mixing VAEs shifts color and, subtly, facial structure. If you upscale, do it after all clips are rendered with a consistent upscaler (4x-UltraSharp is a common pick) and keep tile sizes identical — tiled upscalers can introduce different seams on different frames.

Who this workflow suits

Fits individuals and small teams making short character-driven clips — music video inserts, product mascots, social spots — where a face must survive 5 to 10 shots. It also works for weekly series where the visual style is fixed.

It fits badly for dialogue-heavy scenes, close-ups with lip movement, or anything where a viewer inspects a face for more than two seconds. AnimateDiff’s face detail degrades in tight shots and no amount of LoRA fixes the motion module’s temporal limits.

A fair limitation: this is a per-project setup. Every new character means a new LoRA (1-2 hours of training plus testing), and the setting sheet does not transfer between characters cleanly. Plan for it in your schedule or you will ship inconsistent work under deadline.

Quick Summary

  • Consistency needs three locks: fixed seed, character LoRA at 0.7-0.8 weight, and a fixed first frame.
  • Train on 15-25 images; more than 30 starts carrying the training background.
  • The best LoRA checkpoint is usually epoch 6, not the last epoch.
  • Freeze frame count, sampler, CFG, negative prompt and aspect ratio across all scenes.
  • Use img2img at 0.35-0.45 denoise to pose the character without losing identity.
  • Not suitable for close-up dialogue shots — the motion module limits facial detail.