Broken video output with Wan 2.1 I2V pipeline + quantized transformer

[rolux]

Describe the bug

Since there is no proper documentation yet, I'm not sure if there is a difference to other video pipelines that I'm unaware of – but with the code below, the video results are reproducibly broken.

There is a warning:
Expected types for image_encoder: (<class 'transformers.models.clip.modeling_clip.CLIPVisionModel'>,), got <class 'transformers.models.clip.modeling_clip.CLIPVisionModelWithProjection'>.
which I assume I'm expected to ignore.

Init image:

Result:

test.mp4

Result with different seed:

423258632.0.mp4

Result with different prompt:

423258632.0.mp4

Reproduction

# Tested on Google Colab with an A100 (40GB).
# Uses ~21 GB VRAM, takes ~150 sec per step, ~75 min in total.


!pip install git+https://github.com/huggingface/diffusers.git
!pip install -U bitsandbytes
!pip install ftfy


import os
import torch
from diffusers import (
    BitsAndBytesConfig,
    WanImageToVideoPipeline,
    WanTransformer3DModel
)
from diffusers.utils import export_to_video
from PIL import Image


model_id = "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers"
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)
transformer = WanTransformer3DModel.from_pretrained(
    model_id,
    subfolder="transformer",
    quantization_config=quantization_config
)
pipe = WanImageToVideoPipeline.from_pretrained(
    model_id,
    transformer=transformer
)
pipe.enable_model_cpu_offload()


def render(
    filename,
    image,
    prompt,
    seed=0,
    width=832,
    height=480,
    num_frames=81,
    num_inference_steps=30,
    guidance_scale=5.0,
    fps=16
):
    video = pipe(
        image=image,
        prompt=prompt,
        generator=torch.Generator(device=pipe.device).manual_seed(seed),
        width=width,
        height=height,
        num_frames=num_frames,
        num_inference_steps=num_inference_steps,
        guidance_scale=guidance_scale
    ).frames[0]
    os.makedirs(os.path.dirname(filename), exist_ok=True)
    export_to_video(video, filename, fps=fps)


render(
    filename="/content/test.mp4",
    image=Image.open("/content/test.png"),
    prompt="a woman in a yellow coat is dancing in the desert",
    seed=42
)

Logs

System Info

• 🤗 Diffusers version: 0.33.0.dev0
• Platform: Linux-6.1.85+-x86_64-with-glibc2.35
• Running on Google Colab?: Yes
• Python version: 3.11.11
• PyTorch version (GPU?): 2.5.1+cu124 (True)
• Flax version (CPU?/GPU?/TPU?): 0.10.4 (gpu)
• Jax version: 0.4.33
• JaxLib version: 0.4.33
• Huggingface_hub version: 0.28.1
• Transformers version: 4.48.3
• Accelerate version: 1.3.0
• PEFT version: 0.14.0
• Bitsandbytes version: 0.45.3
• Safetensors version: 0.5.3
• xFormers version: not installed
• Accelerator: NVIDIA A100-SXM4-40GB, 40960 MiB

Who can help?

No response

[rolux]

Replaced

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

with

quantization_config = BitsAndBytesConfig(load_in_8bit=True)

This uses ~34 GB of VRAM, takes ~84 sec per step (~42 min in total), but the output quality does not seem to improve.

8-bit.mp4

[a-r-r-o-w]

Thanks for the detailed post and reproducible code @rolux. Quantization on video models does not work at times, so it could very well be that. I'll run the same code with torch.bfloat16 first to verify that it's not a problem with our implementation. If not, quantization could probably be resulting in the poorer quality

[a-r-r-o-w]

Btw, if you have a decent amount of RAM and want to run the model in low VRAM without quantization, I would recommend you to try this: https://huggingface.co/docs/diffusers/main/en/optimization/memory#group-offloading. Combining this with a few other memory optimizations can help you run in 7-10 GB VRAM

[rolux]

@a-r-r-o-w: Thanks – yes, I had seen this earlier today.

Added a comment in #10999. TLDR: Didn't work for me, as of now.

[tin2tin]

@rolux Did you somehow figure out how to run wan 2.1 i2v locally with Diffusers?

[bghira]

using fp32 also doesn't help the issue unfortunately

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.