Load both lora and ip-adapter in StableDiffusionPipeline shows error: 'UNet2DConditionModel' object has no attribute 'attn_processors' #7280

auiphc · 2024-03-11T11:24:11Z

auiphc
Mar 11, 2024

I want to load multiple lora and ip-adapter models to StableDiffusionPipeline. And I want to set lora weights and adapter weights each time I call api. Here is my code:

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_single_file(
    './models/NullStyle_V 2.0.safetensors',
    original_config_file='./models/v1-inference.yaml',
    torch_dtype=torch.float16,
    use_safetensors=True,
)
pipe.load_lora_weights('./models/lora', weight_name='otome_v0.7.safetensors')
pipe.load_ip_adapter('./models', subfolder='.', weight_name='ip-adapter-plus-face_sd15.safetensors')

pipe.set_ip_adapter_scale(0.5)

However, method set_ip_adapter_scale throws an error:

Traceback (most recent call last):
  File "C:\Users\50728\Documents\code\stable-diffusion-service\test.py", line 18, in <module>
    pipe.set_ip_adapter_scale(0.5)
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\diffusers\loaders\ip_adapter.py", line 187, in set_ip_adapter_scale
    for attn_processor in unet.attn_processors.values():
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\diffusers\models\modeling_utils.py", line 220, in __getattr__
    return super().__getattr__(name)
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'UNet2DConditionModel' object has no attribute 'attn_processors'. Did you mean: 'set_attn_processor'?

I found the problem falls in method get_processor in class Attention in attention_processor.py (https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py) where it is trying to find class LoRAIPAdapterAttnProcessor2_0 where there is no such class.

Then I found one implementation of class LoRAIPAdapterAttnProcessor2_0 in (https://github.com/huggingface/diffusers/blob/main/examples/community/ip_adapter_face_id.py)
And I overloaded method get_processor with my custom function as below:

def get_processor(self, return_deprecated_lora: bool = False):
    r"""
    Get the attention processor in use.

    Args:
        return_deprecated_lora (`bool`, *optional*, defaults to `False`):
            Set to `True` to return the deprecated LoRA attention processor.

    Returns:
        "AttentionProcessor": The attention processor in use.
    """
    if not return_deprecated_lora:
        return self.processor

    # TODO(Sayak, Patrick). The rest of the function is needed to ensure backwards compatible
    # serialization format for LoRA Attention Processors. It should be deleted once the integration
    # with PEFT is completed.
    is_lora_activated = {
        name: module.lora_layer is not None
        for name, module in self.named_modules()
        if hasattr(module, "lora_layer")
    }

    # 1. if no layer has a LoRA activated we can return the processor as usual
    if not any(is_lora_activated.values()):
        return self.processor

    # If doesn't apply LoRA do `add_k_proj` or `add_v_proj`
    is_lora_activated.pop("add_k_proj", None)
    is_lora_activated.pop("add_v_proj", None)
    # 2. else it is not posssible that only some layers have LoRA activated
    if not all(is_lora_activated.values()):
        raise ValueError(
            f"Make sure that either all layers or no layers have LoRA activated, but have {is_lora_activated}"
        )

    # 3. And we need to merge the current LoRA layers into the corresponding LoRA attention processor
    non_lora_processor_cls_name = self.processor.__class__.__name__
    lora_processor_cls = getattr(import_module(__name__), "LoRA" + non_lora_processor_cls_name)

    hidden_size = self.inner_dim

    # now create a LoRA attention processor from the LoRA layers
    if lora_processor_cls in [LoRAAttnProcessor, LoRAAttnProcessor2_0, LoRAXFormersAttnProcessor]:
        kwargs = {
            "cross_attention_dim": self.cross_attention_dim,
            "rank": self.to_q.lora_layer.rank,
            "network_alpha": self.to_q.lora_layer.network_alpha,
            "q_rank": self.to_q.lora_layer.rank,
            "q_hidden_size": self.to_q.lora_layer.out_features,
            "k_rank": self.to_k.lora_layer.rank,
            "k_hidden_size": self.to_k.lora_layer.out_features,
            "v_rank": self.to_v.lora_layer.rank,
            "v_hidden_size": self.to_v.lora_layer.out_features,
            "out_rank": self.to_out[0].lora_layer.rank,
            "out_hidden_size": self.to_out[0].lora_layer.out_features,
        }

        if hasattr(self.processor, "attention_op"):
            kwargs["attention_op"] = self.processor.attention_op

        lora_processor = lora_processor_cls(hidden_size, **kwargs)
        lora_processor.to_q_lora.load_state_dict(self.to_q.lora_layer.state_dict())
        lora_processor.to_k_lora.load_state_dict(self.to_k.lora_layer.state_dict())
        lora_processor.to_v_lora.load_state_dict(self.to_v.lora_layer.state_dict())
        lora_processor.to_out_lora.load_state_dict(self.to_out[0].lora_layer.state_dict())
    elif lora_processor_cls == LoRAAttnAddedKVProcessor:
        lora_processor = lora_processor_cls(
            hidden_size,
            cross_attention_dim=self.add_k_proj.weight.shape[0],
            rank=self.to_q.lora_layer.rank,
            network_alpha=self.to_q.lora_layer.network_alpha,
        )
        lora_processor.to_q_lora.load_state_dict(self.to_q.lora_layer.state_dict())
        lora_processor.to_k_lora.load_state_dict(self.to_k.lora_layer.state_dict())
        lora_processor.to_v_lora.load_state_dict(self.to_v.lora_layer.state_dict())
        lora_processor.to_out_lora.load_state_dict(self.to_out[0].lora_layer.state_dict())

        # only save if used
        if self.add_k_proj.lora_layer is not None:
            lora_processor.add_k_proj_lora.load_state_dict(self.add_k_proj.lora_layer.state_dict())
            lora_processor.add_v_proj_lora.load_state_dict(self.add_v_proj.lora_layer.state_dict())
        else:
            lora_processor.add_k_proj_lora = None
            lora_processor.add_v_proj_lora = None
########################################## My code ################################################
    elif lora_processor_cls in [LoRAIPAdapterAttnProcessor, LoRAIPAdapterAttnProcessor2_0]:
        lora_processor = lora_processor_cls(
            hidden_size=hidden_size,
            cross_attention_dim=self.cross_attention_dim,
            rank=self.to_q.lora_layer.rank,
            network_alpha=self.to_q.lora_layer.network_alpha,
        )
        lora_processor.to_q_lora.load_state_dict(self.to_q.lora_layer.state_dict())
        lora_processor.to_k_lora.load_state_dict(self.to_k.lora_layer.state_dict())
        lora_processor.to_v_lora.load_state_dict(self.to_v.lora_layer.state_dict())
        lora_processor.to_out_lora.load_state_dict(self.to_out[0].lora_layer.state_dict())

        state_dict = ?
        lora_processor.to_k_ip.load_state_dict(state_dict)
        lora_processor.to_v_ip.load_state_dict(state_dict)
########################################## My code ################################################
    else:
        raise ValueError(f"{lora_processor_cls} does not exist.")

    return lora_processor

Since I'm new on this and I have no idea which state_dict should I pass to to_k_ip.load_state_dic and to_v_ip.load_state_dict.
Can anyone help me or is there any other way to load both lora and ip-adpater to StableDiffusionPipeline?
Thanks

Answered by fabiorigano

Mar 12, 2024

I see, upgrading to the current diffusers will solve the issue
please run: pip install git+https://github.com/huggingface/diffusers

View full answer

sayakpaul · 2024-03-12T03:37:10Z

sayakpaul
Mar 12, 2024
Maintainer

What version of diffusers are you using?

2 replies

auiphc Mar 12, 2024
Author

Name: diffusers
Version: 0.26.3
Summary: State-of-the-art diffusion in PyTorch and JAX.

sayakpaul Mar 12, 2024
Maintainer

Cc: @fabiorigano

fabiorigano · 2024-03-12T07:12:38Z

fabiorigano
Mar 12, 2024

hi all, thanks Sayak for adding me here

I don't think LoRAIPAdapterAttnProcessor is actually needed, you are able to successfully load both LoRA and IP Adapter weights and this class is not part of the core.

Unfortunately, I am not able to reproduce your problem as I don't have your checkpoints.
Can you share them somewhere or provide the link to the original repository?

Here you have a working code snippet :

from diffusers import StableDiffusionPipeline

base_model_path ="runwayml/stable-diffusion-v1-5/v1-5-pruned.safetensors"

pipeline = StableDiffusionPipeline.from_single_file(
    base_model_path,
    torch_dtype=torch.float16,
    use_safetensors=True
)

pipeline.to("cuda")

pipeline.load_lora_weights("sayakpaul/sd-model-finetuned-lora-t4", weight_name="pytorch_lora_weights.bin")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-plus-face_sd15.safetensors")
pipeline.set_ip_adapter_scale(0.7)

4 replies

auiphc Mar 12, 2024
Author

I run this code and still get the same error

(diffusers) PS C:\Users\50728\Documents\code\stable-diffusion-service> cat .\test.py
from diffusers import StableDiffusionPipeline
import torch

base_model_path ="runwayml/stable-diffusion-v1-5/v1-5-pruned.safetensors"

pipeline = StableDiffusionPipeline.from_single_file(
    base_model_path,
    torch_dtype=torch.float16,
    use_safetensors=True
)

pipeline.to("cuda")

pipeline.load_lora_weights("sayakpaul/sd-model-finetuned-lora-t4", weight_name="pytorch_lora_weights.bin")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-plus-face_sd15.safetensors")
pipeline.set_ip_adapter_scale(0.7)

(diffusers) PS C:\Users\50728\Documents\code\stable-diffusion-service> python .\test.py
v1-5-pruned.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.70G/7.70G [06:57<00:00, 18.5MB/s]
C:\Users\50728\.conda\envs\diffusers\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\50728\.cache\huggingface\hub\models--runwayml--stable-diffusion-v1-5. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.52k/4.52k [00:00<?, ?B/s]
C:\Users\50728\.conda\envs\diffusers\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\50728\.cache\huggingface\hub\models--openai--clip-vit-large-patch14. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 905/905 [00:00<00:00, 898kB/s]
vocab.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 961k/961k [00:00<00:00, 2.54MB/s]
merges.txt: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 21.9MB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 389/389 [00:00<?, ?B/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.22M/2.22M [00:01<00:00, 2.09MB/s]
In this conversion only the non-EMA weights are extracted. If you want to instead extract the EMA weights (usually better for inference), please make sure to add the `--extract_ema` flag.
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
pytorch_lora_weights.bin: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.29M/3.29M [00:01<00:00, 2.21MB/s]
C:\Users\50728\.conda\envs\diffusers\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\50728\.cache\huggingface\hub\models--sayakpaul--sd-model-finetuned-lora-t4. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
You have saved the LoRA weights using the old format. To convert the old LoRA weights to the new format, you can first load them in a dictionary and then create a new dictionary like the following: `new_state_dict = {f'unet.{module_name}': params for module_name, params in old_state_dict.items()}`.
ip-adapter-plus-face_sd15.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 98.2M/98.2M [00:03<00:00, 29.3MB/s]
C:\Users\50728\.conda\envs\diffusers\lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\50728\.cache\huggingface\hub\models--h94--IP-Adapter. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
  warnings.warn(message)
models/image_encoder/config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 560/560 [00:00<?, ?B/s]
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.53G/2.53G [05:05<00:00, 8.29MB/s]
Traceback (most recent call last):
  File "C:\Users\50728\Documents\code\stable-diffusion-service\test.py", line 16, in <module>
    pipeline.set_ip_adapter_scale(0.7)
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\diffusers\loaders\ip_adapter.py", line 187, in set_ip_adapter_scale
    for attn_processor in unet.attn_processors.values():
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\diffusers\models\modeling_utils.py", line 220, in __getattr__
    return super().__getattr__(name)
  File "C:\Users\50728\.conda\envs\diffusers\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'UNet2DConditionModel' object has no attribute 'attn_processors'. Did you mean: 'set_attn_processor'?

My environment is

(diffusers) PS C:\Users\50728\Documents\code\stable-diffusion-service> pip show diffusers                                                              
Name: diffusers
Version: 0.26.3
Summary: State-of-the-art diffusion in PyTorch and JAX.
Home-page: https://github.com/huggingface/diffusers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/diffusers/graphs/contributors)
Author-email: [email protected]
License: Apache 2.0 License
Location: c:\users\50728\.conda\envs\diffusers\lib\site-packages
Requires: filelock, huggingface-hub, importlib-metadata, numpy, Pillow, regex, requests, safetensors
Required-by:
(diffusers) PS C:\Users\50728\Documents\code\stable-diffusion-service> pip show torch
Name: torch
Version: 2.1.2+cu121
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: [email protected]
License: BSD-3
Location: c:\users\50728\.conda\envs\diffusers\lib\site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: accelerate, basicsr, facexlib, gfpgan, realesrgan, torchaudio, torchvision, triton, xformers

Is there any problem with my environment?

auiphc Mar 12, 2024
Author

I'm running on windows with conda env

fabiorigano Mar 12, 2024

I see, upgrading to the current diffusers will solve the issue
please run: pip install git+https://github.com/huggingface/diffusers

Answer selected by auiphc

auiphc Mar 12, 2024
Author

This solves my problem, Thx!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Load both lora and ip-adapter in StableDiffusionPipeline shows error: 'UNet2DConditionModel' object has no attribute 'attn_processors' #7280

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 6 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Load both lora and ip-adapter in StableDiffusionPipeline shows error: 'UNet2DConditionModel' object has no attribute 'attn_processors' #7280

Uh oh!

auiphc Mar 11, 2024

Replies: 2 comments · 6 replies

Uh oh!

sayakpaul Mar 12, 2024 Maintainer

Uh oh!

auiphc Mar 12, 2024 Author

Uh oh!

sayakpaul Mar 12, 2024 Maintainer

Uh oh!

fabiorigano Mar 12, 2024

Uh oh!

auiphc Mar 12, 2024 Author

Uh oh!

auiphc Mar 12, 2024 Author

Uh oh!

fabiorigano Mar 12, 2024

Uh oh!

auiphc Mar 12, 2024 Author

auiphc
Mar 11, 2024

Replies: 2 comments 6 replies

sayakpaul
Mar 12, 2024
Maintainer

auiphc Mar 12, 2024
Author

sayakpaul Mar 12, 2024
Maintainer

fabiorigano
Mar 12, 2024

auiphc Mar 12, 2024
Author

auiphc Mar 12, 2024
Author

auiphc Mar 12, 2024
Author