[LoRA] fix peft state dict parsing

[sayakpaul]

What does this PR do?

Fixes #9542 (comment).

Fixes #10599.

https://civitai.com/models/200251?modelVersionId=1081295 has both peft and non peft state dict inside it 🤷

Testing code:

from diffusers import FluxPipeline
import torch

pipeline = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16
).to("cuda")
pipeline.load_lora_weights(
    "sayakpaul/different-lora-from-civitai", weight_name="sharp_detailed_foot.safetensors"
)

prompt = """
UHD, 4k, ultra detailed, cinematic, a photograph of <lora:sharp detailed image (foot focus) v1.1:0.9> sharp detailed image of a female person standing on a black mat with their bare feet in front of camera, perfect image, perfect body, perfect anatomy, sharp image, detailed image, cinematic style, high quality photography, sharp detailed image style, solo, barefoot, feet, toes, black background, toenails, close-up, foot focus, muscular, veins, soles, facing camera, feet in front of camera, polished nails, closeup, feet focus, epic, beautiful lighting, inpsiring
"""

_ = pipeline(prompt, num_inference_steps=20, generator=torch.manual_seed(2810038592))

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[RIOFornium]

LGTM

[sayakpaul]

@DN6 a gentle ping.

[sayakpaul]

Will first run the Flux tests and then merge the PR. I don't think the failing tests are related.

[sayakpaul]

Have run all the integration tests related to Flux LoRA and Flux Control LoRA and they pass. Merging.

[sayakpaul]

@RIOFornium, thanks for reporting the bugs and testing the PRs. Hope things are resolved now.

[RIOFornium]

@sayakpaul Thank you too!