-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
IP-Adapter for FluxImg2ImgPipeline
#10717
Conversation
While testing I also found that if you load the IP-Adapter, you must provide image prompt otherwise the pipeline fails. I will open a PR for that, but it's just a check for |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @guiyrt. I had opened #10708 to cover FluxImg2Img and others.
While testing I also found that if you load the IP-Adapter, you must provide image prompt otherwise the pipeline fails.
Is this the same behavior as other IP-Adapter? I believe it may be expected that we call unload_ip_adapter
if we no longer want to use ip_adapter_image
.
# handle image prompt | ||
any_ip_adapter_input = ip_adapter_image is not None or ip_adapter_image_embeds is not None | ||
any_negative_ip_adapter_input = ( | ||
negative_ip_adapter_image is not None or negative_ip_adapter_image_embeds is not None | ||
) | ||
|
||
if any_ip_adapter_input and not any_negative_ip_adapter_input: | ||
negative_ip_adapter_image = np.zeros((width, height, 3), dtype=np.uint8) | ||
elif any_negative_ip_adapter_input and not any_ip_adapter_input: | ||
ip_adapter_image = np.zeros((width, height, 3), dtype=np.uint8) | ||
|
||
image_embeds = ( | ||
self.prepare_ip_adapter_image_embeds( | ||
ip_adapter_image, | ||
ip_adapter_image_embeds, | ||
device, | ||
batch_size * num_images_per_prompt, | ||
) | ||
if any_ip_adapter_input | ||
else None | ||
) | ||
|
||
negative_image_embeds = ( | ||
self.prepare_ip_adapter_image_embeds( | ||
negative_ip_adapter_image, | ||
negative_ip_adapter_image_embeds, | ||
device, | ||
batch_size * num_images_per_prompt, | ||
) | ||
if any_negative_ip_adapter_input | ||
else None | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this changed, we can just copy from text-to-image pipeline, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does the same thing, I just created the variables any_ip_adapter_input
and any_negative_ip_adapter_input
for readability, as we have many is None
checks chained with bool logical ops.
|
||
|
||
class FluxIPAdapterImg2ImgPipelineSlowTests(FluxIPAdapterPipelineSlowTests): | ||
"""Same test as in FluxIPAdapterPipelineSlowTests, only with inital `image` and `strength` parameters.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need slow tests for IP-Adapter in each pipeline variant, the general functionality is tested with the fast test FluxIPAdapterTesterMixin
.
Oh, I started this yesterday and didn't notice 😢 bad timing, we can close this then
In SD3 you can load the IP-Adapter and not pass an image prompt, and it behaves like the unloaded SD3 pipeline. This can be useful when you don't control the pipeline args, such as in a gradio demo or just general model serving |
Thanks for still taking the time to review @hlky, is there anything else with priority open for the community? I can also check the roadmap, but if you have something in mind let me know :) |
Would you like to try We need to restore all the
I'll handle integrating diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py Line 1144 in 0914fa3
|
@hlky If I got it right, the idea is to make controlnet: Union[
ControlNetUnionModel,
List[Union[ControlNetModel, ControlNetUnionModel]],
Tuple[Union[ControlNetModel, ControlNetUnionModel]],
MultiControlNetModel,
] I noticed the PR you have open regarding the experiments with per control type scale, but from your previous message I didn't get if you already started with the integration of I didn't know about ControlNet union, very interesting work, thanks for pointing me towards this! :) |
Yes, so In this case we need a
controlnet: Union[
ControlNetUnionModel, List[ControlNetUnionModel], Tuple[ControlNetUnionModel], MultiControlNetUnionModel
], Essentially compare to |
Yeah, this was my main concern if we tried to just glue together Got it, much clearer now, thanks! |
What does this PR do?
Adds image prompting support for
FluxImg2ImgPipeline
viaFluxIPAdapterMixin
, as part of #10689.Example output:
![astro_grid](https://private-user-images.githubusercontent.com/35548192/409655293-b22451fa-d7c0-4e0e-aea1-e3ac7748a64d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkxNTkxNzksIm5iZiI6MTczOTE1ODg3OSwicGF0aCI6Ii8zNTU0ODE5Mi80MDk2NTUyOTMtYjIyNDUxZmEtZDdjMC00ZTBlLWFlYTEtZTNhYzc3NDhhNjRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjEwVDAzNDExOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY3OTgyNTYyOGRhZjY0OTc2ODVhOTlkZjUwYjhkY2E4M2VlZTliZGUwMThjM2Q5Yjc1OTc0NjU1MTFhZWVjMzAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.7kFXTGnLqI4YQ4KN4_burQTSKs-OV3ZP3GfE5vhHY4o)
I did some minor changes on
pipeline_flux.py
regarding documentation and argument order, for consistency.I also adapted a bit the ip_adapter code for clarity, hope you find it simpler this way :)
I'm not sure if you prefer it this way, but I made a change on
FluxIPAdapterPipelineSlowTests
so that the rest of the pipelines can use it simply by adding input arguments and changing expected output. Let me know if subclassing like this is ok, or if you prefer having the full class definition for each pipeline test.Inference code
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@hlky @yiyixuxu