-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Idefics 3! #32473
Merged
+4,482
−2
Merged
Add Idefics 3! #32473
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit
Hold shift + click to select a range
842a28d
Add Idefics 3!
andimarafioti afce007
fixes to make both pipelines identical
andimarafioti 3e3b31d
fix for quantized models
andimarafioti 9c8ffc4
First pass at the review
andimarafioti 7e3d7a6
remove vocab size from the main config (it's still in the text_config)
andimarafioti dd99bca
hot fix for merve
andimarafioti ddac9ec
Apply suggestions from code review
andimarafioti 188bb76
re-add model_type for text_config
andimarafioti 43fb214
remove support for old_cache
andimarafioti c9e0d85
remove hidden_size from main config
andimarafioti 1b2b89c
rename idefics3 HF repo
andimarafioti 6ff766f
few changes suggested in the PR
andimarafioti 11c2e1a
fix to input_data_format computation
andimarafioti c1048ed
remove overwrite of _autoset_attn_implementation following @zucchini-…
andimarafioti a163564
improve example
andimarafioti 6f0a479
few improvements from amy's review
andimarafioti 8361fce
big change to enable processing input images as numpy arrays
andimarafioti 32970d0
Changes to the code to uniformize processor kwargs
andimarafioti c504f00
image processing tests
andimarafioti a914e41
image processing tests fixes and some bugs they discovered
andimarafioti 6722d13
addressed review comments from Yoni
andimarafioti 0533eda
fix modeling tests
andimarafioti b034091
remove special tokens that are not special
andimarafioti 47fb7ce
fixes tests
andimarafioti 4032a6f
skip failing tests - they also fail for idefics2
andimarafioti 757e834
added paper and readded the tests with multi gpu, who knows
andimarafioti 7797279
Update docs/source/en/model_doc/idefics3.md
andimarafioti b478124
Apply suggestions from code review
andimarafioti ada6219
review amy until image_processing_idefics3
andimarafioti 164fbe8
last comments from Amy
andimarafioti 000c8ea
review amy
andimarafioti 4d02e0c
Update src/transformers/models/idefics3/image_processing_idefics3.py
andimarafioti 3bf03c2
Update src/transformers/models/idefics3/modeling_idefics3.py
andimarafioti 57bfd51
Update docs/source/en/model_doc/idefics3.md
andimarafioti 63b1d7f
doc improvement - amy review
andimarafioti 6325fbc
fix runtime error during fine-tuning
andimarafioti 76b8892
amy's review
andimarafioti 9a20306
Update src/transformers/models/idefics3/image_processing_idefics3.py
andimarafioti 3129920
Update src/transformers/models/idefics3/image_processing_idefics3.py
andimarafioti e1a10b3
Update src/transformers/models/idefics3/modeling_idefics3.py
andimarafioti 4c3756f
ruff
andimarafioti fbaf07e
amy's comment on the order
andimarafioti 87fa179
ruff ruff
andimarafioti 23d4cf8
fix copies
andimarafioti 9e925b9
square images when they are not splitted
andimarafioti 215b636
ruff :(
andimarafioti 2967974
Update src/transformers/models/idefics3/image_processing_idefics3.py
andimarafioti ee041bf
Update tests/models/idefics3/test_processing_idefics3.py
andimarafioti 4aad266
fix small bug introduced in refactor
andimarafioti f1ae8ae
amy's image processing changes
andimarafioti 39d88b2
fixes peft tests and ruff
andimarafioti 383f0db
modify to_pil_image from transformers. and review from emanuele.
andimarafioti 682b82b
add modified to_pil_image
andimarafioti File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
modify to_pil_image from transformers. and review from emanuele.
commit 383f0dbca31b5da59c247bfa06a15fce6ab1d809
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As it is, this always assigns
self.config.text_config.initializer_range
while, from what I understand, it should assignself.config.initializer_range
in casehasattr(self.config, "initializer_range")
. Is it possible?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, you're right. This seems to also be a mistake on idefics2. Thanks!