Add SmolVLM2 support #919

MilkClouds · 2025-10-25T16:24:53Z

Summary

This PR adds support for SmolVLM2 to liger-kernel.
SmolVLM2 is present on transformers after 4.50.0 version: https://github.com/huggingface/transformers/releases/tag/v4.50.0

Testing Done

Hardware Type: A100
run make test to ensure correctness


(4480 durations < 0.005s hidden.  Use -vv to show these durations.)
=============================================================================================== 2142 passed, 215 skipped, 45 warnings in 1941.33s (0:32:21) ===============================================================================================

run make checkstyle to ensure code style
run make test-convergence to ensure convergence

(35 durations < 0.005s hidden.  Use -vv to show these durations.)
================================================================================================================= short test summary info =================================================================================================================
FAILED test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_glm4v_moe-32-1e-05-dtype17-0.01-0.01-0.1-0.01-0.01-0.01] - AssertionError: [Loss]Number of mismatched elements: 2
=================================================================================================== 1 failed, 22 passed, 1 warning in 262.08s (0:04:22) ===================================================================================================
make: *** [Makefile:36: test-convergence] Error 1

There's no test failure for smolvlm2!

…cation - Introduced a new tokenizer configuration file for SmolVLM2-256M-Video-Instruct. - Added a function to check the availability of the SmolVLM model. - Implemented tests to ensure proper application of Liger kernels to SmolVLM2 model instances. - Added a utility function to revert Liger kernel patches for SmolVLM2.

Tcc0403 · 2025-10-26T09:00:38Z

src/liger_kernel/transformers/monkey_patch.py

+        logger.info("Apply liger cross entropy")
+
+        from transformers.loss.loss_utils import nn
+
+        nn.functional.cross_entropy = liger_cross_entropy


unintended changes in internvl?

Sorry for the confusion, I'm author of PR adding InternVL3(#878) to liger-kernel and this change is intended. I checked that on the transformers version when InternVL3 and SmolVLM2 implemented they aren't using CrossEntropyLoss inside modeling_*.py instead using loss_utils. So I fixed the InternVL3 implementation togather.

If you hope I'll separate this into another PR

Let's separate it into another PR and open an issue to track if there's any other cross entropy not being patched correctly!

src/liger_kernel/transformers/monkey_patch.py

src/liger_kernel/transformers/model/smolvlm.py

Tcc0403 · 2025-10-26T09:32:15Z

src/liger_kernel/transformers/monkey_patch.py

+        if vision_liger_fn:
+            accept_params = inspect.signature(vision_liger_fn).parameters
+            remain_params = set(kwargs) - (set(accept_params) & set(kwargs))
+            vision_kwargs = {k: v for k, v in kwargs.items() if k not in remain_params}
+
+            if remain_params:
+                logger.warning(
+                    f"These parameters are not supported by {vision_model_name}. Enter the remaining {list(vision_kwargs.keys())} except for {list(remain_params)}\n"
+                    f"Parameters accepted by {vision_model_name}: {list(accept_params.keys())}"
+                )
+            vision_kwargs["model"] = model.model.vision_model
+            vision_liger_fn(**vision_kwargs)
+        elif vision_model_name not in MODEL_TYPE_TO_APPLY_LIGER_FN:
+            logger.warning(f"{vision_model_name} is not supported by Liger kernel.")


The default SmolVLMForConditionalGeneration.vision_model is SmolVLMVisionTransformer

WARNING liger_kernel.transformers.monkey_patch:monkey_patch.py:2194 smolvlm_vision is not supported by Liger kernel.

Do you want to support it in this PR as well? (SmolVLMVisionMLP and LayerNorm)

SmolVLMVisionMLP seems not easy to patch kernel but LayerNorm seems possible, so I patched them!

test/transformers/test_monkey_patch.py

Co-authored-by: Tcc0403 <[email protected]>

… InternVL models

…rnvl function

MilkClouds · 2025-10-26T12:54:57Z

Ready for review again, @Tcc0403

I've checked following tests pass:

$ pytest test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal -k internvl test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal -k internvl test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_internvl -v
$ pytest test/convergence/bf16/test_mini_models_multimodal.py::test_mini_model_multimodal -k smolvlm2 test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal -k smolvlm2 test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_smolvlm2 -v

There are some modifications also to InternVL, I can (1) modify this PR's title and description to include InternVL's changes or (2) seperate InternVL to separate PR. Please inform me you prefer.

Tcc0403

Thank you! Seperate internvl change into another PR then we're good to merge!

…branch

MilkClouds · 2025-10-27T07:07:07Z

@Tcc0403 I've separated InternVL things to #921. Please have a look

Tcc0403 · 2025-10-27T14:48:56Z

Thank you!

MilkClouds added 3 commits October 25, 2025 15:32

feat(smolvlm): add support for SmolVLM model and apply Liger kernels

4a3cb15

style: run make checkstyle

c7a63be

Tcc0403 reviewed Oct 26, 2025

View reviewed changes

MilkClouds and others added 8 commits October 26, 2025 19:04

Update src/liger_kernel/transformers/model/smolvlm.py

4cb7389

Co-authored-by: Tcc0403 <[email protected]>

Update src/liger_kernel/transformers/monkey_patch.py

8e7f0c0

Co-authored-by: Tcc0403 <[email protected]>

Update src/liger_kernel/transformers/model/smolvlm.py

965ed78

Co-authored-by: Tcc0403 <[email protected]>

fix: ensure num2words availability check in smolvlm multimodal tests

65bb25a

style: run make checkstyle

e579681

fix typo!

b5c85fe

feat: add layer normalization support in Liger kernel for SmolVLM and…

3e84433

… InternVL models

fix: reload torch.nn and model imports in revert_liger_kernel_to_inte…

6062160

…rnvl function

Tcc0403 requested changes Oct 26, 2025

View reviewed changes

revert: remove InternVL layer normalization changes from add-smolvlm …

2754166

…branch

Tcc0403 approved these changes Oct 27, 2025

View reviewed changes

Tcc0403 merged commit f7c17b5 into linkedin:main Oct 27, 2025
3 of 7 checks passed

Add SmolVLM2 support #919

Add SmolVLM2 support #919

Uh oh!

Conversation

MilkClouds commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing Done

Uh oh!

Tcc0403 Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

MilkClouds Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

MilkClouds Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

Tcc0403 Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tcc0403 Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tcc0403 Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

MilkClouds Oct 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MilkClouds commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tcc0403 left a comment

Choose a reason for hiding this comment

Uh oh!

MilkClouds commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tcc0403 commented Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MilkClouds commented Oct 25, 2025 •

edited

Loading

Tcc0403 Oct 26, 2025 •

edited

Loading

MilkClouds commented Oct 26, 2025 •

edited

Loading

MilkClouds commented Oct 27, 2025 •

edited

Loading