FIX Task-type add_adapter: forward autocast_adapter_dtype, stop mutating modules_to_save

[Vedant-Agarwal]

Two related bugs in the task-specific PeftModel subclasses (PeftModelForSequenceClassification, PeftModelForTokenClassification, PeftModelForQuestionAnswering), both in the same __init__ / add_adapter methods. They are independent defects but touch adjacent lines, so they are here as two commits in one PR.

1. autocast_adapter_dtype is accepted, documented, and then dropped

All three add_adapter overrides declare autocast_adapter_dtype: bool = True and document it, but call

super().add_adapter(adapter_name, peft_config, low_cpu_mem_usage=low_cpu_mem_usage)

without forwarding it, so PeftModel.add_adapter's default True always wins. A user who explicitly opts out still gets fp32 adapters on an fp16/bf16 base model, silently.

This also reaches PeftModel.load_adapter, which forwards autocast_adapter_dtype into add_adapter; its trailing cast_adapter_dtype call cannot repair it, because that function returns early when the flag is False.

Reproducing on main with an fp16 base model — __init__ is the control and behaves correctly, which isolates the fault to add_adapter:

PeftModelForSequenceClassification
  __init__     (autocast=False) -> {torch.float16}
  add_adapter  (autocast=False) -> {torch.float32}   # expected float16
  load_adapter (autocast=False) -> {torch.float32}   # expected float16

(identical for TokenClassification and QuestionAnswering)

Fix: forward the argument. After it, all three rows are {torch.float16}.

2. modules_to_save is mutated in place and grows without bound

Six sites do:

peft_config.modules_to_save.extend(classifier_module_names)

This mutates the list the caller passed in, and has no membership check, so it is not idempotent. Since PEFT stores the config by reference, reusing one config object — across CV folds or a sweep — appends the classifier names again each time, and retroactively corrupts the config of models already built from it. That corrupted list is what save_pretrained writes to adapter_config.json:

user config          : ['my_head']
after model 1        : ['my_head', 'classifier', 'score']
after models 2 and 3 : ['my_head', 'classifier', 'score', 'classifier', 'score', 'classifier', 'score']
model 1's saved adapter_config.json: [same 7-element list]

Fix: rebind rather than mutate, and dedupe while preserving order. The rebind stops the caller's list being touched; the dedupe makes repeat calls idempotent.

Scope note: this stops the growth and the aliasing, but the config attribute is still rebound to include the head names after first use, so reusing that same object for a later non-classification model still carries ["classifier", "score"]. Removing that too would mean deep-copying the config, which is a larger behaviour change and out of scope here.

Tests

grep -rn "classifier_module_names\|qa_module_names" tests/ currently returns nothing — this injection logic has no coverage at all, which is why both bugs survived. Added TestTaskTypeAddAdapterAutocastDtype and TestTaskTypeModulesToSave in tests/test_other.py, parametrized over all three task types: 24 tests covering the autocast path via both add_adapter and load_adapter (including the True case, so the tests also guard against over-correcting), config growth across repeated get_peft_model and add_adapter calls, the written adapter_config.json, and that the caller's list object is left alone.

Verified non-vacuous: reverting fix 1 alone fails exactly the 6 autocast=False cases while the 6 autocast=True cases still pass; reverting fix 2 alone fails all 12 of its tests.

pytest tests/test_other.py 74 passed, tests/test_auto.py 16 passed, ruff check/format and doc-builder style clean.

(AI-assisted: implemented with Claude Code.)

[BenjaminBossan]

Thanks for bringing this to our attention, I could indeed reproduce the bugs.

First, please split the PR into two, as the changes are unrelated. Please keep the autocast issue in this PR, as this is the significant error, the fix is rather low priority.

Regarding the autocast issue, the fix itself is good but the test needs work. Instead of putting the test into test_other.py, it should move to testing_common.py. Then the corresponding test files, test_seq_classifier.py and test_token_classification_qa.py. should be updated to call this common test. Let's also update test_decoder_models.py and test_encoder_decoder_models.py, even if they're not affected.

The parametrization of the test should be over the base model dtype with float16 and bfloat16 being tested. There is no need to explicitly test float32 or autocast_adapter_dtype=True. The modules_to_save part should be kept out of this PR.

[Vedant-Agarwal]

Thanks for the review.

Split: this PR now only contains the autocast_adapter_dtype forwarding fix (the three super().add_adapter() calls). The modules_to_save change and its tests are removed and will follow as a separate PR.

Test restructure: the test moved to tests/testing_common.py as
PeftCommonTester._test_add_adapter_no_autocast_adapter_dtype, called from
test_seq_classifier.py, test_decoder_models.py and test_encoder_decoder_models.py.
There is no tests/test_token_classification_qa.py on main, so I added one with
TestTokenClassificationModels and TestQuestionAnsweringModels over the tiny BERT/RoBERTa
backbones. Without it, PeftModelForTokenClassification and PeftModelForQuestionAnswering
(two of the three classes the fix touches) would have no coverage. It only calls the new
common test for now; happy to change the scope or the file name if you had something else
in mind.

Parametrization is over the base model dtype, float16 and bfloat16 only; the float32 and
autocast_adapter_dtype=True cases are gone. The model is created with
autocast_adapter_dtype=False, and the test asserts that the adapter added via add_adapter
and the one added via load_adapter have the same float dtypes as the adapter created by
get_peft_model. It compares against that reference rather than against the base dtype
directly because a few methods (PSOFT, for instance) deliberately keep some weights in
float32 regardless. Prompt learning is skipped (no tuner layers to cast), as are AdaLoRA and
ShadowPEFT with SEQ_CLS, which do not support multiple adapters.

Tests (CPU, macOS, transformers main). The new test alone:

• test_seq_classifier.py: 168 passed, 30 skipped
• test_token_classification_qa.py: 32 passed
• test_decoder_models.py: 378 passed, 102 skipped
• test_encoder_decoder_models.py: 124 passed, 20 skipped

Reverting the source fix makes all SEQ_CLS/TOKEN_CLS/QUESTION_ANS cases fail, so it is a real
regression test. Full runs of those four files plus test_other.py show no new failures
against a clean checkout of main; the failures that remain (FrodConfig, and a local abort in
test_forward_with_labels) reproduce on main as well. make quality passes.

This PR was written with AI assistance (Claude Code). I reviewed every changed line and ran
the tests listed above.

[BenjaminBossan]

Thanks for splitting off the other changes and for reworking the testing. I still have a comment about the test though, please check.

There is no tests/test_token_classification_qa.py on main

Sorry about the confusion, I had that file locally and forgot that it's not checked in.

[BenjaminBossan]

Hmm, I'm not a fan of getting the dtypes of the first adapter as reference. If a new PEFT method is added where this is already incorrect, then we wouldn't even notice. I think we should compare against the expected dtype from the base model, i.e. against dtype. To check this, I updated the test as follows:

    def _test_add_adapter_no_autocast_adapter_dtype(self, model_id, config_cls, config_kwargs, dtype):
        # With autocast_adapter_dtype=False, adapters that are added after the PeftModel was created must keep the
        # dtype of the base model instead of being upcast to float32. This covers add_adapter, which some task types
        # override, as well as load_adapter, which routes through add_adapter.
        if issubclass(config_cls, PromptLearningConfig):
            pytest.skip("Prompt learning does not create tuner layers whose dtype could be autocast.")
        if config_cls == AdaLoraConfig:
            pytest.skip("AdaLoRA does not support multiple adapters")
        if issubclass(config_cls, ShadowConfig) and config_kwargs.get("task_type") == "SEQ_CLS":
            pytest.skip("ShadowPEFT does not support multiple adapters for sequence classification")

        def get_adapter_dtype(model, adapter_name):
            dtypes = set()
            for name, param in model.named_parameters():
                if (model.prefix in name) and (adapter_name in name) and param.is_floating_point:
                    dtypes.add(param.dtype)
            if not dtypes:
                raise ValueError("Could not determine the dtype of this adapter")
            return dtypes

        expected_dtype = {dtype}

        with hub_online_once(model_id):
            model = self.transformers_class.from_pretrained(model_id, dtype=dtype)
            config = config_cls(
                base_model_name_or_path=model_id,
                **config_kwargs,
            )
            model = get_peft_model(model, config, autocast_adapter_dtype=False)
            assert get_adapter_dtype(model, "default") == expected_dtype
            with tempfile.TemporaryDirectory() as tmp_dirname:
                model.save_pretrained(tmp_dirname)

                model.add_adapter("added", config, autocast_adapter_dtype=False)
                assert get_adapter_dtype(model, "added") == expected_dtype

                # load_adapter goes through the same add_adapter code path
                model.load_adapter(tmp_dirname, adapter_name="loaded", autocast_adapter_dtype=False)
                assert get_adapter_dtype(model, "loaded") == expected_dtype

The test generally passes, with the exception of ShadowPEFT and the following three:

FAILED tests/test_encoder_decoder_models.py::TestEncoderDecoderModels::test_add_adapter_no_autocast_adapter_dtype[dtype0-IA3Config-config_kwargs12-peft-internal-testing/tiny-random-T5ForConditionalGeneration-calibrated] - AssertionError
FAILED tests/test_encoder_decoder_models.py::TestEncoderDecoderModels::test_add_adapter_no_autocast_adapter_dtype[dtype0-VBLoRAConfig-config_kwargs25-peft-internal-testing/tiny-random-T5ForConditionalGeneration-calibrated] - AssertionError
FAILED tests/test_encoder_decoder_models.py::TestEncoderDecoderModels::test_add_adapter_no_autocast_adapter_dtype[dtype0-OSFConfig-config_kwargs33-peft-internal-testing/tiny-random-T5ForConditionalGeneration-calibrated] - AssertionError

The reason why those fail is because they target some modules that are intentionally kept in float32 in T5:

https://github.com/huggingface/transformers/blob/3283d5f78ed6836d39430c8190a6e0500be78698/src/transformers/models/t5/modeling_t5.py#L537

So we would either have to extend the expected dtypes for those cases to include float32, not target wo, or use a different model than T5.

As for ShadowPEFT, we also see some float32 weights. I haven't checked the details, but I think that's intentional, so we'd have to add an exception there too.

[Vedant-Agarwal]

Thanks, that's a fair point — I've switched to your version of the test, including get_adapter_dtype and the expected_dtype = {dtype} assertions.

On the two exceptions:

T5. The three failures come from _keep_in_fp32_modules = ["wo"], and all three configs resolve to targets that include wo (IA3 ["k","v","wo"], VBLoRA ["q","k","v","o","wi","wo"], OSF has no t5 entry so it falls back to all-linear). The adapter is then correctly cast to the dtype of its own base layer by _move_adapter_to_device_of_base_layer, which is fp32 — so this is right behaviour, not a bug. VBLoRA's vblora_vector_bank is the same cause: it's a single shared parameter that every target layer casts, and the last target on T5 is a wo.

I went with your first option (extend the expected dtypes) but computed it instead of listing the configs: expected_dtype gains float32 only when the freshly loaded base model actually contains float32 weights. Across the 13 models in these four files × both dtypes, only T5 in float16 hits that branch — T5 in bfloat16 doesn't, because the fp32 pinning only applies to fp16. Everything else keeps the strict single-dtype check, so a new PEFT method that upcasts wrongly still gets caught. "Don't target wo" would have needed per-config target_modules overrides inside a common test, and swapping the model isn't possible since the model id is a test parameter.

The assertions use <= rather than ==: since get_adapter_dtype never returns an empty set, that is equality whenever expected_dtype holds one dtype, and it only relaxes in the T5/float16 branch (where == would wrongly fail e.g. LoRA, which only targets q/v and stays fp16).

ShadowPEFT. I looked into it and I don't think it's intentional. ShadowModel._create_and_replace does explicitly cast shadow_backbone/projection/head to the base dtype, and those come out fp16. The per-layer modules don't: ShadowLayer.update_layer calls _move_adapter_to_device_of_base_layer, but ShadowPEFT wraps whole decoder layers, so get_base_layer() returns a module without a weight, _get_base_layer_device_and_dtype returns (None, None), and the function returns early. shadow_down/shadow_up/shadow_update_* therefore keep nn.Linear's float32 default regardless of autocast_adapter_dtype. On tiny OPT in fp16: backbone 19 params fp16, the four per-layer groups 50 params fp32. The forward pass handles the mixed dtype, so nothing is broken, but the flag has no effect there. I've skipped ShadowPEFT with that reason written out rather than papering over it — happy to open a separate issue.

One other change to your version: param.is_floating_point → param.is_floating_point(). It's a method, so the filter was always truthy.

Results (CPU, transformers 5.16.1), -k add_adapter_no_autocast:

• test_seq_classifier.py: 168 passed, 30 skipped
• test_token_classification_qa.py: 32 passed
• test_decoder_models.py: 366 passed, 114 skipped
• test_encoder_decoder_models.py: 124 passed, 20 skipped

Reverting the fix in peft_model.py fails all 168 seq-cls and all 32 token-cls/QA cases, while decoder and encoder-decoder still pass, so it's a real regression test. make quality passes.

This PR was written with AI assistance (Claude Code). I reviewed every changed line and ran the tests listed above.