Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] 在跑8B-MPO finetune完成后保存模型出现UnboundLocalError: cannot access local variable 'default_config' where it is not associated with a value #841

Open
3 tasks done
JackeyHRan opened this issue Jan 9, 2025 · 6 comments
Open
3 tasks done

[Bug] 在跑8B-MPO finetune完成后保存模型出现UnboundLocalError: cannot access local variable 'default_config' where it is not associated with a value #841

JackeyHRan opened this issue Jan 9, 2025 · 6 comments

Comments

@JackeyHRan
Copy link

JackeyHRan commented Jan 9, 2025 •
edited
Loading

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

我在finetune完8B-MPO后保存模型出现了UnboundLocalError: cannot access local variable 'default_config' where it is not associated with a value

[INFO|trainer.py:3705] 2025-01-09 08:23:38,898 >> Saving model checkpoint to work_dirs/internvl_chat_v2_5/internvl2_5_8b-mpo-v2.2_0108
[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/InternVL/internvl_chat/internvl/train/internvl_chat_dpo.py", line 1056, in <module>
[rank0]:     main()
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/InternVL/internvl_chat/internvl/train/internvl_chat_dpo.py", line 1028, in main
[rank0]:     trainer.save_model()  # Saves the tokenizer too for easy upload
[rank0]:     ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/Weiyun1025/internvl-mpo-torch2.5.1/lib/python3.11/site-packages/transformers/trainer.py", line 3610, in save_model
[rank0]:     self._save(output_dir, state_dict=state_dict)
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/Weiyun1025/internvl-mpo-torch2.5.1/lib/python3.11/site-packages/transformers/trainer.py", line 3727, in _save
[rank0]:     self.model.save_pretrained(
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/Weiyun1025/internvl-mpo-torch2.5.1/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2616, in save_pretrained
[rank0]:     misplaced_generation_parameters = model_to_save.config._get_non_default_generation_parameters()
[rank0]:                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/bn/product-representation-va/MLLM4Relevance/Weiyun1025/internvl-mpo-torch2.5.1/lib/python3.11/site-packages/transformers/configuration_utils.py", line 1050, in _get_non_default_generation_parameters
[rank0]:     if default_config is not None:
[rank0]:        ^^^^^^^^^^^^^^
[rank0]: UnboundLocalError: cannot access local variable 'default_config' where it is not associated with a value

Reproduction

set -x


CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
GPUS=${GPUS:-8}
BATCH_SIZE=${BATCH_SIZE:-16}
PER_DEVICE_BATCH_SIZE=${PER_DEVICE_BATCH_SIZE:-2}
GRADIENT_ACC=$((BATCH_SIZE / PER_DEVICE_BATCH_SIZE / GPUS))
export OMP_NUM_THREADS=32

export PYTHONPATH="${PYTHONPATH}:$(pwd)"
export MASTER_PORT=34229
export TF_CPP_MIN_LOG_LEVEL=3
export LAUNCHER=pytorch

OUTPUT_DIR='work_dirs/internvl_chat_v2_5/internvl2_5_38b_hires_trainv2.4.1_testv2.0.3_20250108_lora'

if [ ! -d "$OUTPUT_DIR" ]; then
  mkdir -p "$OUTPUT_DIR"
fi

# number of gpus: 2
# batch size per gpu: 2
# gradient accumulation steps: 4
# total batch size: 16
# epoch: 1
torchrun \
  --nnodes=1 \
  --node_rank=0 \
  --master_addr=127.0.0.1 \
  --nproc_per_node=${GPUS} \
  --master_port=${MASTER_PORT} \
  internvl/train/internvl_chat_finetune.py \
  --model_name_or_path "./pretrained/InternVL2_5-38B" \
  --conv_style "internvl2_5" \
  --use_fast_tokenizer False \
  --output_dir ${OUTPUT_DIR} \
  --meta_path ./shell/data/relevance_v2.4.1.json \
  --overwrite_output_dir True \
  --force_image_size 448 \
  --max_dynamic_patch 6 \
  --down_sample_ratio 0.5 \
  --drop_path_rate 0.0 \
  --freeze_llm True \
  --freeze_mlp True \
  --freeze_backbone True \
  --use_llm_lora 16 \
  --vision_select_layer -1 \
  --dataloader_num_workers 4 \
  --bf16 True \
  --num_train_epochs 1 \
  --per_device_train_batch_size ${PER_DEVICE_BATCH_SIZE} \
  --gradient_accumulation_steps ${GRADIENT_ACC} \
  --evaluation_strategy "no" \
  --save_strategy "steps" \
  --save_steps 20000 \
  --save_total_limit 1 \
  --learning_rate 2e-5 \
  --weight_decay 0.05 \
  --warmup_ratio 0.03 \
  --lr_scheduler_type "cosine" \
  --logging_steps 1 \
  --max_seq_length 8192 \
  --do_train True \
  --grad_checkpoint True \
  --group_by_length True \
  --dynamic_image_size True \
  --use_thumbnail True \
  --ps_version 'v2' \
  --deepspeed "zero_stage3_config_34b.json" \
  --report_to "tensorboard" \
  2>&1 | tee -a "${OUTPUT_DIR}/training_log.txt"

Environment

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
absl-py                   2.1.0                    pypi_0    pypi
accelerate                1.1.1                    pypi_0    pypi
addict                    2.4.0                    pypi_0    pypi
aiohappyeyeballs          2.4.4                    pypi_0    pypi
aiohttp                   3.11.9                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
attrs                     24.2.0                   pypi_0    pypi
blas                      2.116                       mkl    conda-forge
blas-devel                3.9.0            16_linux64_mkl    conda-forge
boto3                     1.35.73                  pypi_0    pypi
botocore                  1.35.73                  pypi_0    pypi
brotli-python             1.1.0           py311hfdbb021_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py311hf29c0ef_0    conda-forge
charset-normalizer        3.4.0              pyhd8ed1ab_0    conda-forge
coloredlogs               15.0.1                   pypi_0    pypi
contourpy                 1.3.1                    pypi_0    pypi
cuda-cudart               12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.6.77                       0    nvidia
cuda-runtime              12.1.0                        0    nvidia
cuda-version              12.6                          3    nvidia
cycler                    0.12.1                   pypi_0    pypi
datasets                  3.1.0                    pypi_0    pypi
decord                    0.6.0                    pypi_0    pypi
deepspeed                 0.15.4                   pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
docstring-parser          0.16                     pypi_0    pypi
einops                    0.8.0                    pypi_0    pypi
environs                  11.0.0                   pypi_0    pypi
ffmpeg                    4.3                  hf484d3e_0    pytorch
filelock                  3.16.1             pyhd8ed1ab_0    conda-forge
flash-attn                2.7.0.post2              pypi_0    pypi
fonttools                 4.55.0                   pypi_0    pypi
freetype                  2.12.1               h267a509_2    conda-forge
frozenlist                1.5.0                    pypi_0    pypi
fsspec                    2024.9.0                 pypi_0    pypi
giflib                    5.2.2                hd590300_0    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gmpy2                     2.1.5           py311h0f6cedb_2    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
grpcio                    1.68.1                   pypi_0    pypi
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hjson                     3.1.0                    pypi_0    pypi
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
huggingface-hub           0.26.3                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
humanize                  4.11.0                   pypi_0    pypi
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.10               pyhd8ed1ab_0    conda-forge
imageio                   2.36.1                   pypi_0    pypi
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1                    pypi_0    pypi
jpeg                      9e                   h166bdaf_2    conda-forge
kiwisolver                1.4.7                    pypi_0    pypi
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 hfd0df8a_0    conda-forge
ld_impl_linux-64          2.43                 h712a8e2_2    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0            16_linux64_mkl    conda-forge
libcblas                  3.9.0            16_linux64_mkl    conda-forge
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.11.1.6                      0    nvidia
libcurand                 10.3.7.77                     0    nvidia
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libdeflate                1.17                 h0b41bf4_0    conda-forge
libexpat                  2.6.4                h5888daf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.2.0               h77fa898_1    conda-forge
libgcc-ng                 14.2.0               h69a702a_1    conda-forge
libgfortran               14.2.0               h69a702a_1    conda-forge
libgfortran-ng            14.2.0               h69a702a_1    conda-forge
libgfortran5              14.2.0               hd5240d6_1    conda-forge
libhwloc                  2.11.2          default_he43201b_1000    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libjpeg-turbo             2.0.0                h9bf148f_0    pytorch
liblapack                 3.9.0            16_linux64_mkl    conda-forge
liblapacke                3.9.0            16_linux64_mkl    conda-forge
libnpp                    12.0.2.50                     0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvjitlink              12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libpng                    1.6.43               h2797004_0    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libstdcxx                 14.2.0               hc0a3c3a_1    conda-forge
libstdcxx-ng              14.2.0               h4852527_1    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp                   1.2.4                h1daa5a0_1    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.7               hc051c1a_1    conda-forge
libzlib                   1.2.13               h4ab18f5_6    conda-forge
liger-kernel              0.4.2                    pypi_0    pypi
llvm-openmp               15.0.7               h0cdce71_0    conda-forge
lmdeploy                  0.6.4                    pypi_0    pypi
loguru                    0.7.2                    pypi_0    pypi
markdown                  3.7                      pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                3.0.2           py311h2dc5d0c_0    conda-forge
marshmallow               3.23.1                   pypi_0    pypi
matplotlib                3.9.3                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2022.1.0           h84fe81f_915    conda-forge
mkl-devel                 2022.1.0           ha770c72_916    conda-forge
mkl-include               2022.1.0           h84fe81f_915    conda-forge
mmengine                  0.10.5                   pypi_0    pypi
mpc                       1.3.1                h24ddda3_1    conda-forge
mpfr                      4.2.1                h90cbb55_3    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
msgpack                   1.1.0                    pypi_0    pypi
multidict                 6.1.0                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
multiprocessing-logging   0.3.4                    pypi_0    pypi
ncurses                   6.5                  he02047a_1    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
ninja                     1.11.1.2                 pypi_0    pypi
numpy                     2.1.3           py311h71ddf71_0    conda-forge
opencv-python             4.10.0.84                pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.4.0                hb9d3cd8_0    conda-forge
packaging                 24.2                     pypi_0    pypi
pandas                    2.2.3                    pypi_0    pypi
peft                      0.13.2                   pypi_0    pypi
pillow                    9.4.0           py311h50def17_1    conda-forge
pip                       24.3.1             pyh8b19718_0    conda-forge
platformdirs              4.3.6                    pypi_0    pypi
propcache                 0.2.1                    pypi_0    pypi
protobuf                  5.29.0                   pypi_0    pypi
psutil                    6.1.0                    pypi_0    pypi
pthread-stubs             0.4               hb9d3cd8_1002    conda-forge
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   18.1.0                   pypi_0    pypi
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pydantic                  2.10.2                   pypi_0    pypi
pydantic-core             2.27.1                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
pyparsing                 3.2.0                    pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.9          hb806964_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.1                    pypi_0    pypi
python_abi                3.11                    5_cp311    conda-forge
pytorch                   2.5.1           py3.11_cuda12.1_cudnn9.1.0_0    pytorch
pytorch-cuda              12.1                 ha16c6d3_6    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.2                   pypi_0    pypi
pyyaml                    6.0.2           py311h9ecbd09_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
regex                     2024.11.6                pypi_0    pypi
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
rich                      13.9.4                   pypi_0    pypi
s3transfer                0.10.4                   pypi_0    pypi
safetensors               0.4.5                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                75.6.0             pyhff2d567_1    conda-forge
shtab                     1.7.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
tbb                       2021.13.0            hceb3a55_1    conda-forge
tensorboard               2.18.0                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
termcolor                 2.5.0                    pypi_0    pypi
timm                      1.0.11                   pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tokenizers                0.20.3                   pypi_0    pypi
torchaudio                2.5.1               py311_cu121    pytorch
torchtriton               3.1.0                     py311    pytorch
torchvision               0.20.1              py311_cu121    pytorch
tqdm                      4.67.1                   pypi_0    pypi
transformers              4.45.1                   pypi_0    pypi
trl                       0.10.1                   pypi_0    pypi
typeguard                 4.4.1                    pypi_0    pypi
typing_extensions         4.12.2             pyha770c72_0    conda-forge
tyro                      0.9.2                    pypi_0    pypi
tzdata                    2024.2                   pypi_0    pypi
urllib3                   2.2.3              pyhd8ed1ab_0    conda-forge
werkzeug                  3.1.3                    pypi_0    pypi
wheel                     0.45.1             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hb9d3cd8_1    conda-forge
xorg-libxdmcp             1.1.5                hb9d3cd8_0    conda-forge
xxhash                    3.5.0                    pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yapf                      0.43.0                   pypi_0    pypi
yarl                      1.18.3                   pypi_0    pypi
zlib                      1.2.13               h4ab18f5_6    conda-forge
zstandard                 0.23.0          py311hbc35293_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

Error traceback

No response
@laserwave
Copy link

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

@JackeyHRan
Copy link
Author

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

感谢，我去尝试一下

@snowyrain
Copy link

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

请问你遇到这个问题了吗？File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1705, in get_input_embeddings
raise NotImplementedError
应该怎么解决啊，感谢

@JackeyHRan
Copy link
Author

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

请问你遇到这个问题了吗？File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1705, in get_input_embeddings raise NotImplementedError 应该怎么解决啊，感谢

虽然我没遇到过这个问题，但我觉得大概率是transformers版本的问题

@snowyrain
Copy link

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

请问你遇到这个问题了吗？File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1705, in get_input_embeddings raise NotImplementedError 应该怎么解决啊，感谢

虽然我没遇到过这个问题，但我觉得大概率是transformers版本的问题

谢谢，是我自己的代码问题。想问题下你跑过26B的mpo吗？我8卡，80G显存不够。

@JackeyHRan
Copy link
Author

@JackeyHRan transformers 4.45.1 遇到了同样的问题，降级到4.40.0似乎可以正常保存权重

请问你遇到这个问题了吗？File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1705, in get_input_embeddings raise NotImplementedError 应该怎么解决啊，感谢

虽然我没遇到过这个问题，但我觉得大概率是transformers版本的问题

谢谢，是我自己的代码问题。想问题下你跑过26B的mpo吗？我8卡，80G显存不够。

我8B mpo是将将够的，因为默认脚本是全参数微调，所以26b应该不够

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@laserwave @snowyrain @JackeyHRan