Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

求问详细的环境清单 #11

Open
Sunrepe opened this issue Dec 24, 2024 · 5 comments
Open

求问详细的环境清单 #11

Sunrepe opened this issue Dec 24, 2024 · 5 comments

Comments

@Sunrepe
Copy link

Sunrepe commented Dec 24, 2024 •
edited
Loading

你好，我环境调试了两天了，但是始终有各种错误，无法正确运行

请问你可以把这个仓库对应的完整的环境信息给出来吗？

也就是pip list里的所有的版本信息，以及cuda版本、python版本，感谢
@Yifan-Song793
Copy link
Owner

请问你具体遇到了什么报错。我使用的是 CUDA 11.8，Python 3.10 环境

@Sunrepe
Copy link
Author

Sunrepe commented Dec 25, 2024 •
edited
Loading

问题列表：
1，在cuda 11.8，python-3.10环境下，直接运行setup.sh 第一个问题：

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.2.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

之后将numpy降级为numpy==1.26.4，重新运行setup.sh无报错
2，选择model_name=Llama-2-7b-hf, 运行以下命令：

bash run_eto.sh sciworld eto_o1 /data1/llm_models/ saves/

遇到错误：ImportError: cannot import name 'EncoderDecoderCache' from 'transformers' (/data1/envs/miniconda3/envs/eto1/lib/python3.10/site-packages/transformers/__init__.py)
选择重新将peft降级为peft==0.10.0,
3，此时模型和数据都可以正常加载，但是trainer = Trainer()这里继续报错：TypeError: Accelerator.__init__() got an unexpected keyword argument 'dispatch_batches', 选择降级pip install -U accelerate==0.30.1
4，此时出现错误RuntimeError: handle_0 INTERNAL ASSERT FAILED at "../c10/cuda/driver_api.cpp":15, please report a bug to PyTorch.
找遍全网没有找到合适的解决方案。

我试过降低batch等操作，但是都没用，我的显卡驱动器：
 >>>nvidia-smi -a

==============NVSMI LOG==============

Timestamp                                 : Wed Dec 25 13:54:01 2024
Driver Version                            : 525.60.13
CUDA Version                              : 12.0

我的cuda版本
>>>nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

@Yifan-Song793
Copy link
Owner

这是我的环境，请参考

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
accelerate                0.30.1                   pypi_0    pypi
ai2thor                   2.1.0                    pypi_0    pypi
aiohappyeyeballs          2.4.3                    pypi_0    pypi
aiohttp                   3.10.9                   pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
alfworld                  0.3.4                    pypi_0    pypi
alsa-lib                  1.2.3.2              h166bdaf_0    conda-forge
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.6.0                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     24.2.0                   pypi_0    pypi
backoff                   2.2.1                    pypi_0    pypi
bcrypt                    4.2.0                    pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
blis                      0.7.11                   pypi_0    pypi
ca-certificates           2024.8.30            hbcca054_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
catalogue                 2.0.10                   pypi_0    pypi
certifi                   2024.8.30                pypi_0    pypi
cffi                      1.17.1                   pypi_0    pypi
charset-normalizer        3.4.0                    pypi_0    pypi
cleantext                 1.1.4                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpathlib              0.19.0                   pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
cmake                     3.30.4                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
confection                0.1.5                    pypi_0    pypi
cryptography              43.0.1                   pypi_0    pypi
cymem                     2.0.8                    pypi_0    pypi
cython                    3.0.11                   pypi_0    pypi
datasets                  2.9.0                    pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
diskcache                 5.6.3                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
docker                    6.1.3                    pypi_0    pypi
docker-compose            1.29.2                   pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
dockerpty                 0.4.1                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docstring-parser          0.16                     pypi_0    pypi
einops                    0.8.0                    pypi_0    pypi
en-core-web-lg            3.6.0                    pypi_0    pypi
env                       0.1.0                    pypi_0    pypi
eval-type-backport        0.2.0                    pypi_0    pypi
exceptiongroup            1.2.2                    pypi_0    pypi
expat                     2.4.8                h27087fc_0    conda-forge
faiss-cpu                 1.7.4                    pypi_0    pypi
farama-notifications      0.0.4                    pypi_0    pypi
fast-downward-textworld   20.6.2                   pypi_0    pypi
fastapi                   0.115.0                  pypi_0    pypi
filelock                  3.16.1                   pypi_0    pypi
flash-attn                2.4.2                    pypi_0    pypi
flask                     2.1.2                    pypi_0    pypi
flatbuffers               24.3.25                  pypi_0    pypi
fontconfig                2.14.0               h8e229c2_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
frozenlist                1.4.1                    pypi_0    pypi
fschat                    0.2.34                   pypi_0    pypi
fsspec                    2024.9.0                 pypi_0    pypi
gdown                     5.2.0                    pypi_0    pypi
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.43                   pypi_0    pypi
graphite2                 1.3.13            h58526e2_1001    conda-forge
gym                       0.24.0                   pypi_0    pypi
gym-notices               0.0.8                    pypi_0    pypi
gymnasium                 1.0.0                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
harfbuzz                  2.9.1                h83ec7ef_1    conda-forge
hashids                   1.3.1                    pypi_0    pypi
httpcore                  1.0.6                    pypi_0    pypi
httptools                 0.6.1                    pypi_0    pypi
httpx                     0.27.2                   pypi_0    pypi
huggingface-hub           0.25.2                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
icu                       68.2                 h9c3ff4c_0    conda-forge
idna                      3.10                     pypi_0    pypi
importlib-metadata        8.5.0                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
intercode-bench           0.1.22                   pypi_0    pypi
interegular               0.3.3                    pypi_0    pypi
itsdangerous              2.2.0                    pypi_0    pypi
jericho                   3.2.1                    pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
jpeg                      9e                   h166bdaf_1    conda-forge
jsonschema                3.2.0                    pypi_0    pypi
langcodes                 3.4.1                    pypi_0    pypi
language-data             1.2.0                    pypi_0    pypi
lark                      1.2.2                    pypi_0    pypi
latex2mathml              3.77.0                   pypi_0    pypi
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.40                 h12ee557_0
libffi                    3.3                  he6710b0_2
libgcc-ng                 11.2.0               h1234567_1
libglib                   2.68.4               h3e27bee_0    conda-forge
libgomp                   11.2.0               h1234567_1
libiconv                  1.17                 h166bdaf_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              11.2.0               h1234567_1
libtiff                   4.2.0                hf544144_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.12                          0    conda-forge
lightgbm                  4.5.0                    pypi_0    pypi
llvmlite                  0.43.0                   pypi_0    pypi
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
marisa-trie               1.2.0                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markdown2                 2.5.1                    pypi_0    pypi
markupsafe                3.0.1                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mementos                  1.3.1                    pypi_0    pypi
more-itertools            10.5.0                   pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
msgpack                   1.1.0                    pypi_0    pypi
multidict                 6.1.0                    pypi_0    pypi
multiprocess              0.70.14                  pypi_0    pypi
murmurhash                1.0.10                   pypi_0    pypi
mysql-connector-python    9.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0
nest-asyncio              1.6.0                    pypi_0    pypi
networkx                  3.2.1                    pypi_0    pypi
nh3                       0.2.18                   pypi_0    pypi
ninja                     1.11.1.1                 pypi_0    pypi
nltk                      3.9.1                    pypi_0    pypi
nmslib                    2.1.1                    pypi_0    pypi
numba                     0.60.0                   pypi_0    pypi
numpy                     1.26.3                   pypi_0    pypi
nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
nvidia-cudnn-cu12         8.9.2.26                 pypi_0    pypi
nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
nvidia-nccl-cu12          2.18.1                   pypi_0    pypi
nvidia-nvjitlink-cu12     12.6.77                  pypi_0    pypi
nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
onnxruntime               1.19.2                   pypi_0    pypi
openai                    0.28.1                   pypi_0    pypi
opencv-python             4.9.0.80                 pypi_0    pypi
openjdk                   11.0.9.1             h5cc2fde_1    conda-forge
openssl                   1.1.1o               h166bdaf_0    conda-forge
outcome                   1.3.0.post0              pypi_0    pypi
outlines                  0.0.34                   pypi_0    pypi
packaging                 24.1                     pypi_0    pypi
pandas                    2.2.3                    pypi_0    pypi
paramiko                  3.5.0                    pypi_0    pypi
pathlib-abc               0.1.1                    pypi_0    pypi
pathy                     0.11.0                   pypi_0    pypi
pcre                      8.45                 h9c3ff4c_0    conda-forge
peft                      0.13.1                   pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.2             py39h06a4308_0
pixman                    0.40.0               h36c2ea0_0    conda-forge
platformdirs              4.3.6                    pypi_0    pypi
pluggy                    1.5.0                    pypi_0    pypi
preshed                   3.0.9                    pypi_0    pypi
progressbar2              4.5.0                    pypi_0    pypi
prometheus-client         0.21.0                   pypi_0    pypi
prompt-toolkit            3.0.48                   pypi_0    pypi
propcache                 0.2.0                    pypi_0    pypi
protobuf                  5.28.2                   pypi_0    pypi
psutil                    6.0.0                    pypi_0    pypi
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   17.0.0                   pypi_0    pypi
pybind11                  2.6.1                    pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pydantic                  2.9.2                    pypi_0    pypi
pydantic-core             2.23.4                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
pyjnius                   1.6.1                    pypi_0    pypi
pynacl                    1.5.0                    pypi_0    pypi
pynvml                    11.5.0                   pypi_0    pypi
pyopenssl                 24.2.1                   pypi_0    pypi
pyrsistent                0.20.0                   pypi_0    pypi
pyserini                  0.17.0                   pypi_0    pypi
pysocks                   1.7.1                    pypi_0    pypi
pytest                    8.3.3                    pypi_0    pypi
python                    3.9.0                hdb3f193_2
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             0.21.1                   pypi_0    pypi
python-utils              3.8.2                    pypi_0    pypi
pytz                      2024.2                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
rank-bm25                 0.2.2                    pypi_0    pypi
rapidfuzz                 3.10.0                   pypi_0    pypi
ray                       2.37.0                   pypi_0    pypi
readline                  8.2                  h5eee18b_0
referencing               0.35.1                   pypi_0    pypi
regex                     2024.9.11                pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
requests-mock             1.12.1                   pypi_0    pypi
responses                 0.18.0                   pypi_0    pypi
rich                      13.9.2                   pypi_0    pypi
rpds-py                   0.20.0                   pypi_0    pypi
safetensors               0.4.5                    pypi_0    pypi
scikit-learn              1.2.2                    pypi_0    pypi
scipy                     1.13.1                   pypi_0    pypi
selenium                  4.2.0                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
sentry-sdk                2.16.0                   pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                75.1.0           py39h06a4308_0
shellingham               1.5.4                    pypi_0    pypi
shortuuid                 1.0.13                   pypi_0    pypi
shtab                     1.7.1                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smart-open                6.4.0                    pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
soupsieve                 2.6                      pypi_0    pypi
spacy                     3.6.1                    pypi_0    pypi
spacy-legacy              3.0.12                   pypi_0    pypi
spacy-loggers             1.0.5                    pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0
srsly                     2.4.8                    pypi_0    pypi
starlette                 0.38.6                   pypi_0    pypi
svgwrite                  1.4.3                    pypi_0    pypi
sympy                     1.13.3                   pypi_0    pypi
tatsu                     5.8.3                    pypi_0    pypi
termcolor                 2.5.0                    pypi_0    pypi
texttable                 1.7.0                    pypi_0    pypi
textworld                 1.6.1                    pypi_0    pypi
thefuzz                   0.20.0                   pypi_0    pypi
thinc                     8.1.12                   pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tiktoken                  0.6.0                    pypi_0    pypi
tk                        8.6.14               h39e8969_0
tokenizers                0.15.2                   pypi_0    pypi
tomli                     2.0.2                    pypi_0    pypi
torch                     2.1.0+cu118              pypi_0    pypi
torchaudio                2.1.0+cu118              pypi_0    pypi
torchvision               0.16.0+cu118             pypi_0    pypi
tqdm                      4.66.5                   pypi_0    pypi
transformers              4.37.2                   pypi_0    pypi
trio                      0.26.2                   pypi_0    pypi
trio-websocket            0.11.1                   pypi_0    pypi
triton                    2.1.0                    pypi_0    pypi
trl                       0.7.7                    pypi_0    pypi
typer                     0.9.4                    pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tyro                      0.8.11                   pypi_0    pypi
tzdata                    2024.2                   pypi_0    pypi
urllib3                   1.26.20                  pypi_0    pypi
urllib3-secure-extra      0.1.0                    pypi_0    pypi
uvicorn                   0.31.1                   pypi_0    pypi
uvloop                    0.20.0                   pypi_0    pypi
vllm                      0.4.0.post1              pypi_0    pypi
wandb                     0.18.3                   pypi_0    pypi
wasabi                    1.1.3                    pypi_0    pypi
watchfiles                0.24.0                   pypi_0    pypi
wavedrom                  2.0.3.post3              pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
weasel                    0.4.1                    pypi_0    pypi
websocket-client          0.59.0                   pypi_0    pypi
websockets                13.1                     pypi_0    pypi
werkzeug                  2.0.3                    pypi_0    pypi
wheel                     0.44.0           py39h06a4308_0
wrapt                     1.16.0                   pypi_0    pypi
wsproto                   1.2.0                    pypi_0    pypi
xformers                  0.0.23.post1             pypi_0    pypi
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xxhash                    3.5.0                    pypi_0    pypi
xz                        5.4.6                h5eee18b_1
yarl                      1.14.0                   pypi_0    pypi
zipp                      3.20.2                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_1
zstd                      1.5.0                ha95c52a_0    conda-forge

@Yifan-Song793
Copy link
Owner

另外可以参考我们最新项目 IPR 的 setup：https://github.com/WeiminXiong/IPR

@Sunrepe
Copy link
Author

Sunrepe commented Dec 26, 2024 •
edited
Loading

你好，你这个环境我确实是配不好了，想请问能加个微信更细问问你吗？我的微信号是sunrepe

我用别的方法复现了eto的工作，但是结果和论文里的差距有10个点左右，也想问问是不是细节有问题！

希望你能同意！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Sunrepe @Yifan-Song793