pull: ssh remote fails, error in asyncssh 2.19.0, missing arguments

[sebastianbeyer]

Bug Report

Description

When I try to do a dvc pull for a ssh remote, I do get the following error:

TypeError: SSHClientConfig.__init__() missing 2 required positional arguments: 'host' and 'port'

This happens when using asyncssh==2.19.0, with asyncssh==2.18.0 it does work.
So my guess is that something changed in asyncssh api.

Reproduce

1. init dvc on one machine, add any testdata
   2 on another machine add the first machine as an ssh remote
2. dvc pull

Expected

The data should be pulled.

Environment information

verbose output

2024-12-19 11:29:08,486 DEBUG: v3.58.0 (pip), CPython 3.11.4 on Linux-5.14.21-150500.55.49_13.0.56-cray_shasta_c-x86_64-with-glibc2.31
2024-12-19 11:29:08,486 DEBUG: command: /users/sbeyer/mambaforge/envs/snakemake/bin/dvc pull -v
Collecting                                                                                                           |0.00 [00:00,    ?entry/s]
2024-12-19 11:29:10,178 ERROR: failed to connect to ssh (/gpfs/scratch/ehpc01/ifs-inputs_dvc-remote/files/md5) - SSHClientConfig.__init__() missing 2 required positional arguments: 'host' and 'port'
Traceback (most recent call last):
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc_data/index/fetch.py", line 118, in fetch
    data.fs.exists(data.path)
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 451, in exists
    return self.fs.exists(path)
           ^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/funcy/objects.py", line 47, in __get__
    return prop.__get__(instance, type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
                                                  ^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc_ssh/__init__.py", line 126, in fs
    return _SSHFileSystem(**self.fs_args)
                            ^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
                                                  ^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 90, in fs_args
    ret.update(self._prepare_credentials(**self._config))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc_ssh/__init__.py", line 55, in _prepare_credentials
    user_ssh_config = parse_config(host=config["host"])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/sshfs/config.py", line 23, in parse_config
    return SSHClientConfig.load(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/asyncssh/config.py", line 433, in load
    config = cls(last_config, reload, canonical, final, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: SSHClientConfig.__init__() missing 2 required positional arguments: 'host' and 'port'

Fetching
2024-12-19 11:29:10,312 ERROR: failed to pull data from the cloud - 96 files failed to download
Traceback (most recent call last):
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc/commands/data_sync.py", line 35, in run
    stats = self.repo.pull(
            ^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc/repo/pull.py", line 30, in pull
    processed_files_count = self.fetch(
                            ^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/users/sbeyer/mambaforge/envs/snakemake/lib/python3.11/site-packages/dvc/repo/fetch.py", line 200, in fetch
    raise DownloadError(failed_count)
dvc.exceptions.DownloadError: 96 files failed to download

2024-12-19 11:29:10,385 DEBUG: Analytics is enabled.
2024-12-19 11:29:10,644 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmplwq32hb2', '-v']
2024-12-19 11:29:10,658 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmplwq32hb2', '-v'] with pid 67719

Output of dvc doctor:

DVC version: 3.58.0 (pip)

Platform: Python 3.11.4 on Linux-5.14.21-150500.55.49_13.0.56-cray_shasta_c-x86_64-with-glibc2.31
Subprojects:
dvc_data = 3.16.7
dvc_objects = 5.1.0
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.3.9
Supports:
http (aiohttp = 3.11.10, aiohttp-retry = 2.9.1),
https (aiohttp = 3.11.10, aiohttp-retry = 2.9.1),
s3 (s3fs = 2024.10.0, boto3 = 1.35.36),
ssh (sshfs = 2024.9.0)
Config:
Global: /users/sbeyer/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: lustre on:/lustrep3
Caches: local
Remotes: s3, ssh
Workspace directory: lustre on :/lustrep3
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/818a09880835c36dded81b0960402c9f

Additional Information (if any):

[shcheklein]

Main research and discussion is happening here fsspec/sshfs#54 (comment)

[shcheklein]

There is a PR to pin the version, but I'm looking into mitigating it in a different way. For now please install the previous version of the asyncssh or pin it directly in your project.

[99-NinetyNine]

The issue is updated code[some ssh lib functions], and not reflected in dvc-ssh.

Passing port along side host solves the issue.

To edit this,

code is here

[jehout]

Hi,

I encounter the same problem.
But changing the line L57 does not fix it:

ERROR: unexpected error - SSHClientConfig.init() missing 2 required positional arguments: 'host' and 'port'

Do you have suggestions on how to solve this situation?

Best regards,

[shcheklein]

@jehout pin the asyncssh version to the previous one for now please.

[jehout]

pinning asyncssh works

but I got another issue:
ERROR: unexpected error - Permission denied for user

I will create another issue for this

[Crown421]

The same error occurs with the 3.59.0 debian package. After downgrading to 3.58.0 it works.

( In case it is helpful for anyone, apt-get install dvc=3.58.0)

[skshetry]

This has been fixed in the latest versions of dvc-ssh and sshfs. If you are having this issue, please pip install --upgrade dvc-ssh.

If you are using an older version of dvc (<3.44.0) for some reason, you can try upgrading to the latest version of sshfs:

pip install --upgrade sshfs

The same error occurs with the 3.59.0 debian package. After downgrading to 3.58.0 it works.

( In case it is helpful for anyone, apt-get install dvc=3.58.0)

Debian package will pick up the new versions the next time we release it. For now, I think using 3.58.0 is a good idea. I'll notify when we create a new release.