Forward args to _get_remote_config() and honour core/no_scm if present #10719

rgoya · 2025-04-10T05:29:44Z

This is a proposed fix for #10608, the code here makes steps 9 and 10 described in the issue work.

Summary:

This change allows a user to access the dvc information in an environment that is disconnected from the original Git backend (e.g. in a deployed container, see #10608), by using something like:

dvc.api.get_url(path,
                repo,
                config={"core": {"no_scm": True}}
                )

Description:

Mainly, a call to dvc/repo/open_repo.py:open_repo(url, *args, **kwargs) may contain a parameter config in **kwargs. With this config a user might indicate they do not want to access the repo with Git support, by using config={"core": {"no_scm": True}}.

During the execution of dvc/repo/open_repo.py:open_repo(), there is a call to a function dvc/repo/open_repo.py:_get_remote_config() that returns the remote configuration({"core": {"remote"}}. This is then merged to the user provided config parameter before calling Repo(url, *args, **kwargs).

dvc/repo/open_repo.py:_get_remote_config(), in turn, does a quick Repo() call to get the remote configuration. However, it does not use any of the parameters requested via dvc/repo/open_repo.py:open_repo() and thus relies entirely on the contents of .dvc/config. This means that even if the user requested no SCM support, it will try to look for a Git repo if .dvc/config says so, and fail if it does not find it.

This PR modifies dvc/repo/open_repo.py:_get_remote_config() to receive *args, **kwargs and honour the request to use or ignore Git support when accessing the dvc repo.

❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

codecov · 2025-04-10T05:33:22Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.08%. Comparing base (2431ec6) to head (0712ad2).
Report is 45 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10719      +/-   ##
==========================================
+ Coverage   90.68%   91.08%   +0.39%     
==========================================
  Files         504      504              
  Lines       39795    39970     +175     
  Branches     3141     3158      +17     
==========================================
+ Hits        36087    36405     +318     
+ Misses       3042     2938     -104     
+ Partials      666      627      -39

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rgoya · 2025-04-10T05:52:24Z

dvc/repo/open_repo.py

+        # It seems some tests might be passing a 'config' key that is not a dict
+        if not isinstance(user_config, dict):
+            user_config = {}


Looking into this, some tests send kwargs = {'config': None, ...; this safeguard protects against this.

skshetry · 2025-04-10T06:34:06Z

dvc/repo/open_repo.py

+
+        if no_scm_flag is not None:
+            # Honour specific SCM treatment if requested in the call
+            repo = Repo(url, config={"core": {"no_scm": no_scm_flag}})


AFAIR, Repo(config=...) should just work.

Suggested change

repo = Repo(url, config={"core": {"no_scm": no_scm_flag}})

repo = Repo(url, config=kwargs.get("config"))

I don't want to specialize core.no_scm in any way or handle it.

I agree that it doesn't feel ideal to handle core.no_scm itself. Your solution makes sense and it works for my specific use case, but it triggers other errors in the dvc test suite; which makes me think there are other non-core.no_scm configuration options that are being used that _get_remote_config() doesn't like.

(I went with the core.no_scm specific approach to highlight the need.)

These are the errors I get when using repo = Repo(url, config=kwargs.get("config")):

FAILED tests/func/test_import.py::test_import_no_hash[files1-expected_info_calls1] - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_repo_index.py::test_data_index - dvc_data.index.index.DataIndexDirError: failed to load directory ('edir',) FAILED tests/func/repro/test_repro_pull.py::test_repro_pulls_missing_import - dvc.exceptions.ReproductionError: failed to reproduce 'foo.dvc' FAILED tests/func/test_data_cloud.py::test_pull_external_dvc_imports - FileNotFoundError: [Errno 2] No storage files available: 'foo' FAILED tests/func/test_data_cloud.py::test_pull_external_dvc_imports_mixed - FileNotFoundError: [Errno 2] No storage files available: 'foo' FAILED tests/func/repro/test_repro.py::test_repro_pulls_missing_import - dvc.exceptions.ReproductionError: failed to reproduce 'foo.dvc' FAILED tests/func/test_import.py::test_import_dir - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_import.py::test_import_file_from_dir - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_import.py::test_import_file_from_dir_to_dir - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_import.py::test_import_rev - FileNotFoundError: [Errno 2] No storage files available: 'foo' FAILED tests/func/test_import.py::test_pull_imported_stage - dvc.exceptions.CheckoutError: Checkout failed for following targets: FAILED tests/func/test_import.py::test_pull_import_no_download - FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/_3/h8jc4f6d5gg6dwk7t6464_z80000gp/T/pytest-of-rodrigo.goya/pytest-17/popen-gw13/test_pull_import_no_download0/.dvc/cache/fs/local/e3501e821bcee8f40107794afbe767d1/.F0VDC8H_fGFnvnEcrt... FAILED tests/func/test_import.py::test_pull_import_no_download_rev_lock - dvc.exceptions.DownloadError: 1 files failed to download FAILED tests/func/test_import.py::test_pull_imported_directory_stage[dir] - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_import.py::test_pull_imported_directory_stage[dir/] - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir',) FAILED tests/func/test_import.py::test_pull_wildcard_imported_directory_stage - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir123',) FAILED tests/func/test_update.py::test_update_import[True] - FileNotFoundError: [Errno 2] No storage files available: 'version' FAILED tests/func/test_import.py::test_pull_non_workspace - FileNotFoundError: [Errno 2] No storage files available: 'foo' FAILED tests/func/test_update.py::test_update_import_after_remote_updates_to_dvc - FileNotFoundError: [Errno 2] No storage files available: 'version' FAILED tests/func/test_import.py::test_import_with_jobs - dvc_data.index.index.DataIndexDirError: failed to load directory ('dir1',)

rgoya · 2025-04-26T07:05:00Z

AFAIR, Repo(config=...) should just work.
- repo = Repo(url, config={"core": {"no_scm": no_scm_flag}})
+ repo = Repo(url, config=kwargs.get("config"))
I don't want to specialize core.no_scm in any way or handle it.

@skshetry, Ok, I finally got some time to track down the errors.

I took your suggestion of using something closer to Repo(url, config=kwargs.get("config")).

I tracked the errors that were triggered by that call to the way that dvc manages imports. Mainly, it creates a DVCFileSystem for a remote repo using a local cache (see issue #9385 / fix #9423 ). This means that when _get_remote_config() is eventually called, it receives the url of the remote repo, but a kwargs["config"]["cache"] that points to the local cache.

Since the function of _get_remote_config() is to get the actual remote's cache location, in order for dvc import to work, _get_remote_config() needs to ignore anything sent in kwargs["config"]["cache"].

I modified the code to to the above, and added some comments to the code for future reference.

I also added a related test to the test suite.

Please let me know what you think.

Note: FYI, I've been seeing some flakiness in the test suite from what seems to be a race condition in the logs, where a message of DEBUG s3fs:utils.py:80 RC: discarding all clients sometimes sneaks in and fails the assert, e.g.:

>       assert first(caplog.messages) == expected_message
E       assert 'RC: discarding all clients' == "Output 'path... 'stage.dvc'."
E         
E         + RC: discarding all clients
E         - Output 'path'(stage: 'stage.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
E         - You can also use `dvc commit stage.dvc` to associate existing 'path' with stage: 'stage.dvc'.

tests/unit/output/test_output.py:106: AssertionError
----------------------------- Captured stderr call -----------------------------
WARNING: Output 'path'(stage: 'stage.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit stage.dvc` to associate existing 'path' with stage: 'stage.dvc'.
------------------------------ Captured log call -------------------------------
DEBUG    s3fs:utils.py:80 RC: discarding all clients
WARNING  dvc.output:output.py:1160 Output 'path'(stage: 'stage.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
You can also use `dvc commit stage.dvc` to associate existing 'path' with stage: 'stage.dvc'.

skshetry · 2025-04-26T14:04:18Z

Great research, @rgoya. I have taken a quick look at the PR and it looks good. Please give me some time (maybe this week) to look at this more closely.

dvc/repo/open_repo.py

skshetry

Thank you for contributing! The way RepoDependency sets the cache looks fragile and we should figure out a better solution in the future. Your current solution is practical and works well within the constraints.

I also appreciate the effort you’ve put into adding detailed comments and a very good test.

That said, I want to note that core.no_scm may not always work reliably in all scenarios. Configuration options like this are contextual, so they might not work on every cases.

I’m happy to approve this. Thank you for your time and effort in improving the project! 🙂

rgoya · 2025-04-30T06:46:42Z

Thank you for contributing! The way RepoDependency sets the cache looks fragile and we should figure out a better solution in the future. Your current solution is practical and works well within the constraints.

I also appreciate the effort you’ve put into adding detailed comments and a very good test.

That said, I want to note that core.no_scm may not always work reliably in all scenarios. Configuration options like this are contextual, so they might not work on every cases.

I’m happy to approve this. Thank you for your time and effort in improving the project! 🙂

Thank you, @skshetry. It's been quite illustrative peeking into the dvc code. Thanks for the heads up on core.no_scm, I'll keep an eye out for that; if something breaks I might come back with another issue ;)

rgoya · 2025-05-05T20:32:08Z

@skshetry, incidentally, do you have a defined release schedule? (Just wondering when this change would make it into my conda-environment)

skshetry · 2025-05-06T02:27:56Z

@skshetry, incidentally, do you have a defined release schedule? (Just wondering when this change would make it into my conda-environment)

Hopefully, by Wednesday. If not, then I'll release by the end of next week (I am out till early next week).

rgoya · 2025-05-06T02:50:22Z

@skshetry, incidentally, do you have a defined release schedule? (Just wondering when this change would make it into my conda-environment)

Hopefully, by Wednesday. If not, then I'll release by the end of next week (I am out till early next week).

Alright! I'll keep an eye out for it. Thanks!

This `_get_remote_config()` should use `uninitialized=True` so that it supports more broader kinds of broken/partially-initialized repositories, including some that have `.dvc` directory missing, or `.git` directory missing. This partially reverts #10719. #10608 is also fixed, and no longer requires `core.no_scm` to be passed. This was already supported by `dvc.api.get_url` as it uses `uninitialized=True`, but this was not respected in `_get_remote_config()` where it would fail before. That said, this whole `open_repo`/`_get_remote_config` is terribly broken. For one, it is opening a local repository, and forcing it's remote config to a repository opened with `Repo(rev=...)`, where the config may be different.

rgoya mentioned this pull request Apr 10, 2025

dvc.api.get_url(): ignores request to avoid source control and crashes #10608

Closed

rgoya commented Apr 10, 2025

View reviewed changes

skshetry reviewed Apr 10, 2025

View reviewed changes

rgoya requested a review from skshetry April 10, 2025 07:51

Forward args to _get_remote_config and honour core/no_scm if present

f7a3308

rgoya force-pushed the respect_no_scm_in_open_repo_call branch 2 times, most recently from 91cf291 to f598ecd Compare April 26, 2025 06:38

rgoya added 5 commits April 26, 2025 00:02

Pass full config, except cache, to Repo()

30bb3ad

Avoid modifying original kwargs contents

5527042

Simplify if config

52c0f09

Add test

0af2876

Relax error matching.. .. Avoid windows vs unix paths

652560a

rgoya force-pushed the respect_no_scm_in_open_repo_call branch from f598ecd to 652560a Compare April 26, 2025 07:03

skshetry reviewed Apr 30, 2025

View reviewed changes

dvc/repo/open_repo.py Outdated Show resolved Hide resolved

Update dvc/repo/open_repo.py

0712ad2

skshetry approved these changes Apr 30, 2025

View reviewed changes

skshetry merged commit 4bde0e7 into iterative:main Apr 30, 2025
36 of 41 checks passed

skshetry mentioned this pull request May 6, 2025

repo: try to support opening uninitialized/broken repositories #10736

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Forward args to _get_remote_config() and honour core/no_scm if present #10719

Forward args to _get_remote_config() and honour core/no_scm if present #10719

Uh oh!

rgoya commented Apr 10, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 10, 2025 •

edited

Loading

Uh oh!

rgoya Apr 10, 2025

Uh oh!

skshetry Apr 10, 2025

Uh oh!

rgoya Apr 10, 2025 •

edited

Loading

Uh oh!

rgoya commented Apr 26, 2025 •

edited

Loading

Uh oh!

skshetry commented Apr 26, 2025

Uh oh!

Uh oh!

skshetry left a comment

Uh oh!

Uh oh!

rgoya commented Apr 30, 2025

Uh oh!

rgoya commented May 5, 2025

Uh oh!

skshetry commented May 6, 2025

Uh oh!

rgoya commented May 6, 2025

Uh oh!

Uh oh!

	repo = Repo(url, config={"core": {"no_scm": no_scm_flag}})
	repo = Repo(url, config=kwargs.get("config"))

Forward args to _get_remote_config() and honour core/no_scm if present #10719

Forward args to _get_remote_config() and honour core/no_scm if present #10719

Uh oh!

Conversation

rgoya commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rgoya Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

skshetry Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

rgoya Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgoya commented Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skshetry commented Apr 26, 2025

Uh oh!

Uh oh!

skshetry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rgoya commented Apr 30, 2025

Uh oh!

rgoya commented May 5, 2025

Uh oh!

skshetry commented May 6, 2025

Uh oh!

rgoya commented May 6, 2025

Uh oh!

Uh oh!

rgoya commented Apr 10, 2025 •

edited

Loading

codecov bot commented Apr 10, 2025 •

edited

Loading

rgoya Apr 10, 2025 •

edited

Loading

rgoya commented Apr 26, 2025 •

edited

Loading