Compilation Error "dbt deps" to install package dependencies

[ecarranza-sh]

Hello everyone,

I am getting this error:

[2025-03-28, 15:48:15 UTC] {logging_mixin.py:188} INFO - 15:48:15 Encountered an error: Compilation Error dbt found 2 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.

I am running Cosmos in Composer 2 with this setup:

• ProjectConfig: install_dbt_deps=True
• ProfileConfig: GoogleCloudOauthProfileMapping
• ExecutionConfig: Local
• RenderConfig: DBT_LS

I have tried also to previously install the dependencies first on separate task and I am also getting the same error on some Dags, sometimes after 2 retries it works, but I need it to work the first time.

[dosubot]

I couldn't find specific information on how Cosmos handles the dbt deps command in the context of GoogleCloudOauthProfileMapping or any specific configurations required for running dbt commands in Composer 2. However, there are a few general steps you can take to troubleshoot this issue:

1. Check packages.yml: Ensure that the packages.yml file is correctly configured and accessible in your environment. Verify that the package sources are correct and reachable from your Composer environment.

2. Environment Variables: Make sure that any environment variables required for authentication or accessing the packages are correctly set in your Composer environment.

3. Enable install_deps: Ensure that the install_deps flag is enabled in your Cosmos configuration. This flag is part of the ProjectConfig class and is set to True by default, which should automatically run dbt deps during DAG parsing and task execution ^[1].

4. Logging and Debugging: Check the logs for any additional error messages or information that might indicate what is going wrong. Cosmos logs the output of dbt commands, which can help in identifying the issue ^[2].

If these steps do not resolve the issue, consider checking for any network restrictions or permissions that might be preventing the installation of packages in your Composer environment.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[ecarranza-sh]

@dosu I already did what you mention but still, It fails once, maybe twice but after that it works. I need it to run the dbt deps right the first time. Also one thing to mention is that I am running approximately 600 models with dependencies, Cosmos takes care of figuring out the dependencies between models but that may be a factor in the error I am getting, i don't know.

[dosubot]

To ensure that the dbt deps command runs successfully on the first attempt when using Cosmos in Composer 2, especially with a large number of models, make sure that the install_dbt_deps parameter is set to True in the ProjectConfig class. This setting ensures that dbt deps is executed during DAG parsing and task execution, which is crucial for managing dependencies effectively ^[1].

Additionally, consider leveraging Cosmos's caching mechanism to improve performance. Cosmos provides a caching system that can pre-fetch or cache dependencies, reducing the time taken by dbt deps by using cached dependencies instead of fetching them every time. This can be particularly beneficial when dealing with a large number of models and dependencies ^[2][3].

If the issue persists, it might be worth investigating any network or resource constraints in your Composer environment that could be affecting the execution of dbt deps.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[github-actions]

This issue is stale because it has been open for 30 days with no activity.