cs_tools stopped working in docker

[jerreith]

as of today the cs_tools stopped working in docker.

First the build failed, I had to add "RUN ln -s /usr/bin/python3 /usr/bin/python" to the Dockerfile to get the install working again.

Now the "CMD /root/.config/cs_tools/.cs_tools/bin/cs_tools self info --anonymous" fails with below error:

File "/root/.config/cs_tools/.cs_tools/bin/cs_tools", line 5, in
from cs_tools.cli.commands.main import run
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/commands/main.py", line 11, in
from cs_tools.cli import _analytics
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/_analytics.py", line 26, in
from cs_tools.cli.ux import rich_console
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/ux.py", line 12, in
from cs_tools.cli.types import SyncerProtocolType
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/types.py", line 16, in
from cs_tools.cli.dependencies.syncer import DSyncer
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/dependencies/init.py", line 1, in
from .thoughtspot import thoughtspot
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/cli/dependencies/thoughtspot.py", line 14, in
from cs_tools.settings import (
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/settings.py", line 61, in
class MetaConfig(_GlobalModel):
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/settings.py", line 68, in MetaConfig
environment: Optional[ExecutionEnvironment] = ExecutionEnvironment()
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/pydantic_settings/main.py", line 84, in init
super().init(
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/pydantic/main.py", line 171, in init
self.pydantic_validator.validate_python(data, self_instance=self)
File "/root/.config/cs_tools/.cs_tools/lib/python3.10/site-packages/cs_tools/utils.py", line 194, in determine_editable_install
for path in pathlib.Path(site_directory).iterdir():
File "/usr/lib/python3.10/pathlib.py", line 1017, in iterdir
for name in self._accessor.listdir(self):
FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/cs_tools/.cs_tools/local/lib/python3.10/dist-packages'

It seems the directory does not exists: ls: cannot access '/root/.config/cs_tools/.cs_tools/local': No such file or directory

Any idea?

Thanks,

Jeroen

[boonhapus]

Hey @jerreith , can you share your docker file? I'm surprised that site would return an invalid path at all. I might have to loosen this check.

[jerreith]

Here is the Dockerfile:

FROM ubuntu
WORKDIR /app
RUN apt update && apt install -y locate && apt install -y curl && apt install -y python3 && apt install -y python3-venv
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN curl --silent --show-error --location-trusted https://raw.githubusercontent.com/thoughtspot/cs_tools/master/cs_tools/updater/_bootstrapper.py | python3 - --install
#CMD updatedb && locate bin/cs_tools
CMD /root/.config/cs_tools/.cs_tools/bin/cs_tools self info --anonymous

[boonhapus]

ahhh, this actually makes sense, thanks @jerreith. Docker's base ubuntu is a slim version, which doesn't include many of the guarantees you'd normally get when creating a virtual environment via python's native distrib. I'll roll this fix into the next version early this week.

[jerreith]

It did work last week so something changed...

Please let me know when a new version is available.

Thank you!

[boonhapus]

@jerreith were you already on the 1.5.0 series of releases? Your docker file likely rebuilds the environment each time you spin up a new container. This is likely because RUN curl --silent --show-error --location-trusted https://raw.githubusercontent.com/thoughtspot/cs_tools/master/cs_tools/updater/_bootstrapper.py | python3 - --install is non-deterministic from the perspective of the container.

You might consider taking this approach instead now that we have the latest version of the tools with better support for serverless environments.

[jerreith]

I did a docker build with --no-cache so it pulled the source.

Let me check the approach you suggested.

Thanks

[jerreith]

Is there already a fix for the "FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/cs_tools/.cs_tools/local/lib/python3.10/dist-packages'" issue?

I cannot run cs_tools anymore from the cmdline after I upgraded.

[RafRoc]

We face the same issue / error when running the bash:

sudo /home/vsts/.config/cs_tools/.cs_tools/bin/cs_tools config create \
  --...

It used to work until the 24/03/2024.

[boonhapus]

@RafRoc can you share how you installed? The latest 1.5.3 includes some fixes for the issues we were seeing on bootstrapping the environment. Hopefully it fixes the issue you're seeing as well.

[RafRoc]

@boonhapus It is installed by running:

curl --show-error --location-trusted https://raw.githubusercontent.com/thoughtspot/cs_tools/master/cs_tools/updater/_bootstrapper.py | python3 - --install

on a linux-machine.

Unfortunately it is still not working. Should we look for a different approach or do you think this can be fixed?

[boonhapus]

@RafRoc can you email me? I'd like to find a time to meet live and see if we can observe together or fix it.

[email protected]

[jerreith]

Ok, I also see this one in the release notes for 1.5.0: "temporarily removed BigQuery Database Syncer".

Any idea when that returns? We can not use it without the BigQuery syncer.

Thanks.

[boonhapus]

Is this still your whole docker file?

#122 (comment)

[jerreith]

I am running this in the google cloud shell, just the cmd-line not the docker. I reinstalled cs_tools with the curl command as per the documentation.

[jerreith]

Is there any update on the error reported and the bigquery syncer reactivation?

Thanks.

[boonhapus]

Hey @jerreith sorry I keep missing your replies. GH only send notifications on mentions or new threads. Can we meet directly to figure out the GCP issue? If you shoot me an email we can find a time that works best in your tz.

[email protected]

I don't have an update on bq syncer yet, but have some time blocked off late this week to reimplement it and a postgres syncer, so hoping to have these ready by next week!

[boonhapus]

@jerreith I've completed the bq syncer, but I'd like to understand why you're unable to install. Can we find a time to meet together? Please shoot me an email above so we can find a good time. 👆