Difficulty running the rtsload tool

[alexwillingham98]

I keep getting errors when trying to load csv data via the rtsload tool. When I run this command:

cs_tools tools rtsload file --target_database SNOWFLAKE_POC --target_table ts_user_49014588-20e7-45a2-8e2e-488229f1cb3e ts_user.csv --config TS_Prod --has_header_row --empty_target

I keep getting:

WARNING Could not connect to your ThoughtSpot cluster: [WinError 10035] A non-blocking _rest_api_v1.py:90
socket operation could not be completed immediately

When I add a timeout option of 100:

WARNING Could not connect to your ThoughtSpot cluster: [WinError 10060] A connection _rest_api_v1.py:90
attempt failed because the connected party did not properly respond after a
period of time, or established connection failed because connected host has
failed to respond

Sorry for all the questions!

[billdback-ts]

Hi Alex. Thanks for the question. First off, can you clarify what you are attempting to accomplish? Is this uploading to tables in Snowflake? This tool will only upload data to Falcon tables (i.e. inside ThoughtSpot), which is a feature on software (non-cloud) version.

If this is a Falcon table on software, then please check your network connectivity. Usually, timeouts like this are firewall issues. Maybe try pinging the server from the command line.

[alexwillingham98]

Did not realize this was specific to the software version. What I'm trying to do is set up an automatic command that will upload the contents of searchable gather on a daily basis without having to manually run the command and upload the csv files to thoughtspot. The reason im using gather is to get the user and group information.

[billdback-ts]

OK, so you are using searchable gather and want that to be in Snowflake. Here are the recommended steps:

1. Create a database in Snowflake to load to.
2. Cerate a connection (without tables) that is connected to the database.
3. Create a syncer that is configured for the snowflake database that you want to use (see https://thoughtspot.github.io/cs_tools/syncer/snowflake/).
4. Now you will run searchable gather and specify the syncer to be the one you created for Snowflake. Gather will load the data directly to Snowflake.
5. Finally, run deploy (this is a one-time activity).

For example (your values will differ):

I have a syncer like the following:

[configuration]
snowflake_account_identifier = 'a_snowflake_partner'
username = 'user'
password = 'password123'
warehouse = 'MY_DW'
role = 'MY_ROLE'
database = 'MY_DB'
schema_ = 'CSTOOLS'
auth_type = 'local'
truncate_on_load = false

Then I run a command to gather: cstools tools searchable gather --include-column-access --syncer snowflake:///Users/bill.back/ThoughtSpot/embed1/searchable/syncers/cstools.snowflake.truncate.toml --config embed1

Finally I deploy: cstools tools searchable deploy --connection-guid 245e515b-e04b-4bfb-99c6-86c2777f5abe --database BDB --schema CSTOOLS --config embed1

[alexwillingham98]

This is great info thank you! Will gather need to be run again continuously to update the TS metadata to snowflake?

[DevinMcPherson-TS]

@alexwillingham98 yes, gather will create a snapshot of the metadata at that moment. Most organizations schedule a job to update this daily. If you want to be able to use the SpotApp and join to usage data, you'll have to do extracts of the bi server data daily as well (with a different syncer that is set to append as opposed to truncate/reload). Example
cs_tools tools searchable bi-server --syncer snowflake:///Users/bill.back/ThoughtSpot/embed1/searchable/syncers/cstools.snowflake.append.toml --config embed1 --from-date 2022-01-14 --to-date 2023-01-15