Using CS Tools as a library to get dependencies.

[billdback-ts]

Overview

As of v1.4, CS Tools can be used as a library to make it easy to write your own applications. One app that I've had come up several times is the ability to get dependencies in a spreadsheet format. Right now, there isn't a tool that provides such a capability, but it's easy to write with cstools. In this example, we will create a script that will get the tables and worksheets along with their dependencies and write to a CSV file.

Setup

Create a new project with a venv (highly recommended).

Next, install cs_tools as a library into your environment pip install cs_tools @ https://github.com/thoughtspot/cs_tools/archive/v1.4.0.zip. Note that this will get a bit simpler in the future. Substitute a more recent version as appropriate.

That's it other than libraries you may need for your own code.

Config

One nice thing about using cstools is that you can create and test a config file for your code. Run cstools config create --config <name> and create a new TOML file. Then run cstools config show to see the names and locations of the config file. The actual file will be named something like cluster-cfg_<myconfig>.toml, but you can rename it. Move the file to a convenient location for your script.

The code

The actual code is pretty straightforward.

import csv
import pathlib

from cs_tools.settings import CSToolsConfig
from cs_tools.thoughtspot import ThoughtSpot

from cs_tools.types import MetadataObjectType, MetadataObjectSubtype

cfgName = "./my_config.toml". # replace with your config file.
cfg = CSToolsConfig.from_toml(fp=pathlib.Path(cfgName))  # path to your config file

ts = ThoughtSpot(cfg)
ts.login()

# ts.org.switch(0). # if using an instance with orgs enable and set to the desired org.  0 is Primary.

# Get all the tables in the system that come from an external source and are not owned by TS.
r = ts.api.metadata_list(metadata_type=MetadataObjectType.logical_table.value,
                         subtypes=[MetadataObjectSubtype.system_table.value])
tables = [t for t in r.json()["headers"] if t["isExternal"] and t.get("authorName", "") != "system"]

# Also get the worksheets not owned by TS.
r = ts.api.metadata_list(metadata_type=MetadataObjectType.logical_table.value,
                         subtypes=[MetadataObjectSubtype.worksheet.value])
worksheets = [w for w in r.json()["headers"] if w.get("authorName") != "system"]

# Get all the dependencies for each table and add them to a CSV file.
with open("dependents.csv", mode="w", newline="") as csv_file:
    # Create a CSV writer object
    writer = csv.writer(csv_file)
    writer.writerow(['table type', 'table name', 'table guid', 'dependent type', 'dependent name', 'dependent guid'])

    # Get all the dependents and write to the CSV file.
    for t in tables:
        print(f"Getting dependents for {t['name']}")
        dependents = ts.api.dependency_list_dependents(guids=[t["id"]],
                                                       metadata_type=MetadataObjectType.logical_table).json()
        for dependent_list in dependents.values():
            for type in dependent_list.keys():
                for content in dependent_list[type]:
                    content_type = content.get("type", type)
                    dependent_row = ['table', t["name"],t["id"],content_type,content["name"],content["id"]]
                    writer.writerow(dependent_row)

    # get the worksheet dependents and write them as well.
    for w in worksheets:
        print(f"Getting dependents for {w['name']}")
        dependents = ts.api.dependency_list_dependents(guids=[w["id"]],
                                                       metadata_type=MetadataObjectType.logical_table).json()
        for dependent_list in dependents.values():
            for type in dependent_list.keys():
                for content in dependent_list[type]:
                    content_type = content.get("type", type)
                    dependent_row = ['worksheet', t["name"],t["id"],content_type,content["name"],content["id"]]
                    writer.writerow(dependent_row)

That's it. In less than sixty lines of code, we are able to get all of the objects and their dependencies and put them in CSV file that can be opened in Excel or other application.