cs_tools tools scriptability --metadata-types LIVEBOARD may have caused system to become unresponsive

[bmofkn]

@boonhapus, this may not be related and I have created a TS Support Ticket to get SRE to look into it... so it might just be coincidental.

We were running the following v1.6.2 script:
cs_tools tools scriptability export --directory ./ --metadata-types LIVEBOARD --include-system --log-errors --config prod_ts

It failed with:
[DEBUG - 2025-03-19T17:21:26-0600] [cs_tools.cli.commands.main - main.run 110] whoopsie, something went wrong!
httpx.ReadTimeout

At that moment, our entire cluster went offline and was unresponsive for a few minutes. Almost as if the cluster rebooted. A subsequent command responded with:
[17:23:45] WARNING HTTP 502 for https://<redacted>.thoughtspot.cloud/api/rest/2.0/auth/session/login on attempt 1, retrying in 60.0s

By [2025-03-19T17:25:09-0600] both the subsequent scripts and the web interface were working again. It's possible the outage was unrelated to the script... just thought I'd post in the event this triggered anything on your end or perhaps you can point out if I'm using the command inaccurately. We use this script everyday to backup the TML for all objects in our cluster.

Thanks!

[boonhapus]

@bmofkn definitely let's follow that support ticket. There's only two key differences between v1.5.x and v1.6.x on CS Tools.

1. We've migrated to V2 APIs only in 1.6
2. We now run API calls concurrently.. standard operations have a limiter of 15 concurrent calls while TML operations are capped at 4 concurrent operations.

That said.. both of these limits are well tested and shouldn't cause performance degradation. What you're seeing is strange. My guess is that your cluster was already approaching some thresholds, and the added workload that tml/export represents caused an internal service failure or restart. ThoughtSpot support can help tune memory limits or upsize the cluster based on this.

Can you provide the times to support? New logs should all record TZ as well so it can help them figure out what service needs a bit of tuning.

gonna tag @DevinMcPherson-TS (re: internal cs comms) @billdback-ts (tse scale) as well to make sure they can run it with the right folks as well.

[bmofkn]

Thanks @boonhapus. The ticket is Case # 00389049. They said tomcat crashed around the same time the script failed.

This may be related: About a week ago TS Support “identified optimisation changes that can be implemented to improve memory utilisation”.

They applied those changes on March 13, 2025. Could be related 🤷‍♂️

[boonhapus]

@bmofkn There's an internal epic tracking some performance related optimizations, so the intentionally vague comments from support do make sense.

Now that CS Tools is fully on V2 apis, TS is seeing some complex REST API patterns implemented across lots of clusters (over 150 customers use searchable and scriptability!) "in the wild" so to speak. Most customers won't get as advanced as the CS Tools flows. 😄

[bmofkn]

@boonhapus, FYI, we ran seven v1.6.2 scripts this morning (covering seven metadata-types) and it ran without failure. No crashes on our cluster 👍. Thanks for all of your support over the years. We love cs_tools ❤️