Speed up test execution for non-cached tests

[gladyshcodes]

What

Speed up test execution by finding ways to addressing issues outlined below.

Why

While working on #179, I have found that screenshooting perhaps takes the most time when test runs in --no-cache mode. Sometimes, screenshots are taken several times when there's no need for it. Also, delay before making a screenshot is about a second or so.

[m2rads]

That is a good observation and I here is my plan to speed it up. Here's an outline of what I have in mind:

• Instead of saving screenshots in the folder, we can save it in the memory thus making efficient use of space and
• Adding screenshots automatically after executing an action. (Currently we wait for AI to instruct us when to take sc)
• Performing multiple actions per screenshot whenever possible. (Not sure if this is possible with Computer use API yet)

I think implementing these simple changes should speed up the AI execution by 2x at least.

@slavingia, @gladyshcodes Wdyt?

[slavingia]

Makes sense. I pinged Anthropic to see if they'd support multiple actions in one step.

[gladyshcodes]

Makes sense. I pinged Anthropic to see if they'd support multiple actions in one step.

Have you received a callback from Anthropic yet?

[slavingia]

Not yet, will bump

[Shawns2759]

The executions are already pretty expensive. Do we have ways to cut down on cost as well as speed up executions?

[slavingia]

We should probably tackle #187 first, to see that, and then evaluate. Anything that caches computer use should help.

[gladyshcodes]

With this, we can debug things faster and measure results

Recently we introduced caching #179 that made test execution about 6 times faster. I have several more ideas in mind:

• Running tests in parallel. Similarly to how Jest does that. This can skyrocket exec time
• Running 'pre-validation' phase makes an initial LLM request to evaluate the test suite and answer questions like: do we need to lunch chromium? and what tests can be ran in parallel? letting the LLM decide the most efficient execution order itself
  • For example, the new Bash tool API tests (Bash tool #233) we’re rolling out soon—most of them can be run in parallel.
  • The initial LLM request may increase input tokens count, increasing costs but there may be a way to round it. Maybe leverage
    LLM memory and sending all tests at once and then referencing them by name to start execution
• Batching multiple requests into a single LLM call could also help.

Hoping quota of LLM providers will decrease over time (similar to how the price of GFLOPS or disk space has dropped, making this tool more affordable for everyone

[slavingia]

Batching multiple requests into a single LLM call could also help.

This will be huge and eventually happen.

Running tests in parallel

This seems like relatively low-hanging fruit to explore. In theory a server could run one browser for every test (just keeping in mind chaining/caching) that needs be run, and the entire test suite should just take as long as the slowest chain of tests.

[PedroAVJ]

Wouldn't running tests in parallel run into rate limit issues, and in turn, make null the speed gains? I suppose it depends partly on the API key tier, but when I ran the original claude computer use demo I would constantly get rate limited

[slavingia]

Wouldn't running tests in parallel run into rate limit issues, and in turn, make null the speed gains? I suppose it depends partly on the API key tier, but when I ran the original claude computer use demo I would constantly get rate limited

Things may have changed, but overall you're right it'll be a bottleneck. I'll bring it up with them!