Integrating Parallelism Support into dotcom-toolkit

[Nazehs]

I have a proposal that I have drafted here for toolkit to support circleCI features that will improve the circleCI execution time

Detailed Proposal is here

[ivomurrell]

I'm sceptical that adding support for CircleCI's parallelism parameter makes sense in the general case. It will add a significant amount of complexity to both Tool Kit installation and execution. For installation, it will require updating all parallel-enabled jobs to run circleci tests instead of Tool Kit, with specific commands to pipe the input files to the process. For execution, we'd need to work out a way to handle Tool Kit commands with multiple tasks being split across machines. Additionally, I'm not sure if there's much benefit to be gained from distributing linting and build tasks – the former rarely takes any time to complete, and the latter is hard to distribute (neither Webpack nor TypeScript would support such decomposition.) I don't think we should be particularly concerned about resource class scalability right now in the general case as I'm not aware of any projects using greater than the base xlarge class, with most projects using the default medium (even big projects like next-article!) So adding support for parallelism feels like a premature optimisation when we still have so much more room to scale up single machines for CPU-bound tasks.

Therefore, I'd suggest this work be focussed specifically on the case of end-to-end testing. Like you say, the Cypress docs explicitly recommend such an approach, and natively support it. Cypress tasks tend to already be run in a separate command so we don't need to worry about splitting tasks up. They would also be the ones that stand to benefit the most from distribution. We also have quite a bit of custom code in place to support Cypress (as it requires a specific executor to run in.) I think adding custom, targeted support for parallelism in the cypress plugin would provide us all the benefits of distribution whilst significantly minimising the incurred complexity. What do you think?

[Nazehs]

Yeah I agree I think this does make sense to me too. We can go with what will be the best for now.