[NO TICKET] Improving logging of Azure e2e test and increase WDS timeout

[cahrens]

I have noticed that the message being passed to withClue doesn't appear in the test execution log in the case of these tests. Therefore I created a different method that logs the message (instead of just appending it).

And I am increasing the timeout for WDS to be out of PROVISIONING to 15 minutes; it's not clear if this will help at all with the sporadic "over 10 minute" failures we have seen (it's possible that WDS is in a state will it will never succeed), but we can at least get more data.

I have run this workflow, both with a short timeout to ensure that the log message does indeed print in the log (see screenshot below), and with the final version to ensure everything passes.

[cahrens]

jenkins retest

[aherbst-broad]

Very curious why withClue isn't doing what we want here, but this seems like a reasonable compromise.

[cahrens]

@aherbst-broad from what I read, withClue appends the message to the stack trace information. But I don't think we see the stack trace from a test failure in the execution log in these tests in the same way that we would if the test was being executed as a normal unittest.

Also, logging it in this way keeps the message in synchronous order with the other things that are being printed out.

[cahrens]

@aherbst-broad I think it may be because we catch the timeout exception and attempt workspace deletion (which then fails)… so the timeout isn't what causes the test to fail. Perhaps withClue doesn't really need to be replaced throughout, but I do like having the messaging appear synchronously.

Example where we don't see "WDS did not become deletable within the timeout period of…": https://github.com/broadinstitute/terra-github-workflows/actions/runs/7565178023/job/20600517481