AJ-1347: entityQuery streams entities

[davidangb]

Ticket: https://broadworkbench.atlassian.net/browse/AJ-1347

Moves the entityQuery API to a streaming model, in an effort to reduce large memory spikes when users hit this API hard.

Manually tested in a BEE, above and beyond the unit/swatomation tests.

Also, I inspected the response headers when running this branch vs. hitting dev directly. When running this branch, I see an extra Transfer-Encoding: chunked header, indicating a streaming response.

High-Level Design

At a high level, the old code flow was:
route definition in EntityApiService -> EntityService.queryEntities -> LocalEntityProvider.queryEntities
this returned a fully materialized EntityQueryResponse, which was serialized to JSON and sent to the user.

The new code flow is:
route definition in EntityApiService -> EntityService.queryEntitiesSource -> LocalEntityProvider.queryEntitiesSource
this returns a tuple of (EntityQueryResultMetadata, Source[Entity]). The EntityQueryResultMetadata is fully materialized, but the Source[Entity] is a stream. This tuple is processed through EntityStreamingUtils.createResponseSource to incrementally send response bytes to the user.

In this PR:

• EntityStreamingUtils.createResponseSource is new
• multiple methods were moved to make them composable and accessible to calling code
• multiple db-related methods were tweaked to return SqlStreamingAction instead of ReadAction
• The EntityStreamingUtils.gatherEntities method is extracted out of EntityService and modified to allow results which are not sorted strictly by entity id. This method still requires all rows of a given entity id to be contiguous in the result set - but they do not have to be sorted purely ascending or descending.

---

PR checklist

• Include the JIRA issue number in the PR description and title
• Make sure Swagger is updated if API changes
  • ...and Orchestration's Swagger too!
• If you changed anything in model/, then you should publish a new official rawls-model and update rawls-model in Orchestration's dependencies.
• Get two thumbsworth of PR review
• Verify all tests go green, including CI tests
• Squash commits and merge to develop (branches are automatically deleted after merging)
• Inform other teams of any substantial changes via Slack and/or email

[davidangb]

I found some tests in this class were timing out intermittently; it was using the default timeout of 150ms. Increasing the timeout made it stable. The changes to streaming likely tweaked timing.

[jladieu]

Confirmed this looks like Scala. 😈

[dvoet]

This probably works but the structure is confusing. I think there are actually 2 separate transactions:

1. calls dataAccess.entityQuery.loadEntityPageCounts
2. calls dataAccess.entityQuery.loadEntityPageSource

Structurally, it looks like 2 is within the transaction that includes 1 but it is not. Yet 2 uses information from 1 before 1 completes. I would like to see this broken up a little more cleanly.

[davidangb]

Two 504 Gateway Time-out errors in swatomation; I believe these are unrelated. Jenkins retest.

[dvoet]

This tracing does not encompass streaming the results, it probably just encompasses up until the start of streaming. Probably not much to do about it right now, I am just calling it out.