Implement pagination of output for the api/allocations endpoint

[QuanMPhm]

While interacting with the api/allocations endpoint for the NERC Coldfront server in our production NERC cluster, it was observed that occasionally a time out error would be raised. We believe this time out is caused by the fact that the output list is becoming too long. A solution to this would be to implement pagination.

@knikolla Should I worry with implementing pagination for the SCIM endpoints as well?

[knikolla]

@QuanMPhm Pagination is not the only solution. There's also the possibility of streaming the response, and there's the possibility of having the client of sending a new request for each of the difference resource allocations using filtering, ex. return the resource allocation that has an attribute of Allocated Project ID of XYZ (this we already support in coldfront).

Do a little bit of investigation and proof of concepting and come back to me with a proposal of what you think would work best.

[QuanMPhm]

@knikolla From what I've found so far, Django Rest Framework has a built-in solution for pagination. I have tested their pagination feature, and it works as expected. The code changes to coldfront-plugin-api would only be minimal (~5 lines). The code changes required by our billing scripts and other services that fetches from the coldfront API should also be fairly small.

DRF does not have a built-in solution for streaming. I would have to use Django's StreamingHTTPResponse. Streaming would require custom code to divide the massive json list into chunk, along with some extra code to make sure StreamingHTTPResponse integrates nicely with DRF. The upside is that since streaming would return just a JSON as before, we won't need to make code changes to our other services.

I suppose whether which solution is better depends on how many services we expect will use our coldfront plugin api in the future. If there's alot, it might be worth the extra coding to implement streaming over pagination.

[QuanMPhm]

From @knikolla's suggestion in the 2:30 yesterday, we believe it is best to update how our services are querying the allocations API, rather than implementing features to prevent timeout. Instead of querying for all allocations, our services should use filters to limit the number of allocations they are searching for.

I will write up the appropriate issues and close this.