[Feature] Don't Persist Inflator/Indexer Containers

[techman83]

The Inflator + Indexer often sit around doing nothing, querying the SQS queues every 20 seconds generates for both services potentially 6,220,800 messages a month. Whilst that's only a few dollars (and we're saving more than this by only needing a micro vs a small instance), we can do better.

Suggestions

• Indexer/Inflator after X consecutive time without receiving a message terminate (configurable with an optional flag. Run forever or --exit-after-mins 10)
• Drop the Service Definitions for the Indexer/Inflator
• Scheduler will call 'run-task' on the indexer/inflator when a run is started

[HebaruSan]

Should we consider setting ReceiveMessageRequest.WaitTimeSeconds to a value higher than 20? I don't see a mention of an upper limit or a downside to high values in the documentation, so it looks like it ought to work:

• https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/SQS/TReceiveMessageRequest.html
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html#sqs-long-polling
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/working-with-messages.html#setting-up-long-polling

[techman83]

Unfortunately 20 seconds is the max wait time. But I'm really liking the idea of only spinning up the containers when we need to. It's obnoxiously fast to do it this way and I have a bunch of ideas around doing stuff like this at work now.

[HebaruSan]

Scheduler will call 'run-task' on the indexer/inflator when a run is started

Would the webhooks also need to do that?

[techman83]

Whatever path we take, it would be a generic call any of the services could call to make it happen.

[techman83]

I should note I was several orders of magnitude off here. We're around 400,000 messages while running the indexer/inflator full time.