The current fastpath queue has the following issues:
- Loaded in memory in the same fastpath process:
- Measurements in queue are lost on reboots to fastpath
- Can't significantly increase memory buffer for enqueued measurements: current max size is 500 measurements, ~10s of current traffic if we assume a worst case of 50 msm/s.
- Measurements lost from the queue are lost forever. Since they were already sent to the fastpath, ooniprobe will not upload the measurement to s3 and since fastpath is shutting down, it has no chance to back up the queue
- Since the queue has a very small max size, it's very sensible to workers slowing down, mostly due to Clickhouse slowing down
- When Clickhouse goes down, we start losing measurements immediately
- We would like to have more buffer for errors to have more time to address an incident or performance issues
We have to replace the in memory python queue with a better queue implementation. Ideally such implementation should have the following features:
- Persistent storage: The queue content should be persistent between reboots
- A different process: We should be able to reboot the fastpath without losing the queue content
- Parallel access: We have one producer (the http feeder) and multiple parallel consumers (
msm_processor workers)
- A configurable max size: To avoid filling the entire disk
- Good observability: To set up alerts when the queue is unusually filled
- Simple implementation: to avoid doing significant changes to the fastpath
- (bonus) reliability: It would be even better if the queue has some way to recover measurements that failed to be processed. For example if a worker crashes without saving the measurement in DB
The current fastpath queue has the following issues:
We have to replace the in memory python queue with a better queue implementation. Ideally such implementation should have the following features:
msm_processorworkers)