Request timed out causing meteor instances to be restarted

[kaihendry]

https://media.dev.unee-t.com/2019-04-09/request-timed-out.mp4

Meteor instances in production seem to get stuck and fail to respond to health checks.

[kaihendry]

@nbiton I'm pretty confident it appears to go down (become unresponsive) in one of those repeated bugzilla ajax retries like mentioned upon #699

Actually I am not sure anymore since the log times and the event don't line up exact.

[kaihendry]

We need to improve logging since production doesn't appear to log requests right now. Did we pull out of #631 on prod? Wondering why the logs are not on one line.

https://media.dev.unee-t.com/2019-04-09/case-dev-vs-prod-logging.mp4

[kaihendry]

If NodeJS locks up and gets killed, there is a some time before it comes back online. This could be expedited by #750 since this proposal would avoid the npm install stage in our current Docker image.

[kaihendry]

We reproduced the problem locally in the sense we see Meteor taking a very long time to process a HTTP.call

I20190412-10:35:46.717(8)? { statusCode: 200,
I20190412-10:35:46.717(8)?   method: 'get',
I20190412-10:35:46.718(8)?   endpoint: '/rest/bug/73397/comment',
I20190412-10:35:46.718(8)?   payload: { api_key: 'secret' },
I20190412-10:35:46.718(8)?   duration: 74517 }
I20190412-10:35:46.722(8)? { statusCode: 200,
I20190412-10:35:46.722(8)?   method: 'get',
I20190412-10:35:46.723(8)?   endpoint: '/rest/bug/73397',
I20190412-10:35:46.723(8)?   payload: { api_key: 'secret' },
I20190412-10:35:46.723(8)?   duration: 74525 }

MEFE says 74s and Bugzilla below says half a second. This is the crux of the problem.

bugzilla_1   | 172.24.0.1 - - [12/Apr/2019:02:25:05 +0000] "GET /rest/bug/70142/comment?api_key=secret HTTP/1.
1" 200 1352 500 "-" "-"
bugzilla_1   | 172.24.0.1 - - [12/Apr/2019:02:25:05 +0000] "GET /rest/bug/70142?api_key=secret HTTP/1.1" 200 3
239 585 "-" "-"

[kaihendry]

Since getting things running locally, we know it's not a DevOps issue. It appears to be something to do with how caseNotifications are read causing a huge delay. @nbiton is working on a fix.

[franck-boullier]

this shoudl now be fixed thatnks to throttling and the implementation of unee-t/bz-database#129