Add GitHub Actions health check workflow

[KevinMind]

Fixes: mozilla/addons#15429

Description

Introduces a new GitHub Actions workflow to periodically check service health and version information for specified endpoints. The workflow includes:

• Scheduled runs every 5 minutes
• Checks version information
• Monitors service status
• Raises an error if any services are down

Context

This PR adds the workflow for checking the health of our services/dependencies. A future PR will integrate a slack notification when things are found

Testing

Running the workflow

You can run the workflow manually via bash (example here)

gh --ref kevinmind/addons/15308 workflow run

Select "Health Check" and then visit https://github.com/mozilla/addons-server/actions/workflows/health_check.yml to see your run

Running locally

To inspect the logic of the health check script you can run it manually passing in an environment to ping.

Note

You should do this inside a running web container if you don't want to have to install dependencies manually on your host

./scripts/health_check.py --env prod

Warning

Passing no environment should error

Checking for failure

You can manually run the bash script the healthcheck runs and modify the monitors json to test for error state.

• first run the python script with the below script:

./scripts/health_check.py --verbose

• in the monitors, set at least one service "state" to false and add a status message.
• manually run the logic in the bash script and expect errors to be printed out

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[KevinMind]

@diox should be ready to verify.

[diox]

While the run I just did is marked as successful if you dig through the logs it raised a 404 and logged an error:

Requesting https://addons-dev.allizom.org/services/monitors.json for dev
{'error': JSONDecodeError('Expecting value: line 1 column 1 (char 0)'), 'data': None, 'response': <Response [404]>}

[KevinMind]

While the run I just did is marked as successful if you dig through the logs it raised a 404 and logged an error:

Requesting https://addons-dev.allizom.org/services/monitors.json for dev
{'error': JSONDecodeError('Expecting value: line 1 column 1 (char 0)'), 'data': None, 'response': <Response [404]>}

typo in the URL its services/monitor.json

[KevinMind]

@diox the issue was in the retry mechanism in the healtcheck script. We were not raising on the final attempt. That is fixed now and I've verified it exits 1 with the incorrect URL. I've also fixed the URL and the related tests.