2025-10-28 CMS Admin not accessible

[saumier]

Incident Report

Summary

CMS Admin is not available in production or staging. Customer websites using Footlight Open API are all OK. This is only impacting access to CMS Admin. @Suhail Aliyar and @gregory are working to fix this. There was a big change today in our hosting setup as we switched to CloudFlare. This triggered the problems.

Timeline

2025-10-28 13:15

[P2] Triggered: cms-login(staging)

2025-10-28 14:13

[P1] Triggered: cms-login(production)

Investigation starts.

@saumier notices that http is working but not https.

@saumier Looks at the Cloudflare DNS and notices warning on CNAME that have sub.subdomain domains like api.cms.footlight.io and staging.api.footlight.io.

@saumier Reads documentation on Cloudflare SSL. Learns that SSL is being handled by Cloudflare BUT only *.footlight.io, footlight.io are covered. This means api.cms.footlight.io is NOT.

@saumier Purchases a paid feature in CloudFlare called Total TLS that costs 10$ per month.

Total TLS : Cloudflare will issue individual certificates for every proxied hostname. Once enabled, all hostnames in your domain will be covered by a TLS certificate.

This issues SSL certificates for all our domains including the ones with sub.subdomain that were not initially covered.

The systems returned to normal.

Next Steps

• Study the options for SSL management on Cloudflare
• Configure our SSL setup so we manage certificates in a single place (move all to Cloudflare or remove Cloudflare)
• Study feasibility of changing api.cms.footlight.io to api-cms.footlight.io to remove sub.subdomain. Same for all staging?
• Cancel paid server for Total TLS on Cloudflare if possible.

[saumier]

Related issue:

• URGENT: cannot log into the production CMS footlight-app#2017

[saumier]

@sahalali Please look at the next steps. You can add/edit the ones I wrote. Please convert the ones you start working on into issues (click the 3 dots and convert to issue).