Added public incident report for Degradation in email, SMS, and realtime delivery #20
Conversation
WalkthroughThis change adds a new documentation file at 2025/08/24/readme.md detailing an incident affecting Appwrite Cloud from 2025-08-24 to 2025-08-26. It outlines detection (missing/delayed emails and OTPs), impacted components (email delivery, SMS/email OTPs, realtime events), user impact, root cause (messaging queue overload causing dropped/delayed messages), mitigations (service restarts, migration of critical queues, disabling overload-inducing tasks), lessons learned, action items (monitoring/alerting improvements), and links to external incident documentation. No public API changes. Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (5)
2025/08/24/readme.md (5)
3-5: Confirm timestamps and metadataPlease verify start/end times against the status page and consider adding Severity and Regions to the header block for quick scanning.
9-9: Define OTP on first mention and standardize casing of “Realtime”Improves clarity and consistency.
-Between 24–26 August 2025, some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email OTPs, and realtime events. The issue was caused by instability in our messaging system. The problem was resolved by stabilizing the messaging infrastructure, and all services are now fully operational. +Between 24–26 August 2025, some Appwrite Cloud users experienced intermittent issues with email delivery, SMS and email one-time passwords (OTPs), and Realtime events. The issue was caused by instability in our messaging system. The problem was resolved by stabilizing the messaging infrastructure, and all services are now fully operational.
19-21: Format affected components as a bulleted listCurrent lines read as run-on text; list improves scanability.
-Email delivery (invitations, password resets, notifications) -OTP delivery via email and SMS -Realtime events +- Email delivery (invitations, password resets, notifications) +- OTP delivery via email and SMS +- Realtime events
64-64: Wording: singular “overload”Tiny consistency tweak.
-Improve monitoring and alerting for earlier detection of overloads. +Improve monitoring and alerting for earlier detection of overload.
62-65: Action items: add owners and due datesHelps drive accountability. Example:
-### Action items - -Improve monitoring and alerting for earlier detection of overload. +### Action items + +- Improve monitoring and alerting for earlier detection of overload. (Owner: SRE On-call; Due: 2025-09-15) +- Add queue depth/error-rate SLOs with paging alerts. (Owner: Platform Eng; Due: 2025-09-20)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
2025/08/24/readme.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
2025/08/24/readme.md
[grammar] ~3-~3: There might be a mistake here.
Context: ...Incident Start: 2025-08-24 07:26 UTC - Incident End: 2025-08-26 05:10 UTC - *...
(QB_NEW_EN)
[grammar] ~4-~4: There might be a mistake here.
Context: ...- Incident End: 2025-08-26 05:10 UTC - Report Prepared By: Divyansha ## Summ...
(QB_NEW_EN)
[grammar] ~19-~19: There might be a mistake here.
Context: ...tations, password resets, notifications) OTP delivery via email and SMS Realtime ...
(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ... analysis ### Preliminary findings We noticed few errors in the messaging system resp...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
2025/08/24/readme.md
70-70: Bare URL used
(MD034, no-bare-urls)
| # Short title | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Replace placeholder heading with a concrete incident title
Use a clear, user-facing title for indexing and readability.
-# Short title
+# Degradation: Email, SMS, and Realtime Delivery (Aug 24–26, 2025)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Short title | |
| # Degradation: Email, SMS, and Realtime Delivery (Aug 24–26, 2025) | |
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around lines 1 to 2, the placeholder heading "# Short
title" should be replaced with a concrete, user-facing incident title; update
the markdown H1 to a descriptive, searchable title that clearly summarizes the
incident (e.g., include service/component and brief issue summary) to improve
indexing and readability.
|
|
||
| ### Preliminary findings | ||
|
|
||
| We noticed few errors in the messaging system responsible for handling message delivery. |
There was a problem hiding this comment.
Grammar: “a few errors”
Minor clarity fix.
-We noticed few errors in the messaging system responsible for handling message delivery.
+We noticed a few errors in the messaging system responsible for handling message delivery.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| We noticed few errors in the messaging system responsible for handling message delivery. | |
| We noticed a few errors in the messaging system responsible for handling message delivery. |
🧰 Tools
🪛 LanguageTool
[grammar] ~31-~31: There might be a mistake here.
Context: ... analysis ### Preliminary findings We noticed few errors in the messaging system resp...
(QB_NEW_EN)
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 31, the phrase "We noticed few errors in the
messaging system responsible for handling message delivery." uses incorrect
grammar; change "few" to "a few" so the sentence reads "We noticed a few errors
in the messaging system responsible for handling message delivery." to improve
clarity and correctness.
|
|
||
| ### Resolution | ||
|
|
||
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload. |
There was a problem hiding this comment.
Grammar: pluralize “task” and add article
Also reads better with “the overload”.
-We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload.
+We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling tasks contributing to the overload.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling task contributing to overload. | |
| We stabilized the messaging infrastructure by migrating critical queues to a more reliable system and disabling tasks contributing to the overload. |
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 49, the sentence uses the singular “task”
and omits an article before “overload”; change it to pluralize “task” to “tasks”
and add “the” so the phrase reads, for example, “disabling the tasks
contributing to the overload,” resulting in: “We stabilized the messaging
infrastructure by migrating critical queues to a more reliable system and
disabling the tasks contributing to the overload.”
|
|
||
| ### Supporting documentation | ||
|
|
||
| https://status.appwrite.online/incident/711585 |
There was a problem hiding this comment.
Fix bare URL (MD034) with descriptive markdown link
Resolves markdownlint warning.
-https://status.appwrite.online/incident/711585
+[Status incident 711585](https://status.appwrite.online/incident/711585)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| https://status.appwrite.online/incident/711585 | |
| [Status incident 711585](https://status.appwrite.online/incident/711585) |
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
70-70: Bare URL used
(MD034, no-bare-urls)
🤖 Prompt for AI Agents
In 2025/08/24/readme.md around line 70, replace the bare URL
"https://status.appwrite.online/incident/711585" with a descriptive markdown
link to resolve MD034; update it to use bracketed link text like "[Appwrite
status: Incident 711585]" (or another concise description) followed by the URL
in parentheses so the markdown is more descriptive and passes markdownlint.
What does this PR do?
Added public incident report for Degradation in email, SMS, and realtime delivery on 2025-08-24
Test Plan
(Write your test plan here. If you changed any code, please provide us with clear instructions on how you verified your changes work.)
Related PRs and Issues
(If this PR is related to any other PR or resolves any issue or related to any issue link all related PR and issues here.)
Have you read the Contributing Guidelines on issues?
(Write your answer here.)
Summary by CodeRabbit