Final Update:
Dear Customer,
We would like to provide you with a summary of the recent outage and the actions taken to ensure the system's stability moving forward.
Incident Summary:
Time & Date Range: Friday, 2024.11.29, 18:05 ET – Saturday, 2024.11.30, 20:24 ET
Event Symptoms: Customer reported a production outage at 09:13am ET
Event Trigger: Customer report.
Contributing Factors: The primary database (DB) experienced a disk space issue, causing the system to stop and login functionality to fail.
Severity Level: Highest.
% of Users Affected: 100% of BF/PT users.
Resolution:
The system was restored within 2 hours by promoting the database replica to primary and standing up a new backup replica.
A precautionary banner requested users not to write data until the backup was confirmed stable.
By 13:00 PST on Sunday 12/2, all precautions were lifted, and normal operations resumed.
Post-Incident Review:
We confirm no data loss or functionality issues occurred during the outage.
We have implemented measures to prevent similar incidents in the future, including improved monitoring and capacity management.
Thank you for your understanding and patience. Please feel free to contact us if you have any questions or need further assistance.
Sarah Schenkerberg
VP Customer Success