Delay on incoming mail
Incident Report for ROBIN
Postmortem

First of all, our apologies for the delays in processing e-mail and syncing our search function yesterday, March 4.

At 9:46 CET we noticed that the ROBIN API for incoming e-mail experienced a delay of 30 minutes.

To restore improve the processing speed, we have scaled-up our technical infrastructure. As we continuously scale our Azure resources to demand, this was our first go-to remediation. Unfortunately, this process didn't result in the expected improvement.

Between 12:00 and 16:30 CET we have analysed our databases, found out the root cause was database setup related and defined the required setup changes to resolve the issue and make the setup better for the future. We have planned these changes at 22:00 PM. As the load on the platform decreases after 18:00 CET we were able resolve the message and search sync queue around 20:13 CET. Around 22:00 we changed the database setup.

Again, our apologies for the inconvenience.

Posted 20 days ago. Mar 05, 2019 - 09:46 CET

Resolved
This incident has been resolved.
Posted 21 days ago. Mar 04, 2019 - 20:18 CET
Update
Today we are experiencing delays between 30 and 120 minutes throughout the day on all incoming e-mail and search sync. The root-cause is database related. The required database optimization actions to resolve the issue could result in a slow performing app, that’s why we schedule this optimization after 10 PM tonight. After 10 PM the mail queues are expected to be empty before midnight.
Posted 21 days ago. Mar 04, 2019 - 17:03 CET
Update
At this moment we are experiencing a delay of +/- 120 minutes in receiving and processing emails.

If you have any questions or if there is anything else, please reach out to our support team at support@robinhq.com or +31 85 877 00 23.
Posted 21 days ago. Mar 04, 2019 - 14:35 CET
Monitoring
We are experiencing a delay of 50 minutes in receiving and processing emails. We have made adjustments and are monitoring the queue.

We are actively monitoring this issue now and will post updates as soon as they become available. If you have any questions or if there is anything else, please reach out to our support team at support@robinhq.com or +31 85 877 00 23
Posted 21 days ago. Mar 04, 2019 - 10:53 CET
This incident affected: Incoming Mail.