Previous incidents

Authentication Backend recovered.

Authentication Backend recovered.

Authentication Frontend recovered.

API recovered.

The concurrency limit has now been increased, and misconfigured autoscaling settings which would have caused the limit to scale up the service in response to reaching the limit, is now correctly configured. This incident is now resolved.

The issue has been fixed, and as a nice side benefit loading the UI should generally be a bit snappier.

API recovered.

Checkout recovered.

Authentication Backend recovered.

Everything is back to normal. The root cause appears to be the same as previous incidents last month, and a hopefully permanent fix is still being worked on.

This issue has the same root cause as yesterday's incident. The good news is that the changes to our automated monitoring did pick up on the issue this time. The bad news is it happened whilst everyone was asleep and so was not seen for 6 hours.

It is unnacceptable to be offline for such a long period of time and as such anyone 'on-call' (mostly just me!) will be automatically called, alongside the email + text already sent.

As for the underlying root cause, I am still investigating thi...

Everything is working as normal now. Our automated checks failed to detect this error and as a result the fix took longer than it otherwise would have. Changes have now been made which should notify us if this kind of issue happens again.