Previous incidents
Authentication Backend is down
Resolved Aug 15, 2022 at 5:48am UTC
Authentication Backend recovered.
1 previous update
Authentication Backend is down
Resolved Aug 14, 2022 at 3:32pm UTC
Authentication Backend recovered.
1 previous update
API, Authentication Frontend, Checkout, and 1 other service are down
Resolved Aug 13, 2022 at 5:57am UTC
Authentication Frontend recovered.
5 previous updates
API is down
Resolved Aug 12, 2022 at 3:13pm UTC
API recovered.
1 previous update
Some services are down
Resolved Aug 5, 2022 at 12:21am UTC
The concurrency limit has now been increased, and misconfigured autoscaling settings which would have caused the limit to scale up the service in response to reaching the limit, is now correctly configured. This incident is now resolved.
2 previous updates
Stream outputs loading slowly or not at all
Resolved Jul 28, 2022 at 1:30pm UTC
The issue has been fixed, and as a nice side benefit loading the UI should generally be a bit snappier.
1 previous update
API is down
Resolved Jul 28, 2022 at 5:17am UTC
API recovered.
1 previous update
Checkout is down
Resolved Jul 23, 2022 at 6:39am UTC
Checkout recovered.
1 previous update
Authentication Backend is down
Resolved Jul 22, 2022 at 3:17pm UTC
Authentication Backend recovered.
1 previous update
Authentication database timeouts
Resolved Jul 19, 2022 at 5:30pm UTC
Everything is back to normal. The root cause appears to be the same as previous incidents last month, and a hopefully permanent fix is still being worked on.
3 previous updates
Authentication Backend is down
Resolved Jun 28, 2022 at 6:21am UTC
This issue has the same root cause as yesterday's incident. The good news is that the changes to our automated monitoring did pick up on the issue this time. The bad news is it happened whilst everyone was asleep and so was not seen for 6 hours.
It is unnacceptable to be offline for such a long period of time and as such anyone 'on-call' (mostly just me!) will be automatically called, alongside the email + text already sent.
As for the underlying root cause, I am still investigating thi...
2 previous updates
Aircast authentication unavailable
Resolved Jun 27, 2022 at 12:04pm UTC
Everything is working as normal now. Our automated checks failed to detect this error and as a result the fix took longer than it otherwise would have. Changes have now been made which should notify us if this kind of issue happens again.
2 previous updates