Authentication Backend is down
Resolved
Jun 28, 2022 at 6:21am UTC
This issue has the same root cause as yesterday's incident. The good news is that the changes to our automated monitoring did pick up on the issue this time. The bad news is it happened whilst everyone was asleep and so was not seen for 6 hours.
It is unnacceptable to be offline for such a long period of time and as such anyone 'on-call' (mostly just me!) will be automatically called, alongside the email + text already sent.
As for the underlying root cause, I am still investigating this, and the better tracking of the latest incident should help with this.
Affected services
Updated
Jun 28, 2022 at 6:05am UTC
Authentication Backend recovered.
Affected services
Created
Jun 27, 2022 at 11:45pm UTC
Authentication Backend went down.
Affected services