Authentication Backend is down
Resolved
Jun 28 at 07:21am BST
This issue has the same root cause as yesterday's incident. The good news is that the changes to our automated monitoring did pick up on the issue this time. The bad news is it happened whilst everyone was asleep and so was not seen for 6 hours.
It is unnacceptable to be offline for such a long period of time and as such anyone 'on-call' (mostly just me!) will be automatically called, alongside the email + text already sent.
As for the underlying root cause, I am still investigating this, and the better tracking of the latest incident should help with this.
Affected services
Updated
Jun 28 at 07:05am BST
Authentication Backend recovered.
Affected services
Created
Jun 28 at 12:45am BST
Authentication Backend went down.
Affected services