Previous incidents

This issue has the same root cause as yesterday's incident. The good news is that the changes to our automated monitoring did pick up on the issue this time. The bad news is it happened whilst everyone was asleep and so was not seen for 6 hours.

It is unnacceptable to be offline for such a long period of time and as such anyone 'on-call' (mostly just me!) will be automatically called, alongside the email + text already sent.

As for the underlying root cause, I am still investigating thi...

Everything is working as normal now. Our automated checks failed to detect this error and as a result the fix took longer than it otherwise would have. Changes have now been made which should notify us if this kind of issue happens again.