Sophos Cloud Optix
On 19 November at 04:39 UTC (23:39 EST), Microsoft Office 365 and Azure Active Directory users started reporting that they were unable to access the multi-factor authentication (MFA) system or reset passwords, locking them out of their accounts.
When Microsoft’s cloud authentication is working correctly, users should be able to authenticate their username and password credentials via text message, phone call, app verification code, or push request.
This, it turned out, was no mere hiccup, with problems for users across Europe, Asia-Pacific and the Americas continuing for at least eight hours – a long time for users to be unable to log into such an important business platform.
Microsoft eventually offered an explanation:
Preliminary root cause: A recent update to the MFA service introduced a coding issue that prevented users from being able to sign in or carry out self-service password resets when using MFA for authentication.
Twitter complaints soon rolled in on a scale ranging from annoyed to angry:
This is brutal. My team can't work. No one can log in to preform administrative duties.
— HarbinSEC (@harbinsec) November 19, 2018
Others raised the issue of powerless admins:
So, as an admin, I'm not able to log on, since I have MFA enabled. Is there a way to work around this?
— Pascal Engels (@draxken) November 19, 2018
Which is to say, admins couldn’t even temporarily turn off MFA for users as they were locked out by the same issue.
In theory, only organisations hosting Azure MFA on their own servers rather than through Microsoft’s infrastructure would have been unaffected by this.
Microsoft publishes advice on gaining emergency access to Azure AD accounts, including when MFA is unavailable.
It’s unclear whether this also applies when Microsoft’s own cloud authentication is not working (the documentation assumes this never happens).
What’s more, even if this is a viable workaround, implementing an emergency account comes with overheads, as the advice makes clear:
An account password for an emergency access account is usually separated into two or three parts, written on separate pieces of paper, and stored in secure, fireproof safes that are in secure, separate locations.
A memorably bad day eventually came to an end by around 21:30 UTC when MFA access returned to normal after Microsoft applied some “hotfix” medicine.
All of this matters because adopting multi-factor authentication is one of the most useful security upgrades you can make (it combats phishing, password reuse and weak passwords), but it’s one that users are already reluctant to make.
So is this a knock for the MFA cause? Maybe, but it shouldn’t be.
While it’s true that losing MFA hosted by a cloud provider as fundamental as Microsoft is bad news, the same could be said when losing access to any service, whether hosted in the cloud or not – the issue is downtime, not the merits of MFA.
MFA remains a good idea. So is having a plan B.
So, what would Naked Security recommend as a good Plan B then?
As an admin we set up MFA properly in the first place and we only had 2 users that were unable to log in. Also we were able to log in to global admin using a backdoor. come on guys! get creative!!
Nice job MS. People are already hesitant to enable MFA. I wonder how much this error will negatively affect adoption.
I think it’s more of a commentary on giving someone else all of your data than it is on MFA.
New name, Microsoft Office 364
Bravo! We’re wishing we’d used that as our headline now…
I couldn’t login to O365 on my PC coz the authentication code thing failed, but it was OK on my iphone. Possibly because of fingerprint validation. A pretty appalling advert for cloud services generally though. Everyone affected should get a refund/discount/apology in a sane world. Dream on.
Testing in production, just like w10.
It’s stunning how some of the big cloud services can have a single point of failure for so many.
8+ hours recovery seams like a really long time to patch/roll back a change. Wouldn’t fly where I work.
If you got locked out as an admin, you didn’t do you due diligence before implementing. You should use a strong, regularly rotated globaladmin account that is only used for situations like this (and some powershell commands that don’t support MFA) in addition to Whitelisted your place of business.
Silly headline. It was a bug that caused service disruption, not strong security. Something has happened at Microsoft that has seriously, materially eroded their quality.
I think it was intended to be sarcasm or a joke. Which is what that day was for some of MS’ customers.
I just found this article and I see many questions as to how best to mitigate this issue should it occur again. We spoke with support on the issue and they advised us to have a Global Admin Account that is a “break glass in case of emergency” account. No one should use it, it should have a random string for the account name, and as complex of a password as O365 will allow (which i am glad to say, as of today has been updated to 256 characters).
Hope that helps close the loop on this!
Thanks,
Harbinsec.com
Unfortunately Murphy’s law applies to cloud hosted 2FA as it does with anything else and you have to assume that it will happen again some day so a backup plan needs to be in place to deal with it before it happens again.