There’s a bit of a kerfuffle in the web hosting community just at the moment over an expired web security certificate from a certificate authority called Sectigo, formerly Comodo Certificate Authority.
Expired certificates are a problem because they cause the web server that relies on them to show up as “invalid” to any program that tries to do the right thing and verify the validity of the site it’s connecting to.
But this problem isn’t Sectigo’s fault – indeed, the company has had a warning about the impending problem available for a while now, explaining what was about to happen and why.
The problem comes from what’s known as backwards compatibility, which is a jargon way of saying “trying to support old software reliably even though it really ought to have been upgraded to a newer and more reliable version”.
When your browser visits a website, it’s almost certain to be using HTTPS, short for secure HTTP, which means using the Transport Layer Security protocol (or TLS for short) to encrypt and validate the connection.
As you probably know, TLS doesn’t vouch for the content that’s ultimately served up by a web server – crooks can use TLS to deliver malware “securely” if they like – but it is nevertheless a vital part of everyday browsing.
Not only does it shroud your traffic from surveillance and snooping, it stops someone in between you and the server you’re visiting from tampering with the content on the way out or back. (Because rogues of this sort can be anywhere along the network path, it’s known colloquially as a MiTM attack, short for man-in-the-middle.)
Of course, if crooks could trivially issue certificates in the names of other websites, MiTM attacks would still be easy, even with TLS, because the crooks could put a fake site half way along the network path to the real one, and you would be unable to tell it from the real deal.
So, to make it harder for crooks to mint a web certificate in your name, you need to get your certificate vouched for by someone else, known as a certificate authority.
You then present your certificate and their certificate, and they vouch for you; if their certificate is, in turn, vouched for by your browser itself (i.e. is in a list of already-trusted-certificates-that-can-sign-other-certificates), then your browser will automatically accept your certificate because it’d been signed by someone that the browser already trusts.
This forms a chain of trust.
What this means is that every browser (or every operating system on behalf of the browsers you might use) needs to have access to an up-to-date list of what are called root certificates, which is the name given to certificates that aren’t vouched for by anyone else, but that are explicitly trusted to vouch for others.
Obviously, the part of a root certificate that’s called the private key, which is used for signing purposes, needs to be kept extra-super-secure, because replacing or re-issuing root certificates is a much trickier exercise than updating or issuing so-called leaf certificates – the ones that go with your website and typically only last anywhere for 3 months to 2 years anyway.
To make it easier and safer to sign and distribute new keys, most leaf certificates use a chain of three links, not just two, to “prove” their validity.
There’s the leaf certificate that vouches for your website; there’s an intermediate certificate that vouches for your leaf; and then the intermediate certificate is vouched for by a root certificate that is itself magically imbued with vouching power because it is trusted directly by your browser or your operating system.
Root certificates therefore often have long lifetimes, typically 10 or 20 years, and the assumption is that everyone will have plenty of time to stop relying on old root certificates long before they expire.
But old software programs, and old operating systems, have long shelf-lives too, and old software programs, tied to an old database of trusted root certificates, often end up relying on ageing root certificates in their so-called “chain of trust” long after they should.
So, even if you do the right thing and ask your certificate authority – the company that’s vouching for you – to use their latest intermediates and their latest root certificates every time you renew your certificates, which is usually at least once a year, you might end up confusing customers with old software (possibly even with old software of your own manufacture).
That’s because old software that hasn’t yet been taught about the latest and greatest root certificates that are available – because it’s not getting reliably updated, for example – will keep on trusting the old root certificates you are keen to move away from, even as they edge towards expiry, yet will keep on rejecting the new ones as “untrusted” even though the new ones have years of life left in them.
Ironically, then, the newer and fresher your chain of trust, the less reliable your certificates will seem to old-timer programs out there.
What many companies do, to support both ends of the equation, is what’s called cross-signing, where they denote two different intermediate certificates to vouch for your leaf certificate, one signed by an old root; the other by a new one.
The idea is to please most of the people most of the time.
Of course, that can make your security situation seem better than it is.
Old and possibly insecure web clients – which will include all sorts of software tools other than browsers, notably including autoupdate programs and licence-checking tools that are supposed to keep the software running correctly – will give you a false sense of being “up to scratch”.
When the tired old root certificate expires, software that has never heard of the all-new root certificate that replaced it will simply stop working. (Unless it isn’t checking the validity of your web certificate at all, but that’s increasingly rare because it’s easy for researchers to detect and will guarantee bad publicity if they do.)
It’s worse than that
But, as Andrew Ayer of SSLMate explains, the situation is worse than that.
Technically speaking, certificate chains where there’s a choice of cross-signed intermediate certificates, can be “resolved” more than one way.
You can follow the old-style intermediate certificate to the now-expired root certificate, or you can try the other way home, validating with the new-style intermediate and correctly determining that it is signed by a new and valid root.
Ideally, newer certificates should trump older ones, so that as long as one of the certificate chains checks out, the leaf certificate should be accepted.
But, as Ayer explains, some older TLS software (or some older versions of current TLS libraries) fail if the first certificate chain they try has expired, even though trying again with fresher data would find that the HTTPS connection was valid.
That’s the trouble here – even though one of Sectigo’s backwards-compatible root certificates has now expired, some web software is still relying on that old root certificate, which expired on 30 May 2020, even though it already knows about the new root certificate and should be verifying the certificate chain as valid.
What to do?
If you are getting web connection errors on software that was working fine until the end of last month, where the error lists an invalid certificate called AddTrust External CA Root, you need to take action.
You may need to update the software that’s trying to make the connection, the software that’s trying to handle the connection, their root certificate “trust stores”, or some combination of these.
If you’re stuck, consult your vendor – and if you are the vendor because it’s your own software, you may need to consider upgrading to a more modern TLS programming library that handles web certificate verification in a more future-proof way.
Ayer has some advice in his blog article – notably, if you are using a TLS library that ought to validate Sectigo certificates but isn’t doing so, you may be able to fix the problem simply by deleting the now-expired AddTrust External CA Root certificate – which is no use anyway but may nevertheless get in the way – from the certificate database on your computer.
The expired certificate was replaced a decade ago (!) by one denoted USERTrust RSA Certification Authority, so many TLS libraries do known about the “new” root certificate perfectly well; the problem is that they still know about the old one too, and get hung up on it even though it serves no purpose any more.
Latest Naked Security podcast
Click-and-drag on the soundwaves below to skip to any point in the podcast. You can also listen directly on Soundcloud.
33 comments on “The mystery of the expiring Sectigo web certificate”
So how much trouble would it be for Microsoft, Apple, and Linus Torvalds to keep issuing certificate updates for their obsolete systems, even if they aren’t creating any other software updates? And the same question for the independent browsers like Firefox and Silk? It’s not like a lot of engineering is involved–it’s the same list update they send to the active software versions.
I vaguely remember this same thing happening in the late 1990s.
The real problem here is that even in the presence of the latest certificates, there may be software that’s still reaching outdated conclusions – a bit like having a current passport but being deemed not to have one because you still have your old, expired one as well, even if it’s got the corner cut off to mark it as irrelevant.
This was perfectly timed. It solved a support ticket that was opened moments before this article arrived in my inbox. Keep up the great work!
The Sophos XG had this problem – had to add in the current root certificate manually to fix a load of certificate expired error pages our users were getting.
Great article and the first one I’ve seen on this problem! You’ve missed the part WHY this is such a huge problem. It’s such a huge problem because Namecheap (SSLs.com) had a bug in their software and they were issuing CA Bundle files to anyone buying a cert. This was happening up until either april or yesterday, depending on whether you got one from a comodo or sectigo root CA. So for instance, I had to fix the bundle file on certs bought as recently as December for my customers.
When you buy the cert, they give you the .crt file (new, signed from the CA) and then they also include an example.com.ca-bundle file for you to use as your intermediaries (in Apache that’s your SSLCACertificateFile directive).
So, come Saturday morning, all the SSL certs from SSLs.com and any other affected providers broke to libcurl on OpenSSL <= 1.1.1. libcurl literally powers most of the automation on the internet and OpenSSL 1.1.1 is on Deiban 9 (oldstable, still supported) and Ubuntu 16.04 anjd probably some RHEL/CentOS combos that are still vulnerable.
In these versions, the bug where OpenSSL fails right away instead of trying to cross sign happens.
Debian and Ubuntu are both rushing to ship a new ca-certificates package that does not have that expired AddTrust .crt in it, and that works around the problem.
But the key piece most people were missing is that none of this would have been an epic crisis like it was, -HAD NAMECHEAP NOT SHIPPED THE WRONG BUNDLE FILE right up until the day it expired. 2 years before May 30th they should have switched to the correct new updated bundles, but due to a bug in their system that old expiring one was cached and now that's what everyone is using on their server config.
This wasn't clear in their original KB articles but they have since fessed up:
Lots of info there about the bug that caused this to be so widespread.
I have some more comments in the Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961907#51
Excellent write up which describes our situation exactly (Namecheap & Ubuntu 16.04 & libcurl).
Now I understand why we started getting weird errors in our web backend on the weekend.
Deleting the expired intermediate (from the hosting end) should cause any modern and capable system that was still throwing errors to start working again.
The big issue here. The part that is absolutely 100% Sectigo’s fault is that they didn’t say that… in fact, their FAQ said the exact opposite of that.
Telling everyone.. “Don’t worry, you’ll be fine unless you are talking to very old and insecure systems” was completely incorrect. I had several external 3rd party “modern” systems unable to connect to my applications. Yeah maybe they were doing it wrong… But this could have easily been avoided if I deleted the intermediate before it expired
A big long support page explaining cross signing is also not helpful. What would have been helpful would have been short, direct, easy to follow steps you can take to resolve the issue.
They should have stopped issuing cross signed certs to an expiring CA 2 years ago so nobody was affected at all. If you have a legacy system, sorry your new cert is secure and won’t work. That’s what happened anyway but it would have spread out any issues people DID have to a manageable flow support could handle vs blowing up all in one day.
I never received an email telling me about this. I don’t keep track of expirations of root certs. So I was blindsided.
Yeah… This WAS Sectigo’s fault. Total mismanagement.
Go back and re-read the part about backwards compatibility…
Great article. Could you please pass this on to the team responsible for the Sophos UTM product so they can fix it. Disabling the AddTrust External CA Root and/or adding the USERTrust RSA Certification Authority does not fix this problem for the web filter in the latest UTM firmware.
I’ve had a support request open for 2 days now and still no response.
Thanks! I’ve passed on your comment as requested.
AFAIK, however, the problem of expired certificates that are still being used explicitly isn’t one that can unilaterally be fixed at the client end – if a server sends you a certificate chain that explicitly includes an expired component, or implicitly chains to an expired root certificate…
…then the certificate chain won’t validate. Should the client go out of its way to find an alternative but still-valid chain from the leaf to a valid root? Or should it validate the chain supplied by the server, as supplied by the server?
Thanks Paul, the issue for us was that the Sophos UTM web filter would block access to some websites using affected certificates claiming that the certificate had expired, however if we took the web filter out of the equation and accessed those websites directly then they were accessible without issue.
It turns out that disabling the built-in AddTrust External CA Root certificate does resolve it for the Sophos UTM web filter, however doing so requires a restart of the UTM for the change to take effect.
Thanks for the feedback!
I wasted a few hours the other day trying to work out what the issue was. A customer purchased the certificate in the first place and asked me to install it last month. Then over the weekend our monitoring system started raising alarms about cert chain errors – at first I thought someone is mucking about with the certs or the server has been compromised.
After looking at Sectigo site from a Google I found the articles and obtained the updates root/inter certs and glued it together, it’s all good again… hours I will never get back!!
can you tell me where the new certs are?
can you give me some idea how you got this working?
Kind a bit of topic, but I have to comment on this one:
> (Unless it isn’t checking the validity of your web certificate at all, but that’s increasingly rare because it’s easy for researchers to detect and will guarantee bad publicity if they do.)
Well in the Erlang Programming language the SSL library at is core doesn’t validate certificates by default, unless you configure it to do so, and I never see it getting bad publicity or getting a CVE. What is the point to have an ssl library that doesn’t check certificates by default???
I’m not sure why Erlang’s default is
verify_nonebut even if it were I would prefer to see the code being specific anyway (setting
verify_peer). I am not convinced that having an insecure default is a CVE-worthy offence… if the Erlang library didn’t support TLS verification at all then I would be worried but I don’t see a problem with turning it on explicitly in your own code. After all, if you are planning to do verification then you need to supply a list of trusted root certificates anyway, and there’s no default for that – yet one poor or rogue choice in the root CA list and you could be making your security worse.
So tell me once more… What is the point of having an SSL/TLS library that by default doesn’t check certificates? That’s just no sense. The library should default to verify them and raise an exception when could not found the path to the root certificates.
The internet is so broken exactly because this type of complacency with bad defaults in pretty much any software we use, like databases, aka MySql, ElasticSearch, MongoDB, etc.
Just use https://www.shodan.io/ to find many open to public, and why are they open? Well it’s because the default is to install them without any security… really???
Security SHOULD be opt-out in software development not opt-in as it is in the majority of the cases.
Until this mindset changes the Internet will continue broken, and data-breaches will be a common thing in the day life of each citizen with an online presence.
I already answered your question and I have not changed my view. The fact that it is possible to use a programming library insecurely does not, in my opinion, constitute a CVE-level security vulnerability in that library.
Erlang’s TLS library provides a way to check a certificate chain, but doesn’t provide a curated list of root certificates for completing that validation. If you want Erlang’s library to do the validation for you, then you need to provide your own list of roots and tell Erlang where to find it using the
client_cafile()options. If you don’t supply one of those options then as far as I know there will be no validation attempted.
I don’t think that “failing to do the programmer’s work for them at every step” counts as a CVE-quality bug in a software library – if there is a bug caused by poor programming in the code that /calls/ the library then I would suggest that any CVE belongs to the code doing the calling, not to the code being called. (Otherwise, every use-after-free bug that ever happened would surely have to be considered a CVE in the heap manager, not in the code that incorrectly used the heap, and that is not generally how CVEs are assigned.)
By your argument, namely that all security-related options should be on my default, you would surely expect Erlang to force certificate validation in its server code, too – i.e. require the client to present a valid certificate – yet it does not, and neither does any other well-known TLS library as far as I know.
Sorry Paul, my simple brain started to overload half-way down this article.
So this is an end-user problem, not a website problem? The end user has to delete the AddTrust External CA Root certificate from their computer, not the website that is serving the content?
I think the answer is, “a bit of both”. A server can send a full or partial certificate chain along with its certificate, so it’s worth helping it avoid using an intermediate that, at the other end, will end in an expired root. And (from memory) there are TLS clients that cache previous intermediates and in the absence of a certificate chain from the server will re-use a cached intermediate. Presumably if the intermediate was signed by a valid root last time it was used, it will get cached and could result in a failed validation now.
The ultimate solution is for anyone with an intermediate that is signed by a now-expired root to retire that intermediate at once, given that it will never be valid again – if it isn’t there then it can’t show up in certificate chains, either by accident or design.
I had both experiences – some customers solved the problem; but we uploaded a better bundle and resolved all the others.
I still have the direct-from-Sectigo bundle from April, with the expiring intermediate included.
The Sophos Web Appliance (SWA) has this issue too – it also has the expired AddTrust External CA Root installed and tries to authenticate against it. We are pretty busy manually adding sites to the exception list until Sophos publishes an update as there is no option to manually remove the certificate.
It’s kind of ironic Sophos publishes this blog which explains the situation and solution while at the same time their products are affected as well 😀
Thanks, I’ll pass your comment on to the team.
Quote: “But old software programs, and old operating systems, have long shelf-lives too, and old software programs, tied to an old database of trusted root certificates, often end up relying on ageing root certificates in their so-called “chain of trust” long after they should.”
You mean Sophos Web Appliance? Maybe Paul Ducklin should work in the development/engineering department at Sophos instead of wasting his knowledge on this blog and the current developers should find something else to do.
That’s a compliment (I think?!? :-) – but the situation isn’t quite as simple as it might at first seem. After all, if the TLS peer at the connecting end receives a certificate chain from the server that says, “Here is the list of certificates I have chosen to present for the validation process”, and the root has expired then either [a] the connecting client will report that fact or [b] the connecting client does not have the expired root and will therefore report the connection as untrusted anyway. (In other words, if you insist that I accept a certificate chain that includes an expired component then the list of certificates I have at my end won’t “unexpire” your chain.) So IMO this is one of those problems that “emerges from the ecosystem”, if you like, on account of the desire to have backwards compatibility.
Yes, you can take this as a compliment.
Concerning the multiple trust paths: The code has to be written, of course. However, logically it should be a no-brainer. The only reason to have multiple trust paths is still having a valid certificate as long as there is still at least one trusted path. You can easily make up a real-life non-technical example to understand how trust works. All of this should have been discussed, standardized and tested for as long as you can have multiple trust paths.
Obviously, the Sophos Web Appliance acting as a proxy doesn’t behave like a client with an implementation like this. If this is actually fixed in OpenSSL, this makes you wonder how flawed/outdated the libraries of the Sophos Web Appliance (SWA) are. Now, we have several options to workaround this issue:
– Disable it completely.
– Disable it for each website with a broken chain.
– Disable certificate validation for all websites.
Since normal browsers don’t seem to care and there are already myriads of websites with missing intermediates, which the SWA doesn’t like, it’ll probably take at least until the current certificates are expired – up to 2 years – until this situation is completely resolved.
Servers don’t remove outdated certificates because the handling of certificates is usually very primitive which is a good thing. All you need, is the path to a base64-encoded private key and certificate chain. This low complexity is part of the reason why so many applications have good TLS support at all and people are actually capable of installing and updating certificates. If anything. a library like OpenSSL/GnuTLS should handle expiry of certs and it should not be the responsibility of the server application.
My understanding is that the point of cross-signing was not to work around the ongoing use of expired certificates. IIRC it exists to support browsers or operating systems that were slow at adding new roots to the their trusted store, so you could simultaneously have two valid certification chains – not so that you could have one valid and one expired.
As for this “being fixed” in OpenSSL (not sure what other libraries do), there are two supported OpenSSL versions and they behave differently – one arguably more correctly and the other more conveniently.
I am not convinced by the idea that this is “too hard” for servers to get right. (I use Let’s Encrypt and I can keep multiple private keys, certificates and chain files current automatically.)
I think this is one of those cases of the old-school internet coding mantra – IRIC it was the late, great Jon Postel who said this – that you should be strict in what you send but liberal in what you receive. I have always understood that theory, which is a bit like driving carefully yourself but assuming everyone else won’t. But it has the unfortunate side-effect that those best placed to get everyone to “do it right” are also inadvertnetly responsible for ensuring that they never need to…
From what I hear, a SWA update should arrive next week to work around this issue – but there will still be loads of servers out there that still don’t realise they are vouching for themselves wrongly.
It is Sectigo’s/Comodo’s fault.
I did buy multiple certificates in the past 6-9 month ( so the 1 year certificate is still good ), but they were signed, (and they sent me) with a root certificate which expired before the certificate expiration date. And in the mails with the certificate&intermediate certificate there was no single word that hey, stuff may begin to fail before the certificate expires. The mails were sent by email@example.com, so not a SSLS/NameCheap problem
From technical point of view their action may be ok, but from a business/customers point of view they failed miserably .
Can confirm, we also saw certs being given out a few months ago along with 20 year intermediate certs that had an expiration date of May 30 2020.
Here’s the headache this I’m having. Web servers are sending us three certificates:
1) C1 is an unexpired server certificate
2) C2 is an unexpired intermediate certificate signed by a trusted root.
3) C3 is an expired intermediate certificate signed by a trusted but expired root.
You can validate C1 with C2 as the signer, and if the web site sent only those two certificates you’d be fine. However the policy on our web proxy rejects this connection because the server also sent the expired C3. Is that the rational policy?
My own opinion (and I am aware this is not a terribly useful answer because it is not a conclusion :-) is, “Yes and no.” Some TLS libraries (OpenSSL 1.1.1 is one, I think) try to find a validation path amongst their own stash of certificates and accept the chain if they can find one; others find *a* path (OpenSSL 1.0.2 is one, I think) and if it doesn’t pass muster, they reject it.
The path of least resistance is the former, and AFAIK browsers (Firefox is one – it uses NSS, not OpenSSL) do it for that very reason. IIRC, Firefox even caches intermediates it has used before (see the file cert9.db in your Firefox data directory) so it can validate without a certificate chain at all – though this can result in the weird behaviour that the validity of a site you visit now may be influenced by the sites you’ve visited before. For example, a site that can’t be validated today might work tomorrow if you happen to visit a different site in the interim that had a certifcate from the same CA and did sent a suitable intermediate that was kept in the cache.
But if a server is presenting you with a certificate chain that includes expired data… is the correct behaviour really to ignore the flaws if you can “find a better way”? It’s less convenient but you can argue that it is more *rational* (in a literal sense) to say, “There’s something ambiguous here, and that can’t be right”
So IMO the proper solution is for servers not to send expired data at all, while the practical solution is for proxies/browsers/clients/validation tools to “make the best of a bad job”.
O, what a tangled web we weave
When first we practise to deceive.
If you delete the expired root certificate from your proxy’s own certificate store so that the proxy can’t follow that route to a root, does that make the problem go away? (Or does it just make the proxy complain that there’s an untrusted/nuvalidatable path in the certificate chain, which would be an equally valid if still undesirable conclusion?)
The Sophos Web Appliance has this problem, too – so Sophos development is working on a fix…