Dragged by the roots
This one had me scratching my head for a while today. A client and an ex-client both contacted me with strange HTTP connectivity issues, which manifest as errors occurring on one server while the exact same code is working elsewhere. The logs revealed that a HTTPS connection was being rejected because the connection to the external site could not be validated. The problem was that the root certificates were out of date, and the external site was using Let’s Encrypt SSL certificates, which as of this month (October 2021) has a new compatibility restriction meaning their certs can only be validated by a client if the client trusts the ISRG Root X1 certificate. That restriction prevents functionality on iPhones running anything before iOS 10, anything earlier than macOS 10.12.1, various Kindles, early versions of Java 7 and 8, Firefox before v50, and much more. If your Web page makes use of a third-party service that behind-the-scenes is connecting to a site with the latest Let’s Encrypt certificate, that interaction with the third-party could fail due to certificate rejection.
In some cases it’s easy to resolve. Just update the OS/Browser/VM (e.g. AMI)/platform/etc. or whatever it is that is connecting to the LE-certified site. In other cases, such as older pre-iOS 10 devices, your luck has run out.
This is going to get worse as the weeks roll on. Many sites still have older LE certs installed so their clients are currently OK, but the sites automatically update and at some point they will be issued a new LE cert that has the ISRG Root X1 requirement. Once that site gets that upgrade, many of its clients could be affected.
Prepare for increased customer service calls and a lot of tearing hair out by the roots.
Categorised as: Networking, Security, Web