Let's Encrypt Transitioning to ISRG's Root

Over an year ago, I posted a piece regarding Let's Encrypt and specifically me starting to use their TLS certificates. This certificate operation they're running is bit weird, but since their price for a X.509 cert is right (they're free-of-charge), they're very popular (yes, very very popular) I chose to jump into their wagon.

In my previous post I had this flowchart depicting their (weird) chain of trust:

I'm not going to repeat the stuff here, just go read the post. It is simply weird to see a CHAIN of trust not being a chain. Technically a X.509 certificate can be issued by only one issuer, not by two.

Last month they announced, they're going this way:

What is this change and how will it affect me?

Read the announcement at https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html. There is a very good chance, it will not affect you. Its more like a change in internal operation of the system. The change WILL take place on 8th of July this year and any certificates issued by users like me and you will originate from a new intermediate CA using a certificate issued by ISRG's own CA. So, the change won't take place instantly. What will happen is, for the period of 90 days (the standard lifetime of a Let's Encrypt certificate) after 8th July, sites will eventually transition to these new certificates.

The part where it might have an impact to you is with legacy devices, browsers, operating systems, etc. Especially, if you're one of the unhappy legacy Android mobile users, it WILL affect you. Lot of concerned people are discussing this change at https://community.letsencrypt.org/t/please-reconsider-defaulting-to-the-isrg-root-its-unsupported-by-more-than-50-of-android-phones/91485. If you're smart enough to avoid those crappy and insecure 'droids not having any security patches by a mobile vendor who doesn't care about your security, then this will not affect you.

Testing - A peek into the future

The way Let's Encrypt does their business is weird. I have stated that opinion a number of times. However, as they are weird, they are not incompetent. A test site has existed for a long time at https://valid-isrgrootx1.letsencrypt.org/, where you can see what will happen after July 8th 2019.

On any browser, platform or device I did my testing: everything worked without problems. List of tested browsers, platforms and devices will include:

• Anything regular you might have on your devices
• Safari on multiple Apple iOS 12 mobile devices
• Chrome on Huawei Honor P9 running Android 8
• Microsoft Edge Insider (the Chrome-based browser)
• Safari Technology Preview (macOS Safari early release branch)
• Firefox 66 on Linux
• Curl 7.61 on Linux

An attempt to explain the flowchart

Yes, any topic addressing certificates, cryptography and their applications is always on the complex side of things. Most likely this explanation of mine won't clarify anything, but give me some credit for trying.

In an attempt to de-chiper the flowchart and put some sense to a chain-not-being-a-chain-but-a-net, let's take a look into the certificate chain details. Some things there are exactly the same, some things there are completely different.

Chain for an old certificate, issued before 8th July

Command: openssl s_client -showcerts -connect letsencrypt.org:443

Will output:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = www.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIHMjCCBhqgAwIBAgISA2YsTPFE5kGjwkzWw/oSohXVMA0GCSqGSIb3DQEBCwUA
...
gRNK7nhFsbBSxaKqLaSCVPak8siUFg==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
...
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = www.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
...

Chain for a new certificate, issued after 8th July

Command: openssl s_client -showcerts -connect valid-isrgrootx1.letsencrypt.org:443

Will output:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFdjCCBF6gAwIBAgISA9d+OzySGokxL64OccKNS948MA0GCSqGSIb3DQEBCwUA
...
yEnNbd5O8Iz2Nw==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQELBQAw
...
rUCGwbCUDI0mxadJ3Bz4WxR6fyNpBK2yAinWEsikxqEt
-----END CERTIFICATE-----
---
Server certificate
subject=CN = valid-isrgrootx1.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
...

But look the same!

Not really, looks can be deceiving. It's always in the details. The important difference is in level 1 certificate, having different issuer. Given cryptographny, the actual bytes transmitted on the wire are of course different as any minor change in a cert details will result in a completely different result.

As a not-so-important fact, the level 0 certificates are complately different, because they are for different web site.

Then again the level 1 certs are the same, both do have the same private key. There is no difference in the 2048-bit RSA modulus:

Certificate:
  Data:
    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Subject Public Key Info:
      Public Key Algorithm: rsaEncryption
        RSA Public-Key: (2048 bit)
        Modulus:
          00:9c:d3:0c:f0:5a:e5:2e:47:b7:72:5d:37:83:b3:
          68:63:30:ea:d7:35:26:19:25:e1:bd:be:35:f1:70:
          92:2f:b7:b8:4b:41:05:ab:a9:9e:35:08:58:ec:b1:
          2a:c4:68:87:0b:a3:e3:75:e4:e6:f3:a7:62:71:ba:
          79:81:60:1f:d7:91:9a:9f:f3:d0:78:67:71:c8:69:
          0e:95:91:cf:fe:e6:99:e9:60:3c:48:cc:7e:ca:4d:
          77:12:24:9d:47:1b:5a:eb:b9:ec:1e:37:00:1c:9c:
          ac:7b:a7:05:ea:ce:4a:eb:bd:41:e5:36:98:b9:cb:
          fd:6d:3c:96:68:df:23:2a:42:90:0c:86:74:67:c8:
          7f:a5:9a:b8:52:61:14:13:3f:65:e9:82:87:cb:db:
          fa:0e:56:f6:86:89:f3:85:3f:97:86:af:b0:dc:1a:
          ef:6b:0d:95:16:7d:c4:2b:a0:65:b2:99:04:36:75:
          80:6b:ac:4a:f3:1b:90:49:78:2f:a2:96:4f:2a:20:
          25:29:04:c6:74:c0:d0:31:cd:8f:31:38:95:16:ba:
          a8:33:b8:43:f1:b1:1f:c3:30:7f:a2:79:31:13:3d:
          2d:36:f8:e3:fc:f2:33:6a:b9:39:31:c5:af:c4:8d:
          0d:1d:64:16:33:aa:fa:84:29:b6:d4:0b:c0:d8:7d:
          c3:93

New certificate from a GUI

Obviously, the same thing can be observed from your favorite browser. Not everybody loves doing most of the really important things from a command-line-interface.

Old one, having DST-root:

New one, having ISRG-root:

Completely different. Same, but different.

Confusion - Bug in Firefox

Chrome, correctly displaying DST-root for www.letsencypt.org:

Firefox failing to follow the certificate chain correctly for the same site:

This is the stuff where everybody (including me) gets confused. Firefox fails to display the correct issuer information! For some reason, it already displays the new intermediate and root information for https://letsencrypt.org/.

Most likely this happens because both of the intermediate CA certificates are using the same private key. I don't know this for sure, but it is entirely possible, that Mozilla TLS-stack is storing information per private key and something is lost during processing. Another explanation might be, that lot of Mozilla guys do work closely with Let's Encrypt and they have hard-coded the chain into Firefox.

Finally

Confused?

Naah. Just ignore this change. This is Internet! There is constantly something changing.