Security

Yesterday I stumbled on Let's Encrypt certificate on a Fedora linux. Read about my initial (failing?) approach from this blog post.

Given the ticket I filed regarding the problem, I got a pointer to study Using Shared System Certificates in Fedora. As you might think, I didn't know anything about that earlier. In a nutshell, ever since Fedora 19 was released in 2013, there has existed an unified storage for X.509 certificates with appropriate tooling to maintain certificates for OpenSSL, which is used by many components of the system, including curl. Besides the most common one, also a number of other certificate storages are included in the unified system, including NSS (for Firefox browser), GnuTLS and Java. Which is nice!

This subsystem has existed there a while, but marketing for it has been not-so-strong. Now that I know it exists, I'll offer the proper and correct way of adding the missing Let's Encrypt intermediate ISRG X3 certificate into your system.

To fix your curl, as regular user, download the X.509 certificate from Let's Encrypt website to /tmp/-folder.

$ wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt

As root, add the newly downloaded file as a trust anchor:

# trust anchor --store /tmp/letsencryptauthorityx3.pem.txt

That's it! No other commands to run. This sequence will update file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem with the current set of certificates to be trusted. That dynamically generated file will be used by curl ia symlink /etc/pki/tls/certs/ca-bundle.crt.

As an alternative, storing the letsencryptauthorityx3.pem.txt into directory /etc/pki/ca-trust/source/anchors/, and running update-ca-trust as root will do exactly the same thing.

Simple as pie! (if you know the pie exists)

Update 8th Aug 2019:
The information in this blog post is not the entire truth and less accurate, than I'd hope for. Updated information regarding certificate system in a Fedora linux can be found from this blog post.

Now that Let's Encrypt is issuing their certificates from ISRG Root X1 / Let's Encrypt Authority X3 certificate authority, my Fedora is failing to access sites with curl. I have addressed both of these earlier in articles Let's Encrypt Transitioning to ISRG's Root and Fixing curl with Go Daddy Secure Certificate Authority G2 CA root. However, the previous fixing instructions in modern curl won't work. This is because Fedora guys won't build curl with NSS anymore, but have fallen back to standard OpenSSL used by almost everything else in Linux. Since I'm writing about it, they're doing a bad job about it.

If you'll never use curl or anything built with libcurl, you'll never notice anything. Also, if you're running curl/libcurl but are never accessing any sites with Let's Encrypt certificates, again, you won't notice it. Most of us will notice. Curl will keep spewing errors like "curl: (60) SSL certificate problem: unable to get local issuer certificate" making everybody mad.

Unlike earlier, curl run with --verbose will announce following certificate locations:

* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

It so happens, the new intermediate certificate used by Let's Encrypt isn't in the precious ca-bundle.crt file, your connection attempts will fail on a trust issue. You can run the classic OpenSSL certificate fix -operation:

# cd /etc/pki/tls/certs
# wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt
# mv letsencryptauthorityx3.pem.txt letsencryptauthorityx3.pem
# openssl rehash

... which will fix a lot in your Fedora, but it won't fix curl. Remember: it will only look inside the /etc/pki/tls/certs/ca-bundle.crt, and you did not add it there.

To get the issue solved for realz, you need to delete the symlink pointing to the certificate bundle file, cook up a new & better one. Like this:

# cd /etc/pki/tls/certs
# rm -f ca-bundle.crt
# cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/letsencryptauthorityx3.pem > ca-bundle.crt

Now your curl will be tamed. But it won't stick. The first time you'll get an update to RPM-package ca-certificates, your changes are gone and you'll need to re-do them. Gee! Thanks for that.

Bug report is at Curl not using CA-path - custom certificates not used. Go add your comments there to let Fedora guys know how much you appreciate their work!