Software

Curl is one weird piece of software. Every time I use it, I get chills. The reason for this is, that it almost works and when it doesn't there isn't a damn thing you can to to fix it. The entire design of that software is ... I'm lost for words here. I'm looking for words that describe: cumbersome, shitty, unorthodox, non-functional, and so on.

Since the lib-version is used by a number of libraries and other software as a means to provide HTTP-protocol implementation I do run into curl-issues often. Many times I didn't even know, that in the end I was using libcurl for access before one of these obscure errors pops. For this reason, my weapon-of-choice is wget, it uses OpenSSL's crypto and is fully compatible with pretty much everything else in a Linux-distro.

Anyway, this time I chose to research this to the bitter and. It took me about a month (calendar time) to resolve this. Of course I didn't spend all my time and energy into this, it just took a very long time to get this one done properly & right.

The problem

One day, I was just tinkering something and ran a command:

$ curl --verbose https://packetstormsecurity.net/

... and it pulled a curl on me.

* About to connect() to packetstormsecurity.net port 443 (#0)
* Trying 198.84.60.198...
* Connected to packetstormsecurity.net (198.84.60.198) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=packetstormsecurity.com,OU=Domain Control Validated
*       start date: May 31 18:04:40 2015 GMT
*       expire date: May 31 18:04:40 2016 GMT
*       common name: packetstormsecurity.com
*       issuer: CN=Go Daddy Secure Certificate Authority - G2,
              OU=http://certs.godaddy.com/repository/,
              O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US
* NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.

* Closing connection 0
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

I hate when that happens. Also I don't know who forgot to do what, but it just won't work and nobody in the entire Internet knows how to handle that.

Figuring out the details of the issue

This is the easy part:

* NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
* Peer's Certificate issuer is not recognized.

In human language that reads: The problem is with HTTPS. The certificate used by the remote site is issued by a Certificate Authority (CA), that we don't know of and because we don't know it we won't trust any certificates issued by it.

Further:

* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

In human that reads: There were three (3) places I tried looking for the root-certificate:

1. NSS is the Network Security Services -library created by Mozilla. Its database is located at: /etc/pki/nssdb
2. OpenSSL is the library/toolkit used by pretty much rest of your Linux for HTTPS-encryption. It has the trusted root-certificates at: /etc/pki/tls/certs/ca-bundle.crt
3. OpenSSL didn't use a directory (CApath) for certificates.

After doing all three of those, no suitable CA was found and curl had to disconnect from the server and inform user about this lack-of-trust -issue.

Possible solutions

Ignore the issue and force curl to continue

There is a suggestion:
turn off curl's verification of the certificate, use the -k (or --insecure) option

This goes right out the window, because I'd rather chew off my left arm than force a prefectly valid check to be ignored. I'd rather spend the time investigating the fix. But that's me. Most of you don't care. I do.

Add the missing certificate to ca-bundle.crt

Good idea, but ... That file is reserved for the operating system / distro by OpenSSL. Actually curl uses this file only to make its own life easier so that curl doesn't have to distribute a set of trusted CA certificates, it just piggy-backs with something OpenSSL has.

Since this approach is OpenSSL-specific there is a procedure for adding own CA root-certificates into your system. When a new CA-root needs to be installed the mechanism is not to copy the certificate into this big file. How to actually do it, we'll get into that later.

The obvious problem with this approach is, that next time your distro gets a new CA-bundle one of two things will happen: 1) your changes will be overwritten and lost, you'll have to add the CA-root again or 2) the new CA-bundle won't be installed, because somebody messed up a file which he/she shouldn't do. This is definitely not a good approach.

Implicitly specify the CA root-certificate file

Aa-ha! There is a command-line option for this purpose:

--cacert <CA certificate>
       (SSL) Tells curl to use the specified certificate file to verify
       the  peer.  The  file  may contain multiple CA certificates. The
       certificate(s) must be in PEM format. Normally curl is built  to
       use a default file for this, so this option is typically used to
       alter that default file.

That's the one I could use, if I'd like to do that every goddamn single time I curl for something. First I don't want to do that every time and second, that command-line option isn't available for me, as I was using a piece of software wrapped to use libcurl.

Add the missing CA root-certificate into NSS database to establish trust

This is the one I chose. This is also the one nobody gets solved.
If you can find precise information on the web how to fix this, please tell me. I've been browsing the net for partial and non-working solutions enough not to care for half-assed solutions which don't work at the end.

Getting the missing certificate

Whatever we do (except just ignore the problem), the missing root-certificate needs to be located. With a little bit of googling I found a page Repository, Here’s a collection of important certificate documentation (https://certs.godaddy.com/repository/) at GoDaddy's server. Sure, the initial impression was "whoa, that was easy!", but when I landed on the page, I realized that there were following root-certificates available for GoDaddy Certificate Chain - G2 to download:

1. GoDaddy Class 2 Certification Authority Root Certificate - G2
2. GoDaddy Secure Server Certificate (Intermediate Certificate) - G2
3. Microsoft to GoDaddy G2 Cross Certificate
4. GoDaddy G2 Code Signing Intermediate
5. GoDaddy Secure Extended Validation Code Signing CA - G2
6. GoDaddy Certificate Bundle for Microsoft Windows Driver Signing - G2
7. GoDaddy Certificate Bundles - G2
8. GoDaddy PKCS7 Certificate Intermediates Bundle (for Windows IIS) - G2
9. GoDaddy Certificate Bundles - G2 With Cross to G1
10. GoDaddy Certificate Bundles - G2 With Cross to G1, includes Root

Ok, which one will I need? Darn!

Luckily I know something about X.509 certificates and especially certificate extensions. There should be an AIA or Authority Information Access -section (see RFC 5280 section 5.2.7 for details) in the cert. At least most CAs provide that information to make people's life easier.

First download the cert with a one-liner:

$ echo | \
  openssl s_client -connect packetstormsecurity.net:443 \
​  > /tmp/packetstormsecurity.net.cert

Btw. the one-liner will say dumb things like:

depth=0 OU = Domain Control Validated, CN = packetstormsecurity.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 OU = Domain Control Validated, CN = packetstormsecurity.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 OU = Domain Control Validated, CN = packetstormsecurity.com
verify error:num=21:unable to verify the first certificate
verify return:1

That's just vomit from the fact, that the certificate isn't trusted. The important thing is, that the one-liner will result a text-file with lot of other garbage, but also the server certificate PEM. Luckily OpenSSL will ignore all the garbage when doing command:

$ openssl x509 -noout -text -in /tmp/packetstormsecurity.net.cert

That one will output a lot of stuff. Most if which are irrelevent for this purpose. The relevant things are:

Certificate:
  Signature Algorithm: sha256WithRSAEncryption
      Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc.,
              OU=http://certs.godaddy.com/repository/,
              CN=Go Daddy Secure Certificate Authority - G2
      X509v3 extensions:
          X509v3 Certificate Policies:
              Policy: 2.16.840.1.114413.1.7.23.1
                CPS: http://certificates.godaddy.com/repository/
          Authority Information Access:
              OCSP - URI:http://ocsp.godaddy.com/
              CA Issuers -
              URI:http://certificates.godaddy.com/repository/gdig2.crt
          X509v3 Authority Key Identifier:
     keyid:40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE

Exactly what we needed! There is an AIA-block with a direct URL of http://certificates.godaddy.com/repository/gdig2.crt in it.

A download:

$ wget http://certificates.godaddy.com/repository/gdig2.crt \
  -O "/etc/pki/tls/certs/Go Daddy Secure Certificate Authority - G2.pem"

... and verify that certificate's serial number:

$ openssl x509 -noout -text \
  -in /etc/pki/tls/certs/Go\ Daddy\ Secure\ Certificate\ Authority\ -\ G2.pem

... will reveal:

Certificate:
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc.,
                CN=Go Daddy Root Certificate Authority - G2
        Subject: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc.,
                 OU=http://certs.godaddy.com/repository/,
                 CN=Go Daddy Secure Certificate Authority - G2
        X509v3 extensions:
            X509v3 Subject Key Identifier:
           40:C2:BD:27:8E:CC:34:83:30:A2:33:D7:FB:6C:B3:F0:B4:2C:80:CE
            X509v3 Authority Key Identifier:
     keyid:3A:9A:85:07:10:67:28:B6:EF:F6:BD:05:41:6E:20:C1:94:DA:0F:DE

Oh yes x 2!! The CA certificate has the correct serial number. It issued the failing certificate. This proof of correct CA-chain. We found the correct file.

Establishing trust to the new CA root-certificate in OpenSSL

This is the easy part. This one I have done hundreds of times.

First get a hash of the certificate:

$ openssl x509 -hash -noout \
  -in /etc/pki/tls/certs/Go\ Daddy\ Secure\ Certificate\ Authority\ -\ G2.pem

For this particular certificate, the hash is 27eb7704. The next thing is to instruct OpenSSL that this newly downloaded certificate is trusted by our server. It can be done like this:

$ ln -s /etc/pki/tls/certs/Go\ Daddy\ Secure\ Certificate\ Authority\ -\ G2.pem \
  /etc/pki/tls/certs/27eb7704.0

The idea is to symlink the downloaded file with a filename from the hash and suffix the file with a .0 (dot-zero).

Now we can verify, that our setup was done correctly (remember the "garbage" file we downloaded earlier):

$ openssl verify /tmp/packetstormsecurity.net.cert

The only valid output would be:

/tmp/packetstormsecurity.net.cert: OK

Anything else, and you fumbled it.

Additional step: Add all hashes of the certificate chain

Command line openssl-command is at level now, however that's not how applications access certificates. Now this is where the CLI-command and library functionality differ. My box has /usr/lib64/libssl.so.10 to do the work for an application.

Looking at the SSL_CTX_use_certificate documentation, it's evident that there are functions to add a known certificate bundle (/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem in my box, symlinked via /etc/pki/tls/certs/ca-bundle.crt), or a single certificate file. Not a directory of certificates, what openssl-command does. An application has to iterate the directory and add every certificate individually. For example in Perl, HTTPS-connections are typically created via IO::Socket::SSL-library. It accempts two options: SSL_ca_path and SSL_ca_file.

As the option of modifying the ca-bundle.crt file was abandoned already, using option SSL_ca_file is out. It leaves us with SSL_ca_path, which requires every certificate hash to be symlinked to the appropriate certificate PEM-file. That's why the already done symlink won't do it completely, two additional ones are required to get IO::Socket::SSL working properly:

$ ln -s ca-bundle.crt cbf06781.0
$ ln -s ca-bundle.crt f081611a.0

Where those two hashes come from is bit complex, but here goes:
The intermediate CA certificate we downloaded, Go Daddy Secure Certificate Authority - G2, was issued by:

$ openssl x509 -noout -issuer_hash
  -in Go\ Daddy\ Secure\ Certificate\ Authority\ -\ G2.pem

... a certificate which has hash of cbf06781, which is already packed into ca-bundle.crt. Here things go weird, the Go Daddy Root Certificate Authority - G2 having hash cbf06781 is self-signed. However, during web-access that exact same certificate (with same serial number and all) is issued by a certificate having hash of f081611a. In ca-bundle.crt there is one with subject Go Daddy Class 2 Certification Authority. So, we need to add both to keep applications happy. Looks like somebody at Go Daddy really dropped the ball. Why should there be two separate CA certificates? Insane.

Actually, for example OpenSuSE Linux distro does that automatically to all bundle-certificates. The system is so stupid, that symlinkin all certificates is the only working method.

Establishing trust to the new CA root-certificate in NSS

Ok, this is the impossible part.

By lot of googling, poking around, failing, reading docs, tracing Curl, etc. I found out that there is a tool called certutil - Utility to manipulate NSS certificate databases. It seems to belong to package nss-tools. There is a man-page and some documentation at Network Security Services. But what's happening and how should I proceed remains bit foggy.

There is a /etc/pki/nssdb/, which we found in the beginning of this. That directory contains the NSS database in form of bunch of files. I found out that cert8.db and key3.db are completely obsoleted and any access methods having certutil -d /etc/pki/nssdb/ are completely useless, because they access only those files. Nobody/nothing uses those. Why are they there?

The files having effect are cert9.db and key4.db. The correct way of accessing those includes certutil -d sql:/etc/pki/nssdb. Notice the sql: -part difference. That's the part causing most confusion.

To get the certificate into the DB run command on a single line:

certutil -d sql:/etc/pki/nssdb -A -t "C,C,C"
  -n "Go Daddy Secure Certificate Authority - G2"
  -i /etc/pki/tls/certs/Go\ Daddy\ Secure\ Certificate\ Authority\ -\ G2.pem

Now your NSS DB should list:

# certutil -d sql:/etc/pki/nssdb -L

Certificate Nickname                        Trust Attributes
                                            SSL,S/MIME,JAR/XPI
Go Daddy Secure Certificate Authority - G2  C,C,C

The three Cs mean that the certificate in the DB is an authority for servers, e-mail and code signing. certutil docs say, that using 'C' for intermediate certificates is discouraged, and I didn't bother to check if that 'C' is needed at all. But having that doesn't break anything now the setup is done.

Testing

Now, running exactly the same command:

$ curl --verbose https://packetstormsecurity.net/

... will result in:

* About to connect() to dl.packetstormsecurity.net port 443 (#0)
*   Trying 198.84.60.200...
* Connected to dl.packetstormsecurity.net (198.84.60.200) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt   CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=packetstormsecurity.com,OU=Domain Control Validated
*       start date: May 31 18:04:40 2015 GMT
*       expire date: May 31 18:04:40 2016 GMT
*       common name: packetstormsecurity.com
*       issuer: CN=Go Daddy Secure Certificate Authority - G2,
               OU=http://certs.godaddy.com/repository/,
               O="GoDaddy.com, Inc.",L=Scottsdale,ST=Arizona,C=US

< HTTP/1.1 200 OK

Yesh! It works!

One hell of a thing to get fixed, but now the trust has been established so that it reaches also Curl and any applications using libcurl.

Final words

This is a multi-mess.

First: Go Daddy messes up their certs. Why isn't their Go Daddy Secure Certificate Authority - G2 in ca-bundle.crt? Why are there two version of Go Daddy Root Certificate Authority - G2?

Second: Having NSS in a Linux is insane! Nobody else is using that for certificate storage. libcurl's support for any own CAs is completely messed up and unusable.