Hacker's ramblings

I previously wrote about upgrading OS X El Capitan. After doing couple of boxes I ran into a SNAFU.

If you don't see anything in that page, that's correct! There is nothing there. It would be a safe assumption, that something had gone wrong.

Here is what web browser console says:

Error was: "Failed to load resource: The certificate for this server is invalid." As the errors were emitting from Amazon CloudFront, it didn't make any sense at all. Either Amazon had some sort of security fault happening, or I did. Unfortunately in such situations, the odds are always against me. I had upgraded couple of Macs already and had no problems with them, this box must have had something wrong with it.

My next move was to get a list of trusted root certificates shipping with an OS X. The list is available in Apple knowledgebase article HT205204. Here is what I got:

Another error: 'Safari can't verify the identity of the website "support.apple.com"'. Right. First Amazon was failing on me, then Apple. At this point I whipped up an already upgraded Mac and went for the page, this time it looked ok:

That was the proof, that something was badly off on that Mac.

For fact gathering I went trough the certificate chain of support.apple.com:

As the certificate wasn't trusted, the page looked horrible and there was no lock-icon on the address bar. The important fact here was, that the root certificate of VeriSign Class 3 Public Primary Certification Authority - G5 had version number 3 and serial number of 25 0C E8 E0 30 61 2E 9F 2B 89 F7 05 4D 7C F8 FD. On the working Mac same certificate:

A completely different serial number of 18 DA D1 98 26 7D E8 BB 4A 21 58 CD CC 6B 2B 4A.

Then the relevant question was: Why do they differ? The facts are at OS X certificate store. It so happens, that all certificates can be viewed and altered via Keychain Access -tool. I went to see the System Roots -keychain:

But that didn't solve my problem! VeriSign Class 3 Public Primary Certification Authority - G5 was there and had the proven correct serial number of 18 DA D1 98 26 7D E8 BB 4A 21 58 CD CC 6B 2B 4A. More poking around, and this is what I found:

A set of Verisign certificates on login-keychain. Weird. One of them was:

There was the 25 0C E8 E0 30 61 2E 9F 2B 89 F7 05 4D 7C F8 FD! The only appropriate action was:
And that solved it! Simply letting the weird ones go made all my websites work again.

But where did those certs come from? By googling I found Why is Symantec/Verisign CA appearing as an invalid authority? [closed] and Invalid certificate after Security Update 2015-004 in Mavericks. They both were pointing a finger to April 2015 security update. The release notes About the security content of OS X Yosemite v10.10.3 and Security Update 2015-004 say:

Certificate Trust Policy
Impact: Update to the certificate trust policy
Description: The certificate trust policy was updated. View the complete list of certificates.

I just happened to update the Mac too early and got a flawed upgrade. It is also possible, that on April, when I got bad certs, I may have gone to Verisign and manually loaded the proper root certificates in to fix my problem at the time. However, it just blew up on my face on OS X 11 upg.

If you never encountered any of this: good. I honestly don't think this issue is touching a wide audience. However, I disclosed this information for archive purposes. If something like this happens in the future, you have a clue what to look for.