Security

Little bit of background about having certificates

A quote from https://www.brightedge.com/blog/http-https-and-seo/:

Google called for “HTTPS Everywhere” (secure search) at its I/O conference in June 2014 with its Webmaster Trends Analyst Pierre Far stating: “We want to convince you that all communications should be secure by default”

So, anybody with any sense in their head have moved to having their website prefer HTTPS as the communication protocol. For that to happen, a SSL certificate is required. In practice any X.509 would do the trick of encryption, but anybody visiting your website would get all kinds of warnings about that. An excellent website having failing certificates is https://badssl.com/. The precise error for you would see having a randomly selected certificate can be demonstrated at https://untrusted-root.badssl.com/.

Google, as the industry leader, has taken a huge role in driving certificate business to a direction it seems fit. They're hosting the most used website (google.com, according to https://en.wikipedia.org/wiki/List_of_most_popular_websites) and  the most used web browser (Chrome, according to https://en.wikipedia.org/wiki/Usage_share_of_web_browsers). So, when they say something, it has a major impact to the Internet.

What they have said, is to start using secured HTTP for communications. There is an entire web page by Google about Marking HTTP As Non-Secure, having the timeline of how every single website needs to use HTTPS or risk being undervalued by GoogleBot and being flagged as insecure to web browsing audience.

Little bit of background about what certificates do

Since people publishing their stuff to the Net, like me, don't want to be downvalued or flagged as insecure, having a certificate is kinda mandatory. And that's what I did. Couple years ago, in fact.

I have no interest in paying the huge bucks for the properly validated certificates, I simply went for the cheapest possible Domain Validated (DV) cert. All validation types are described in https://casecurity.org/2013/08/07/what-are-the-different-types-of-ssl-certificates/. The reasoning, why in my opinion, those different verification types are completely bogus can be found from my blog post from 2013, HTTP Secure: Is Internet really broken?. The quote from sslshopper.com is:
"SSL certificates provide one thing, and one thing only: Encryption between the two ends using the certificate."

Nowhere in the technical specifcation of certificates, you can find anything related to actually identifying the other party you're encrypting your traffic with. A X.509 certificate has attributes in it, which may suggest that the other party is who the certificate says to be, but's an assumption at best. There is simply no way of you KNOWING it. What the SSL certificate industry wants you to believe, is that they doing all kinds of expensive verification makes your communications more secure. In reality it's just smoke and mirrors, a hoax. Your communications are as well encrypted using the cheapest or most expensive certificate.

Example:
You can steal a SSL certificate from Google.com and set up your own website having that as your certificate. It doesn't make your website Google, even the certificate so suggests.

Little bit of background about Symantec failing to do certificates

Nobody from Symantec or its affiliates informed me about this. Given, that I follow security scene and bumped into news about a dispute between Google and Symantec. This article is from 2015 in The Register: Fuming Google tears Symantec a new one over rogue SSL certs. A quote from the article says:

On October 12, Symantec said they had found that another 164 rogue certificates had been issued
in 76 domains without permission, and 2,458 certificates were issued for domains that were never registered.

"It's obviously concerning that a certificate authority would have such a long-running issue
and that they would be unable to assess its scope after being alerted to it and conducting an audit,"

So, this isn't anything new here. This is what all those years of fighting resulted as: Replace Your Symantec SSL/TLS Certificates:

Near the end of July 2017, Google Chrome created a plan to first reduce and then remove trust (by showing security warnings in the Chrome browser) of all Symantec, Thawte, GeoTrust, and RapidSSL-issued SSL/TLS certificates.

And: 23,000 HTTPS certs will be axed in next 24 hours after private keys leak.

In short:
They really dropped the ball. First they issued 164 certificates, which nobody actually ordered from them. Those rogue certificates included one for google.com. Then they somehow "lost" 23k private keys for already issued certificates.

That's really unacceptable for a company by their own words is "Global Leader In Next-Generation Cyber Security". That's what Symantec website https://www.symantec.com/ says, still today.

What next?

Symantec has a website called Check your website for Chrome distrust at https://www.websecurity.symantec.com/support/ssl-checker.

I did check the cert of this blog, and yup. It flagged the certificate as one needing immediate replacement. The certificate details have:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            24:6f:ae:e0:bf:16:8d:e5:7a:13:fb:bd:1e:1f:8d:a1
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=GeoTrust Inc., CN=RapidSSL SHA256 CA
        Validity
            Not Before: Nov 25 00:00:00 2017 GMT
            Not After : Feb 23 23:59:59 2021 GMT
        Subject: CN=blog.hqcodeshop.fi

That's a GeoTrust Inc. issued certificate. GeoTrust is a subsidiary of Symantec. I did study the history of Symantec' certificate business and back in 2010 they acquired Verisign's certificate business resulting as ownership of Thawte and GeoTrust. RapidSSL is the el-cheapo brand of GeoTrust.

As instructed, I just re-issued the existing certificate. It resulted in:

Now, my certificate traces back to a DigiCert CA root.

That's all good. I and you can continue browsing my blog without unnecessary this-website-is-not-secure -warnings.

I got and ad from Microsoft about a security summit they were organizing. Since it was virtual, I didn't have to travel anywhere and the agenda looked interesting, I signed up.

Quotes:

Discussion points:

• Security hygiene
  • Run patches to make your stuff up-to-date
  • Control user's access
  • Invest into your security, to make attackers ROI low enough to attack somebody else
• Security is a team sport!
  • Entire industry needs to share and participate
  • Law enforcement globally needs to participate
• Attacks are getting more sophisticated.
  • 90% of cybercrime start from a sophisticated phishing mail
  • When breached, new malware can steal domain admin's credentials and infect secured machines also.
  • Command & control traffic can utilize stolen user credentials and corporate VPN to pass trough firewall.
• Attackers are financially motivated.
  • Ransomware
  • Bitcoin mining
  • Petaya/Notpetaya being an exception, it just caused massive destruction
• Identity is the perimeter to protect
  • Things are in the cloud, there is no perimeter
  • Is the person logging in really who he/she claims to be?
  • Enabling 2-factor authentication is vital

Finally:

A brief history of CAcert.org

For almost two decades, my primary source for non-public facing X.509 certificates has been CAcert.org. They were one of the first ever orgs handing out free-of-charge certificates to anybody who wanted one. Naturally, you had to pass a simple verification to prove that you actually could control the domain you were applying a certificate for. When you did that, you could issue certificates with multiple hostnames or even wildcard certificates. And all that costing nothing!

The obvious problem with CAcert.org always was, that they were not included in any of the major web browsers. Their inclusion list at https://wiki.cacert.org/InclusionStatus is a sad read. It doesn't even have Google Chrome, the most popular browser of current time in the list. The list simply hasn't been updated during the lifetime of Chrome! On a second thought looking it bit closer, the browser inclusion status -list is an accurate statement how CAcert.org is doing today. Not so good.

Wikipedia page https://en.wikipedia.org/wiki/CAcert.org has a brief history. Their root certificate was included in initial versions of Mozilla Firefox. When CA/Browser Forum was formed back in 2005 by some of the certificate business vendors of that time to have a set of rules and policies in place regarding web site certificates and the issuence policies, they kicked pretty much everybody out by default. Commerical vendors made the cut back in, but CAcert.org simply couldn't (or wouldn't) comply with those and withdrew their application for membership. The reson for CAcert.org not being able to act on anything is that the entire org is (and has been for a long time) pretty much dead. All the key persons are simply busy doing something else.

Today, the current status of CAcert.org is, that their certs are not trusted and signed by non-accepted hash algorithms. Over an year ago, there was a blip of activity and they managed to re-sign their root certificate with SHA-256, but to me it looks like they exhausted all the energy on the actual signing and newly signed root certs were never published. I wrote a post about how to actually get your hands on the new root certificate and install that to your machines.

Today, when CA/Browser Forum is mostly controlled by Google and Mozilla, a stalled CAcert.org would not be accepted as a member and at the same time there is huge pressure to start using properly signed and hashed certificates on all web traffic, I've run out of road with CAcert.org. So, for me, it's time to move on!

A brief history of Let's Encrypt

Two years ago, the certificate business had been hit hard. I collected a number of failures in my blog post about What's wrong with HTTPS. Roughly at the same time also Electronic Frontier Foundation (EFF) saw the situation as not acceptable. Businesses wanted serious money for their certs, but were not doing an especially good job about trustworthy business practices. With help of some major players (including Mozilla Foundation, Akamai Technologies, Cisco Systems, etc.) they created a non-profit organization for the sole purpose of issuing free-of-charge certificates to anybody who wanted one called Let's Encrypt.

They managed to pull that off, in a very short period of time, they become the most prominent certificate authority in the Internet. At least their product price is right, €0 for a cert. And you can request and get as many as you want to. There are some rate limits, so you cannot denial-of-service them, but for any practical uses, you can get all your certs from them. For free!

The practical benefit for Let's Encrypt operation is, that number of web server in The Net having HTTPS enabled has been rising. Both EFF and CA/Browser Forum is strongly suggesting, that all HTTP-traffic should be encrypted, aka. HTTPS. But the obvious hurdle in achieving that, is that everybody needs to have a certificate in their web server to enable encryption. Given Let's Encrypt, now everybody can do that! EFF has stated for long time, that having secure communications shouldn't be about money, it should be about your simply wanting to do that. The obvious next move is, that in coming years CAB Forum will announce, that all web traffic MUST be encrypted. However, we're not quite yet there.

Breifly on Let's Encrypt tech

Since they wanted to disrupt the certificate business, they abandoned current operation procedures. Their target is to run the entire request/issue-process fully automated and do that in a secure manner. To achieve that, they created a completely new protocol for Automatic Certificate Management Environment, or ACME. The RFC draft can be seen at https://datatracker.ietf.org/doc/draft-ietf-acme-acme/.

Their chain of trust (taken from https://letsencrypt.org/certificates/) is bit confusing to me:

As the root certificate, they use Digital Signature Trust Co. (red line with Identrust DST Root CA X3). That's something your browser has had for years. So, ultimately they didn't have to add anything. When you request a certificate, it is issued by the intermediate authority Let's Encrypt Authority X3. And as the bottom level is your very own Server Certificate.

However, I don't undertand why there is ISRG Root X1 to double-sign all the intermediates, and signing only their OCSP-data. My computers don't have that root certificate installed. So, what's the point of that?

As a note:
This is the recommened and typical way of setting up your certificate chain, nothing funny about that. For a layman, it will strike as overly complex, but in X.509-world things are done like that for security reasons and being able to "burn" an intermediate CA in a split second and roll forward to a new one without interruptions and need to install anything to your machines.

What the actual connection between Let's Encrypt and Digital Signature Trust Co. (IdenTrust) is unclear to me. If somebody knows, please drop a comment clarifying that.

My beef with Let's Encrypt

Short version: It's run by idiots! So, Let's not.

Long version:

• Tooling is seriously bad.
  • The "official" python-based software won't work on any of my machines. And I don't want to try fixing their shit. I just let those pieces of crap rot.
• To me, Acme Inc. is heavily associated with Wile E Coyote
  • Mr. Coyote never managed to catch the Road Runners. Many times it was because the Acme Inc. manufactured equipment failed to function. https://en.wikipedia.org/wiki/Acme_Corporation
    
• Did I mention about the bad tools?
  • Every single tool I've ever laid my hands on wants to mess up my Apache and/or Nginx configuration.
  • I have a very simple rule about my configurations: DON'T TOUCH THEM! If you want to touch them, ask me first (at which point I can reject the request).
  • There seems to be no feasible way of running those crappy tools without them trying to break my configs first.
• Most 3rd-party written libraries are really bad
  • There are libraries/tools for all imaginable programming languages implementing parts/all of ACME-protocol.
  • Every single one of them are either bad or worse.
• Certificate life-span i 90 days. That's three (3) months!
  • The disruptive concept behind this is, that you need to renew your certs every 60 days (or at max 90 days). For that to happen, you will need to learn automation. Noble thought, that. Not so easy to implement for all possible usages. The good thing is, that if ever achieved, you won't have to suffer from certificates expiring without you knowing about it.
  • As an alternative, you can get a completely free-of-charge SSL certificate from Comodo, a member of Let's Encrypt, which is valid for 90 days, but you have to do it manually using Comodo's web GUI. See: https://ssl.comodo.com/free-ssl-certificate.php if you want one.
  • I won't list those commercial CAs who can issue you a 30 day trial certificate for free here, because the list is too long. So, if you just want a short-lived cert, you have been able to that for a long time.
• They're not able to issue wildcard certificates, yet.
  • This is on the works, so mentioning this is hitting them below the belt. Sorry about that.
  • From their test-API, you can get a wildcard certificate, if you really, really, really need one. It won't be trusted by your browser, but it will be a real wildcard cert.
  • Original release date for ACME v2 API in production was 27th Feb 2018, but for <insert explanation here> they were unable to make their own deadline. New schedule is during Q1/2018. See the Upcoming Features -page for details.

My solution

It's obvious, that I need to ditch the CAcert.org and the only viable one is Let's Encrypt (run by idiots). To make this bitter choice work for me, after evaluating a number of solutions, I found a reasonable/acceptable tool called acme.sh. This tool is written with Bash and it uses curl for the ACME access. The author publishes the entire package in Github, and I have my own fork of that, which doesn't have a line of code even thinking about touching my web server configurations. You can get that at https://github.com/HQJaTu/acme.sh

Since I love Rackspace for their cloud DNS. I even wrote a Rackspace API Cloud DNS -plugin for acme.sh to make my automated certificate renewals work seamlessly. After carefully removing all other options for domain verification, it is 100% obvious, that I will do all of my ACME domain verifications only via DNS. Actually, for wildcard certs, it is the only allowed approach. Also, some of the certificates I'm using are for appliance, which I carefully firewall out of the wild wild web. Those ridiculous web site verification wouldn't work for me anyway.

And for those, who are wondering, Why Rackspace Cloud DNS? The answer is simple: price. They charge for $0/domain. That is unlike most cloud/DNS service providers, who want actual money for their service. With Rackspace you'll get premium GUI, premium API, premium anycast DNS servers with the right price, free-of-charge. You will need to enter a valid credit card when you create the account, but as they state: they won't charge it unless you subscribe to a paid service. I've been running their DNS for years, and they never charged me once. (If they see this blog post, they probably will!)

What I'm anxiously waiting is the ACME v2 API release, which is due any day now. That will allow me to get my precious wildcard certificates using these new scripts.

Now that my chips are in, its just me converting my systems to use the Wile E Coyote stuff for getting/renewing certs. And I will need to add tons of automation with Bash, Perl, Ansible, Saltstack, ... whatever to keep the actual servers running as intened. Most probably, I will post some of my automations in this blog. Stay tuned for those!

Update 25th June 2019: EBN scammers bankrupt

The text below is a comment from Mr.(?) Gorila to my previous EBN scam post, he kindly translated a German article from year 2011 to English reading audience. Given the length of the text, I'm posting the un-altered comment here. I did add the emphasis for subtitles to make the article easier to read.

Why this is important, is of course the legal precedent. EBN-scammers sued somebody and lost!
So: DO NOT PAY! You will win your case in court.

So, to repeat, the text below is not mine, but I think it being very valuable for the people following EBN scam case.

Indeed, Legal German language is very difficult to translate into English. However, in 2010 there were many German articles addressing this court decision. Language of journal articles is simpler and easier to understand. Yet that article is informative and precise enough to conwey the court ruling message to general public.

Here is a good one, with translation below (Please note, that the translation is not literal to avoid German idioms and phrases inconsistent with English language):
http://www.kostenlose-urteile.de/LG-Hamburg_309-S-6610_LG-Hamburg-zu-Branchenbuchabzocke-Eintragungsformular-Datenaktualisierung-2008-des-DAD-Deutscher-Adressdienst-erfuellt-Straftatbestand-des-Betrugs.news11513.htm

Judgement of Regional Court Hamburg (Urteil vom 14.01.2011 - 309 S 66/10)

Regional Court Hamburg on Business directory rip-off:
Registration form "Data update 2008" of the DAD German Address Service constitutes criminal offense of the fraud

Due to an overall view of the court considers the intent to be deceive

The district court Hamburg has confirmed the complaint of a customer in second instance, who had sued against the DAD Deutscher Adress Dienst. This had taken the customer into the Internet address register at www.DeutschesInternetRegister.de, without making it clear that the entry was subject to a charge. The customer should pay 2,280.04 euros for the entry. The customer hired a lawyer with whom he went to the court. There, he sued for decision that he was not required to pay and for reimbursement of his legal fees. The district court Hamburg Barmbek gave the customer right. The district court Hamburg confirmed the judgment in the appeal.

Defendant is a business directory DAD with about 1.2 million registered companies. The vast majority of the registered are of free entries, which DAD has copied from publicly available sources. The customer received a letter from DAD, entittled "Data Update 2008". The letter requested a review of the existing and updated if necessary. It was also said: "The registration and updating of your basic data is free."

Only at the end of the form was an indication of the costs

An employee of the company then entered missing data on the pre-printed form and sent it to DAD as did a large number of authorities and tradespeople. Latter the company received an invoice for 2,280.04 euros for the entry with reference to a cost indication in the lower quarter of the form (annual price of 958 EUR plus VAT).

Deception about actual costs is fraud

The district court Hamburg evaluated this in its judgment as fraud. This does not change the fact that DAD has quite specifically indicated the cost-bearing nature of the offer in the letter. Rather, it is decisive that the possible act of deception in fraud is not only the pretending of false facts or the disfiguration or concealment of existing facts. Moreover, any behavior other than deception is also considered, provided that it may provoke a mistake on the part of the other person and influence the decision to make the desired declaration of intent.

Deception exists when the victim of fraud, knowing all the circumstances, would act differently

On the other hand, it was not decisive whether the deceived person followed the care required in the course of business dealings or even acted negligently with regard to the ommission of certain contractual information. Insofar as the error on the part of the customer has been triggered by a legally relevant deception. The customers claims do not fail because the error is caused by his own negligence in dealing with advertising mail.

The nature and design of the form produce erroneous ideas

In particular, in cases where the author of a contract offers, by presentation and formulation, a kind of design, which should cause the addressee has erroneous ideas about the actual supplied parameters. A deception can be assumed even if the true character of the letter could be recognized after careful reading. This also follows from a judgment of the Federal Court of 26.04.2001, Az. 4 StR 439/00. The respective deception must have been used according to plan and was not merely a consequence, but the purpose of action.

Costs notice at the end of the form could be overlooked by the customers

According to the Federal Court of Justice, in the case of a merely misleading presentation in the offer letter, it is above all a matter of how strongly significant contractual parameters are presented, distorted or edited. In the present case of DAD, the non-binding appearance of the request for review and correction of well-known data can caused that the price will be at least overlooked by some customers.

Form gave the impression of already existing contractual relations

Finally, another indication of the intended deception was that the form had already been pre-filled with the customer's data. Such an approach was apt to give the recipient the impression that it was not a novel business relationship but that it was intended to maintain or extend an existing contractual relationship.

Simple online entries usually cost no 2,280 euros

It is also crucial that none of the addressee with a total cost of over 1,900 euros for a simple online registration have to expect. With this reasoning also the district court Heilbronn decided by resolution of 23.06.2010, Az. 3 S 19/10 in a similarly stored case.