TLS-fingerprinting. Yes. Most of regular users don't even have a concept for it.

Any regular Jane User browsing around her favorite websites can be "easily" identified as a ... well ... regular user. That identification is done by number of factors:

• Announced User-Agent if HTTP-request
• JavaScript-code executed, what's announced and detected by running library code for available features
• Traffic patterns, how greedy the attempt to chomp a website is
• Behaviour, timing of interactions, machines tend to be faster than humans

When putting any regular user aside and concentrating on the malicious ones, things start to get tricky. A bot can be written to emulate human to pass the detection not to raise any concerns. The only known effective method is to place a CAPTCHA for the user and make them pass it. There are known cases, where human will guide the bot trough and pass control to bot for it to continue its mischief. Given that, there's always room for new options for identifing requests.

There are number of organizations and people (including Google, Edward Snowden and US Government) to encourage people to encrypt everything. An example of this would be The HTTPS-Only Standard by US Government. Early 2020 figures indicate, that the entire Internet is ~60% HTTPS, ~93% requests arriving to Google are using HTTPS instead of non-encrypted HTTP. We're getting there, about everything is encrypted.

Short history of TLS fingerprinting

The thing with TLS-encryption is, the way the encryption is implemented can be fingerprinted. This fingerprint can be added to list of factors used to determine if you are who you say you are. A really good presentation about history TLS/SSL fingerprinting can be found from DETECTION ENGINEERING: Passive TLS Fingerprinting Experience from adopting JA3, but to summarize it here:

• 2009: Apache module mod_sslhaf
• 2012: p0f, TCP/IP fingerprinting tool
• 2015: SquareLemon TLS fingerprinting
• and the good one from Salesforce:
  2017: JA3 for clients
• 2018: JA3S for servers
• a latecomer 2019: Cisco Joy

What's missing from the list is the techniques used by various government agencies and proprietary systems. It is obvious to me, they hire bunch of super-smart individuals and some of them MUST have found the fingerprinting and never telling us about it. Point here is, there has always been methods to fingerprint your encryption, you just didn't know about it.

In this blog post I'll be focusing on JA3 as it is gaining a lot of traction in the detecting Threat Actors -field of security. This post is not about dirty details of JA3, but if you want, you can study how it works. The source code is at https://github.com/salesforce/ja3 and a video of two out of three J.A.s presenting JA3 @ Shmoocon 2018 can be found at https://youtu.be/oprPu7UIEuk?t=407. Unlike source code (which has some prerequisites), the video alone should give most of you an idea how it works.

Theory of detecting Threat Actors (TAs)

Remember, this fingerprinting originates from intrusion detection. It can be used for other purposes, but a lot of the background is from IDS. So, when studying a system for Indicators of Compromise, the theory is as follows:

A Threat Actor to change various aspects of the attack. On the bottom of the pyramid-of-pain, there are things that are trivial to alter. You can change the hash of your attack tool (if running on a system trying to detect you), or approach the target from another IP-address or change your Command&Control server to a new domain. All of these changes would throw off any attempts to detect you. Food for thought: Typical OS-commands won't change, unless upgrades are being applied. If a tool changes constantly, it's a tell.

When placing TLS-fingerprinting on the same pyramid:

Looking into one's encrypted traffic obscures some of the handy points used to detect your activity. So, neeed to shift focus to the top of the pyramid. On top there are tools and attacker's ways of working. Really difficult to change those. That's exactly where TLS-fingerprinting steps in. What if your tools leave breadcrumbs? Enough for a vigilant detection-system to realize somebody is doing probing or already has compromised the system. At minimum there is weird traffic whose fingerprint doesn't look good warranting closer look by a human.

JA3 on Wireshark

My beef with JA3 has (so far) been the fact, that my favorite network analysis tool, Wireshark, doesn't support it. Now it does:

(I took a still frame from JA3 Shmoocon presentation video and pasted Wireshark logo on top of it)

There is a Wireshark dissector for JA3. It is written in LUA by an anonymous GitHub contributor who chose not to divulge any public details of him/herself and me. Original version didn't do JA3S, and I contributed that code to the project. The Wireshark plugin is available at https://github.com/fullylegit/ja3. To get it working in Wireshark my recommendation is to install the LUA-plugin into your personal plugins-folder. In my Windows, the folder is %APPDATA%\Wireshark\plugins. Note: The sub-folder plugins\ didn't exist on my system. I created it and then placed the required .lua-files into it.

In action a JA3-fingerprint would look like this:

What we see on the Wireshark-capture is a TLS 1.2 client hello sent to a server. That particular client has JA3-hash of "66918128f1b9b03303d77c6f2eefd128". There is a public JA3-database at https://ja3er.com/, if you do any fingerprinting, go check your matches there. Based on JA3er, the previously captured hash typically identifies a Windows Chrome 79. Not suspicious, that one.

Trying to evade JA3 fingerprinting

Security is always a chase. As we already learned, lot of (bad?) people knew about this fingerprinting for many many years. They may be already ahead, we don't really know that yet.

A good primer on how to make your fingerprint match on demand is a post JA3/S Signatures and How to Avoid Them.

Let's try some manual tinkering! A simple

curl --tlsv1.2 https://ja3er.com/

Will indicate de4c3d0f370ff1dc78eccd89307bbb28 as the JA3-hash, that's a known curl/7.6x. For example a Windows Firefox 72 gets a JA3-hash of b20b44b18b853ef29ab773e921b03422, so it should be easy to mimic that, right?

Ignoring all other parts of JA3, Windows Firefox 72 will on TLS 1.2 use a 18 cipher set in prioritized order of:

1. 0x1301/4865 TLS_AES_128_GCM_SHA256
2. 0x1303/4867 TLS_CHACHA20_POLY1305_SHA256
3. 0x1302/4866 TLS_AES_256_GCM_SHA384
4. 0xc02b/49195 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
5. 0xc02f/49199 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
6. 0xcca9/52393 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
7. 0xcca8/52392 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
8. 0xc02c/49196 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
9. 0xc030/49200 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
10. 0xc00a/49162 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
11. 0xc009/49161 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
12. 0xc013/49171 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
13. 0xc014/49172 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
14. 0x0033/51 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
15. 0x0039/57 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
16. 0x002f/47 TLS_RSA_WITH_AES_128_CBC_SHA
17. 0x0035/53 TLS_RSA_WITH_AES_256_CBC_SHA
18. 0x000a/10 TLS_RSA_WITH_3DES_EDE_CBC_SHA

Info:
Full list of all TLS/SSL cipher suites is at IANA: https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4 Beware! The list is a living thing and cipher suites are known to be deprecated for old TLS-versions. Also new ones might pop.

That 18 cipher list in OpenSSL-format used by curl is:

1. TLS_AES_128_GCM_SHA256
2. TLS_CHACHA20_POLY1305_SHA256
3. TLS_AES_256_GCM_SHA384
4. ECDHE-ECDSA-AES128-GCM-SHA256
5. ECDHE-RSA-AES128-GCM-SHA256
6. ECDHE-ECDSA-CHACHA20-POLY1305
7. ECDHE-RSA-CHACHA20-POLY1305
8. ECDHE-ECDSA-AES256-GCM-SHA384
9. ECDHE-RSA-AES256-GCM-SHA384
10. ECDHE-ECDSA-AES256-SHA
11. ECDHE-ECDSA-AES128-SHA
12. ECDHE-RSA-AES128-SHA
13. ECDHE-RSA-AES256-SHA
14. DHE-RSA-AES128-SHA
15. DHE-RSA-AES256-SHA
16. AES128-SHA
17. AES256-SHA
18. DES-CBC3-SHA

Info:
My Linux curl is built to use OpenSSL for TLS and OpenSSL has a different naming for ciphers than IANA and everybody else in the known universe.

Going for a curl with --cipher and listing all of the above ciphers colon (:) separated will rather surprisingly, NOT produce what we intended. We're expecting to get full JA3 starting with:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,
but are getting a:
771,4866-4867-4865-4868-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10-255,
instead.

TLS-version of 771 matches, first three ciphers match, in WRONG order. Then curl appends a 4868 into the list. 4868, or 0x1304 is TLS_AES_128_CCM_SHA256. We didn't ask for it, but because OpenSSL knows better, it will append it to the list.

Miserable failure!

Going forward, getting all the 18 ciphers right with OpenSSL can be done. It's just too much work for me to even attempt. And I haven't got to extensions or elliptic curve parameters yet! Uff.... way too difficult to forge the fingerprint. I might have a go with that on a Python. If I have time.

Finally

This is fun stuff!

Understanding how TLS works and why it can be used as a very good indicator of your chosen tool is critical to your success. Your Tor-client WILL have a distinct fingerprint. It will be different than your Chrome, iOS Safari, Python 3 or PowerShell. Anybody having access to your encrypted traffic (like your ISP or the server you're surfing to) will be able to detect with what you're arriving.

Embrace that fact!

This is what happened to me:

A missed call from Papua New-Guinea. Well... I don't know anybody there, so they shouldn't be calling me.

It doesn't take too much investigation to realize, IT'S A SCAM!

Example: Have you been getting unexpected overseas calls? from Australia
and How to identify and report Wangiri fraud to Vodafone from UK.

The Vodafone article says:

What’s Wangiri fraud?

It’s receiving missed calls from international numbers you don’t recognise on either a mobile or a fixed-line phone.
The fraudsters generating the missed calls hope that their expensive international numbers will be called back
so that they can profit.

Looks like that scam has been going on for years. The reason is obvious, it's way too easy! Making automated calls and hanging up when the other side starts ringing doesn't cost you anything. The seriously expensive number unsuspecting victims will call back will apparently play you some music while making you wait as long as you like. Every minute the criminals will get a slice of your money.

How is this possible? How can you change the number you're calling from? Well, easy! You can do it too: https://www.spooftel.com/
"SpoofTel offers you the ability to spoof caller ID and send SMS messages. You can change what someone sees on their call display when they receive a phone call to anything you like!"

Entire world is using ancient telecommunications protocol SS#7. If you're really interested, read The Wikipedia article about it. There are number of flaws in it, as it is entirely based on the assumption only non-criminals have access to global telecommunications network. It used to hold true at the time it was created, but after that. Not so much. And that unchangeable thingie we have to thank for this and multiple other scams and security flaws.

Emotet is a nasty piece of malware. It has been around The Net for number of years now and despite all the efforts, it is still stealing money from unsuspecting victims who log in into their online bank with their computers and suddenly lose all of their money to criminals.

Last month, I bumped into a "historical" Emotet-reference. A document contains the URLs for malicious distribution endpoints of documents and binaries used to spread the malware. It also contains IPv4-addresses for Command & Control servers. There are hundreds of endpoints listed, and every single one I tested was already taken down by ISPs or appropriate government officials. Surprisingly, only 20% of the URLs were for Wordpress. Given its popularity and all the security flaws, I kinda expected the percentage to match its market share, 35% of all the websites in the entire World run Wordpress. If you're reading this in the future, I'd assume the percentage to be higher.

As a coding exercise, I analysed the listed endpoints for all three variants (or Epochs as this malware's generations are called) of Emotet and created a heatmap of them. It would be really fun to get a list of all the infected computers and list of those computers where money was stolen from, but unfortunately for my curious mind, that data wasn't available.

So, no victims, only hijacked servers in this map:

Actual Google Maps -application I wrote is at https://blog.hqcodeshop.fi/Emotet-map/map.html, go investigate it there.

This is a simple project, but if anybody want's do learn data visualization with Google Maps JavaScript API, my project is at https://github.com/HQJaTu/emotet-malware-mapper. Note: You will need an API-key from Google for your own projects. My API-key is publicly available, but restricted. It won't work for you.

As analysis of the hijacked distribution points and C2 -servers, there is lot of heat in obvious places, Europe and North America. But as you can see, there are lots of servers in use all around the globe. That should give everybody an idea why fighting cybercrime is so difficult.

Update 30th Jan 2020:
Emotet seems to be picking up speed, there is a US CISA warning about Increased Emotet Malware Activity. Apparently it is #1 malware currently in the world.

Mr. Bruce Schneier is one of the influencers I tend to follow in The Net. He is a sought after keynote speaker and does that on regular basis. More on what he does on his site @ https://www.schneier.com/.

Quite few of his keynotes, speeches and presentations are made publicly available and I regularly tend to follow up on those. Past couple of years he has given a keynote having roughly the same content. Here is what he wants you, an average Joe Internet-user to know in form of lessons. To draw this with a crayon: Lessons are from Bruce Schneier, comments are mine.

These are things that are true and are likely to be true for a while now. They are not going to change anytime soon.

Truism 1: Most software is poorly written and insecure

The market doesn't want to pay for secure software.

Good / fast / cheap - pick any two! The maket has generally picked fast & cheap over good.

Poor software is full of bugs and some bugs are vulnerabilities. So, all software contains vulnerabilities. You know this is true because your computes are updated monthly with security patches.

Whoa! I'm a software engineer and don't write my software poorly and it's not insecure. Or ... that's what I claim. On a second thought, that's how I want my world to be, but is it? I don't know!

Truism 2: Internet was never designed with security in mind

That might seem odd 2019 saying that. But back in the 1970s, 80s and 90s it was conventional wisdom.

Two basic reasons: 1) Back then Internet wasn't being used for anything important, 2) There were organizational constraints about who could access the Internet. For those two reasons designers consciously made desicisions not to add security into the fabric of Internet.

Where still living with the effects of that decision. Whether its the domain name system (comment: DNS), the Internet routing system (comment: BGP), the security of Internet packets (comment: IP), security of email (comment: SMTP) and email addresses.

This truism I know from early days in The Net. We used completely insecure protocols like Telnet or FTP. Those were later replaced by SSH and HTTPS, but still the fabric of Internet is insecure. It's the applications taking care of the security. This is by design and cannot be changed.

Obviously there are known attempts to retrofit fixes in. DNS has extension called DNSSEC, IP has extensions called IPsec, mail transfer has a secure variant of SMTPS. Not a single one of those is fully in use. All of them are accepted, even widely used, but not every single user has them in their daily lives. We still have to support both the insecure and secure Net.

Truism 3: You cannot constrain the functionality of a computer

A computer is a general purpose device. This is a difference between computers and rest of the world.

No matter how hard is tried a (landline) telephone cannot be anything other than a telephone. (A mobile phone) is, of course, a computer that makes phone calls. It can do anything.

This has ramifications for us. The designers cannot anticipate every use condition, every function. It also means these devices can be upgraded with new functionality.

Malware is a functional upgrade. It's one you didn't want, it's one that you didn't ask for, it's one that you don't need, one given to you by somebody else.

Extensibility. That's a blessing and a curse at the same time. As an example, owners of Tesla electric cars get firmware upgrades like everything else in this world. Sometimes the car learns new tricks and things it can do, but it also can be crippled with malware like any other computer. Again, there are obvious attempts to do some level of control what a computer can do at the operating system level. However, that layer is too far from the actual hardware. The hardware still can run malware, regardless of what security measure are taken at higher levels of a computer.

Truism 4: Complexity is the worst enemy of security

Internet is the most complex machine mankind has ever built, by a lot.

The defender occupies the position of interior defending a system. And the more complex the system is, the more potential attack points there are. Attacker has to find one way in, defender has to defend all of them.

We're designing systems that are getting more complex faster than our ability to secure them.

Security is getting better, but complexity is getting faster faster and we're losing ground as we keep up.

Attack is easier than defense across the board. Security testing is very hard.

As a software engineer designing complex systems, I know that to be true. A classic programmer joke is "this cannot affect that!", but due to some miraculous chain, not ever designed by anyone, everything seems to depend on everything else. Dependencies and requirements in a modern distributed system are so complex, not even the best humans can ever fully comprehend them. We tend to keep some sort of 80:20-rule in place, it's enough to understand 80% of the system as it would take too much time to gain the rest. So, we don't understand our systems, we don't understand the security of our systems. Security is done by plugging all the holes and wishing anything wouldn't happen. Unfortunately.

Truism 5: There are new vulnerabilities in the interconnections

Vulnerabilities in one thing, affect the other things.

2016: The Mirai botnet. Vulnerabilities in DVRs and CCTV cameras allowed hackers to build a DDoS system that was raid against domain name server that dropped about 20 to 30% of popular websites.

2013: Target Corporation, an USA retailer, attacked by someone who stole the credentials of their HVAC contractor.

2017: A casino in Las Vegas was hacked through their Internet-connected fish tank.

Vulnerability in Pretty Good Privacy (PGP): It was not actually vulnerability in PGP, it was a vulnerability that arose because of the way PGP handled encryption and the way modern email programs handled embedded HTML. Those two together allowed an attacker to craft an email message to modify the existing email message that he couldn't read, in such a way that when it was delivered to the recipient (the victim in this case), a copy of the plaintext could be sent to the attacker's web server.

Such a thing happens quite often. Something is broken and its nobody's fault. No single party can be pinpointed as responsible or guilty, but the fact remains: stuff gets broken because an interconnection exists. Getting the broken stuff fixed is hard or impossible.

Truism 6: Attacks always get better

They always get easier. They always get faster. They always get more powerful.

What counts as a secure password is constantly changing as just the speed of password crackers get faster.

Attackers get smarter and adapt. When you have to engineer against a tornado, you don't have to worry about tornadoes adapting to whatever defensive measures you put in place, tornadoes don't get smarter. But attackers against ATM-machines, cars and everything else do.

An expertise flows downhill. Today's top-secret NSA programs become tomorrow's PhD thesis and next day's hacker tools.

Agreed. This phenomenon can be easily observed by everybody. Encryption always gets more bits just because today's cell phone is capable of cracking last decade's passwords.

If you want to see The Man himself talking, go see Keynote by Mr. Bruce Schneier - CyCon 2018 or IBM Nordic Security Summit - Bruce Schneier | 2018, Stockholm. Good stuff there!

This blog post is about TLS-protocol. It is a constantly evolving beast. Mastering TLS is absolutely necessary, as it is one the most common and widely used components keeping our information secure in The Net.

December 2018: Lot of obsoleted TLS-stuff deprecated

Nearly an year ago, Google as the developer of Chrome browser deprecated SHA-1 hashes in TLS. In Chrome version 72 (December 2018) they actually did deprecate TLS 1 and TLS 1.1 among other things. See 72 release notes for all the details.

Making TLS 1.2 mandatory actually isn't too bad. Given Qualsys SSL Labs SSL Pulse stats:

96,0% of the servers tested (n=130.000) in SSL Labs tester support TLS 1.2. So IMHO, with reasonable confidence we're at the point where the 4% rest can be ignored.

Transition

The change was properly announced early enough and a lot of servers did support TLS 1.2 at the time of this transition, so the actual deprection went well. Not many people complained in The Net. Funnily enough, people (like me) make noise when things don't go their way and (un)fortunately in The Net that noise gets carried far. The amount of noise typically doesn't correlate anything, but phrase where there is smoke - there's fire seems to be true. In this instance, deprecating old TLS-versions and SHA-1 hashing made no smoke.

Aftermath in Chrome

Now that we have been in post SHA-1 world for a while, funny stuff starts happening. On rare occasion, this happens in Chrome:

In Chrome developer tools Security-tab has a complaint: Connection - obsolete connection settings. The details state: The server signature users SHA-1, which is obsolete. (Note this is different from the signature in the certificate.)

Seeing that warning is uncommon, other than me have bumped into that one. An example: How to change signature algorithm to SHA-2 on IIS/Plesk

I think I'm well educated in the details of TLS, but this one baffled me and lots of people. Totally!

What's not a "server signature" hash in TLS?

I did reach out to number of people regarding this problem. 100% of them suggested to check my TLS cipher settings. In TLS, a cipher suite describes the algorithms and hashes used for encryption/decryption. The encrypted blocks are authenticated with a hash-function.

Here is an example of a cipher suite used in loading a web page:

That cipher suite is code 0xC02F in TLS-protocol (aka. ECDHE-RSA-AES128-GCM-SHA256). Lists of all available cipher suites can be found for example from https://testssl.sh/openssl-iana.mapping.html.

But as I already said, the obsoletion error is not about SHA-1 hash in a cipher suite. There exists a number of cipher suites with SHA-1 hashes in them, but I wasn't using one.

Second clue is from the warning text. It gives you a hint: Note this is different from the signature in the certificate ... This error wasn't about X.509 certificate hashing either. A certificate signature looks like this:

That is an example of the X.509 certificate used in this blog. It has a SHA-256 signature in it. Again: That is a certificate signature, not a server signature the complaint is about. SHA-1 hashes in certificates were obsoleted already in 2016, the SHA-1 obsoletion of 2018 is about rest of the TLS.

What is a "server signature" hash in TLS?

As I'm not alone with this, somebody else has asked this question in Stackexchange: What's the signature structure in TLS server key exchange message? That's the power of Stackexchange in action, some super-smart person has written a proper answer to the question. From the answer I learn, that apparently TLS 1 and TLS 1.1 used MD-5 and/or SHA-1 signatures. The algorithms where hard-coded in the protocol. In TLS 1.2 the great minds doing the protocol spec decided to soft-code the chosen server signature algorithm allowing secure future options to be added. Note: Later in 2019, IETF announced Deprecating MD5 and SHA-1 signature hashes in TLS 1.2 leaving the good ones still valid.

On the wire, TLS server signature would look like this, note how SHA-1 is used to trigger the warning in Chrome:

During TLS-connection handshake, Server Key Exchange -block specifies the signature algorithm used. That particular server I used loved signing with SHA-1 making Chrome complain about the chosen (obsoleted) hash algorithm. Hard work finally paid off, I managed to isolate the problem.

Fixing server signature hash

The short version is: It really cannot be done!

There is no long version. I looked long and hard. No implementation of Apache or Nginx on a Linux or IIS on Windows have setting for server signature algorithm. Since I'm heavily into cloud-computing and working with all kinds of load balancers doing TLS offloading, I bumped into this even harder. I can barely affect available TLS protocol versions and cipher suites, but changing the server signature is out-of-reach for everybody. After studying this, I still have no idea how the server signature algorithm is chosen in common web servers like Apache or Nginx.

So, you either have this working or not and there is nothing you can do to fix. Whoa!

IRS or Internal Revenue Service, the tax authority of United States of America seems to be an endless scapegoat of scams and malware. Couple days ago, I got an email from Germany stating, that I had received a tax refund from IRS. Nice! I thought, you need to pay your taxes to USA to be eligible for an IRS tax refund. After reading the mail, I knew this particular email was very important, it was sent from Germany after all. I instantly whipped up my credit card and redied myself for punching in the required numbers ... naah! That mail was an obvious scam. Actually, IRS is warning general public about such scams in their website. Again, this type of scam is nothing new, I had a look into finer details of such a scam back in 2016, see my blog post here about that.

An interesting side note here is, the e-mail address the scammers sent the email to, was the one which got harvested from DocuSign leak of 2018. See my blog post about that here.

As 85% of such scams, the email I got has a link in it. That number comes from Proofpoint Q2 2019 Threat Report:

Malicious URLs made up 85% of global combined malicious URL and attachment message volume, a slight increase from May, but overall in-line with the trend for 2019.

What happens when you click the link, there is a login:

Username and password was in the email I received. Username was my DocuSign email and a not-too-long-or-complex random password accompanied it. I didn't hesitate, I was curious to see what the scammers cooked up this time:

Yeah. A download. What could possibly go wrong with that!

To access the good bits, I moved away from web browser, to a Linux prompt and a curl. Initially, the copied link to download.php didn't respond. Wow! A security-minded scammer. That seems to be a trend nowadays. To get past that one, I took a peek into cookies in the web browser and added a session-cookie to the curl-request. Like this:
curl --verbose --header "Cookie: sid=-da-passwrod-" http://download.php

That did the trick, a document.zip landed on my lap. No funny business there, a regular Zip-file containing a single file:

Archive: document.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
  228703  Defl:N    22214  90% 09-30-2019 13:05 68c60f7a  011239-23489234.doc

As usual, jumping too fast for conclusions: Ah, a macro-loader, I thought. To confirm:

$ file 011239-23489234.doc
011239-23489234.doc: Rich Text Format data, version 1, unknown character set

Ok. Interesting. RTF-documents don't have macros. However, I remember reading an article "Using RTF Files as a Delivery Vector for Malware" earlier. Taking a peek with hexdump:

Confirmed! There is a binary-object embedded into RTF. I borrowed the idea from SANS diary "Getting the EXE out of the RTF" and wrote a single-liner Perl:

$ perl -ne 's/([0-9A-F]{2})/print chr(hex($1))/ge' 011239-23489234.rtf > scam.bin

This one will produce extra head and tail, but also the good bits will be in my new scam.bin. Cutting the extra stuff out:

It's an XML containing a VBscript scriptlet. There is some obfuscation in it, but the Base-64 -encoded string is something, that caught my attention instantly. Decoding it resulted in the real payload URL ... which unfortunately didn't load as the website was shut down by authorities and/or vigilant sysadmin of the site. So, I never got to download my promised b.exe. Well. It most likely wouldn't contain much interesting parts in it. The binary simply messes up with your Windows and loads more "goodies" into it and poking a backdoor. No surprises, the usual.

Typically this type of scam targets the not-so-bright ones. Scammers do that intentionally, to weed out "difficult" customers. Neither the email nor IRS website-replica weren't especially elaborate work. Any self-respecting computer user should instantly recognize the scam. For scammers its always worth trying. One out of million recipients might fall into this.

"Hey! Let's be careful out there."

Recently there as been an influx of SMS-messages informing that "your parcel has arrived". The message will contain a short URL bouncing you around the net couple times and landing you to a scam-site. Finnish CERT-FI actually released a warning about the scammers operation recently. You can read the announcement in Finnish here.

Scam

The fradaulent website did look like this:

I was in process of writing this blog post and POOOOOOOF! The domain went away.

Ultimately you will land on a form to enter your information into a subscription trap:

If there would be a honest scammer, these guys are. The fine print on the right hand corner explains how the scam will work. You won't be receiving your non-existent shipment, but you will be paying initial fee of a subscription. Nice!

As usual, I did some digging about the announced website and the company. The website is still running on a Hezner VM in Germany. Since these "honest" scammers are protecting your precious GDPR-covered PII, their website even has a valid TLS-certificate issued by Comodo. The obvious motivation for using encryption is the fact that most browsers will fight back for an user entering credit card information without encrypted traffic. So, a very elaborate scam, this.

Company

Since these scammers are honest, they inform the service "responsible" company being NMS Services Ltd. I doubt that much, given the company given seems to be some sort of blank shell. A company without any activity:

There is a good chance, the above ltd. has nothing to do with this scam.

Huge footprint

The "m.o." or modus operandi of these types of scams is to have a huge footprint. They have bunch of domains and companies as it's guaranteed their entire operation will be shut down by law enforcement as they scam along.

To get their traffic to the right place, the SMS has a very short URL in it. Something with gramkl.com in it. That's a jump-site operating at Google bouncing users to lilorole.com, which is a Cloudflare forwarder. The first form result is forwarded to t.viprsp.nl, a Cloudflare forwarder, which will ultimately land at end result healthybuddy.eu, the real harvesting website at Hezner.

What we have there in the above chain, is numer of top-level-domains and bunch of companies operating under different jurisdictions. Shutting all of them down is very very very difficult. If one is shut down, they'll just pull another rabbit out of their hat and go on scamming.

Alternates

In the TLS-certificate information I found following:

Subject: OU = Domain Control Validated, OU = PositiveSSL, CN = bestplayerwins.com
X509v3 extensions:
  X509v3 Subject Alternative Name:
    DNS:bestplayerwins.com, DNS:www.bestplayerwins.com

An alternate scam site running @ bestplayerwins.com is found. Again, this scam is a honest one, it will inform that they're bullshitting your with a subscription trap.

Just by investigating the site more, there are (unsurprisingly) other domains found:

bpwbill.com and ccbill.info sites look exactly like the initial scam site:

Again they have distributed their operation to multiple domains and TLDs to make it extremely difficult to shut the entire operation down.

I did not check the details of Gibraltar-registered Future Base Ltd because of high cost of the records. Again, there is a very good chance they're not affiliated with this operation at all.

Finally

With the words of Sergeant Phil Esterhaus from Hill Street Blues:

Yesterday I stumbled on Let's Encrypt certificate on a Fedora linux. Read about my initial (failing?) approach from this blog post.

Given the ticket I filed regarding the problem, I got a pointer to study Using Shared System Certificates in Fedora. As you might think, I didn't know anything about that earlier. In a nutshell, ever since Fedora 19 was released in 2013, there has existed an unified storage for X.509 certificates with appropriate tooling to maintain certificates for OpenSSL, which is used by many components of the system, including curl. Besides the most common one, also a number of other certificate storages are included in the unified system, including NSS (for Firefox browser), GnuTLS and Java. Which is nice!

This subsystem has existed there a while, but marketing for it has been not-so-strong. Now that I know it exists, I'll offer the proper and correct way of adding the missing Let's Encrypt intermediate ISRG X3 certificate into your system.

To fix your curl, as regular user, download the X.509 certificate from Let's Encrypt website to /tmp/-folder.

$ wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt

As root, add the newly downloaded file as a trust anchor:

# trust anchor --store /tmp/letsencryptauthorityx3.pem.txt

That's it! No other commands to run. This sequence will update file /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem with the current set of certificates to be trusted. That dynamically generated file will be used by curl ia symlink /etc/pki/tls/certs/ca-bundle.crt.

As an alternative, storing the letsencryptauthorityx3.pem.txt into directory /etc/pki/ca-trust/source/anchors/, and running update-ca-trust as root will do exactly the same thing.

Simple as pie! (if you know the pie exists)

Update 8th Aug 2019:
The information in this blog post is not the entire truth and less accurate, than I'd hope for. Updated information regarding certificate system in a Fedora linux can be found from this blog post.

Now that Let's Encrypt is issuing their certificates from ISRG Root X1 / Let's Encrypt Authority X3 certificate authority, my Fedora is failing to access sites with curl. I have addressed both of these earlier in articles Let's Encrypt Transitioning to ISRG's Root and Fixing curl with Go Daddy Secure Certificate Authority G2 CA root. However, the previous fixing instructions in modern curl won't work. This is because Fedora guys won't build curl with NSS anymore, but have fallen back to standard OpenSSL used by almost everything else in Linux. Since I'm writing about it, they're doing a bad job about it.

If you'll never use curl or anything built with libcurl, you'll never notice anything. Also, if you're running curl/libcurl but are never accessing any sites with Let's Encrypt certificates, again, you won't notice it. Most of us will notice. Curl will keep spewing errors like "curl: (60) SSL certificate problem: unable to get local issuer certificate" making everybody mad.

Unlike earlier, curl run with --verbose will announce following certificate locations:

* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

It so happens, the new intermediate certificate used by Let's Encrypt isn't in the precious ca-bundle.crt file, your connection attempts will fail on a trust issue. You can run the classic OpenSSL certificate fix -operation:

# cd /etc/pki/tls/certs
# wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt
# mv letsencryptauthorityx3.pem.txt letsencryptauthorityx3.pem
# openssl rehash

... which will fix a lot in your Fedora, but it won't fix curl. Remember: it will only look inside the /etc/pki/tls/certs/ca-bundle.crt, and you did not add it there.

To get the issue solved for realz, you need to delete the symlink pointing to the certificate bundle file, cook up a new & better one. Like this:

# cd /etc/pki/tls/certs
# rm -f ca-bundle.crt
# cat /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/letsencryptauthorityx3.pem > ca-bundle.crt

Now your curl will be tamed. But it won't stick. The first time you'll get an update to RPM-package ca-certificates, your changes are gone and you'll need to re-do them. Gee! Thanks for that.

Bug report is at Curl not using CA-path - custom certificates not used. Go add your comments there to let Fedora guys know how much you appreciate their work!

Couple weeks ago, a good friend of this blog, Mr. Gorila approached with his comment (available at https://blog.hqcodeshop.fi/archives/372-comments.html#c2786). He suggested, that our beloved friends at DAD Gmbh were in some sort of financial trouble. I did some digging of my own, but given the huge handicap of me not knowing any German language nor corporate laws or practices of Germany, I found nothing. Today, Mr. Gorila approached me again with proof.

EBN scammers are Bankrupt!

Its obvious, that the people behind the scam went nowhere. They're still there and are cooking up something while trying to maintain the scam from Cyprus. There are comments from people, who received letters from debt collectors operating from Cyprus. The "debt" in question is from EBN scam and is somehow transferred from DAD Gmbh to CLB Solutions Management.

Regardless of these events, my advice is unchanged: Keep your cool and

DO NOT PAY!

There seems to be increasing amounts of proof of malpractice by DAD Gmbh. Their "debts" are completely ficticious and there is no way they can win anybody in a court of law in Germany. They need to sue your ass in Germany court, and that's there they will lose. They already did that and will keep losing. Actually, "they" don't even manage the company anymore. The company is to be dissolved by court order and liquidation is handled by somebody else than the scammers.

Proof

For those wanting to see the proof, go to https://www.bundesanzeiger.de/ and make a search for "DAD Deutscher Adressdienst". You will receive search result page:

The first result will be the official notification of company liquidation. If you click the link, the text simply says the company will be dissolved and all creditors should contact the company.

More interesting is their balance sheet from year 2017 (printed into PDF by me DAD-scam-Bundesanzeiger-DAD-balance-sheet-2017.pdf). That piece of criminal history needs to stay available as long as there is Internet and human life on Earth.

My most important findings are:

• Scammers have almost 1 million € in accounts receivable. That's ficticious "debt" from marks they scammed.
  • Its obvious, that majority of that "income" will never land at their bank accounts. DO NOT PAY!
• Company has nearly 250.000 € cash available
  • Two options here: Their liquidation hit them as a surprise or they are really bad at crime.
  • It makes no sense to leave any easy-to-move assets to the company.
  • Besides the 25.000 € initial capital, that's the profit from their crime. I imagined more.
• Their accumulated operating loss is nearly 100.000 €
  • I don't think they ever targeted for profit.
  • In a company, it is very easy to avoid taxes by never making any profit out of the operation.
• Stated in balance sheet details: There are almost 400.000€ liabilities remaining
  • Given the coarse nature of a balance sheet, I cannot deduce much from that besides they owe more money than they have.
  • That kinda explains the liquidation.

Future

They're already running a new scam. It's called European Register of Commerence or ERC. The webpage looks like this:

Its simply a spin-off from EBN-scam with a GDPR-twist.

Participate!

Anybody of you can participate in fighting these scammers. I'm maintaining the Finnish Wikipedia page for EBN scam to have a reputable source of information available with a really good Google page rank. Page is at: https://fi.wikipedia.org/wiki/European_Business_Number

You can do the same in your language!

Keep people informed about these scams. That will make the work for those scammers harder and harder. Many readers have shared information in comments of my blog, please keep doing that! There are people including me, who will investigate your clues and raise awareness of these scams and what happens in the scam'osphere.

Thank you!

Randomness in computers

You don't need to know much about computers to understand, that computers cannot do random things. Yes, all programming languages and libraries do offer you a rand()-function to emulate randomness. However, the resulting output will follow the carefully crafted programming implementing this "randomness". The most trivial pseudo-random functions will merely provide a sequence of numbers appearing random, but this sequence can be reset to start from beginning making the "randomness" predicatable. That's not really very random, huh!

Improved randomness in computers

To be fair, there does exist improved pseudo-random algorithms which take their initial seed-values from something volatile (time is one such volatile parameter) making the quality of randomness better. Still, even high-quality pseudo-random algorithm is just complex sequence of operations, which will produce duplicate results on same input values. Sometimes its just very tricky to craft a situation where all of the input values would match.

If somebody is capable of doing that, your randomness changes into predictability. Read the story of Dual_EC_DRBG on Wikipedia https://en.wikipedia.org/wiki/Dual_EC_DRBG. When you're generating your precious private keys, you don't want anybody (including NSA) to be able to guess what you have there.

Random source in Linux

Since a proper random source is something every single user, developer and sysadmin would love to have, the problem has been approached on your Linux by authors of the operating system. An excellent description can be found from Wikipedia article https://en.wikipedia.org/wiki//dev/random#Linux. Briefly put, your Linux will collect environmental entropy from number of sources (including human interaction with keyboard and mouse) to a pool, which can then be used to produce naturally random numbers. It actually works very well, the quality of randomness is top-notch.

Obvious problem with this approach is, that you cannot pull too many random numbers out of this source without exhausting it. The fix is to keep typing something while moving your mouse (not a joke!) to generate entropy for the random source. This will eventually help fill the entropy pool and /dev/random will spit couple bytes more.

Those users who have exhausted their /dev/random on an idling rack server without a console keyboard, mouse and video know that it takes painfully long for the entropy pool to fill. A busy server doing something will be able to fill the pool much faster.

A real random source

If you need a real proper random source, which works without human intervention and can provide really good randomness as a stream, there are possibilities on hardware. I know of two good ones, Simtec Electronics Entropy Key and ubld.it TrueRNG Hardware Random Number Generator.

Note: if you consider getting one, get the TrueRNG version 3 (http://ubld.it/truerng_v3). Its just that I have the 1st gen version at hand and haven't found the reason to upgrade.

My TrueRNG looks like this:

It is essentially an USB-stick.

Linux lsusb info essentially identifies it as a Microchip (vendor ID 0x04d8) manufactured USB-device (with ID 0xf5fe) providing RS-232 communications:

Bus 002 Device 009: ID 04d8:f5fe Microchip Technology, Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            2 Communications
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         8
  idVendor           0x04d8 Microchip Technology, Inc.
  idProduct          0xf5fe
  bcdDevice            1.00
  iManufacturer           1 ubld.it
  iProduct                2 TrueRNG
  iSerial                 0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0043
    bNumInterfaces          2
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              0
      CDC Header:
        bcdCDC               1.10
      CDC ACM:
        bmCapabilities       0x02
          line coding and serial state
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1
      CDC Call Management:
        bmCapabilities       0x00
        bDataInterface          1
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x000a  1x 10 bytes
        bInterval               1
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0
      bInterfaceProtocol      0
      iInterface              0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval               0
Device Status:     0x0001
  Self Powered

And by looking at /dev/, there is a /dev/ttyACM0. That's how udevd will populate a CDC-device when it sees one.

How is this a "true" random source?

Oh, that's easy. The device will produce a random 0 or 1 bit constantly when its on. Or to be precise, there is an internal algorithm producing those based on a constant flow of electrons on a transistor PN-surface. The exact phenomenon is called avalance effect or avalance breakdown. For those who can do electronics, there is a good explanation about this in Difference Between Avalanche Breakdown and Zener Breakdown (I borrowed the visualisation pic from above link).

To (over)simplify that, in a carefully constructed electronic circuit, inside a transistor an electron may or may not be emitted on the other side of a semiconducting surface. The occurrence is as random as it can be in nature. Other circuitry will detect this random flow of electrons (or lack of flow) to produce ones and zeros.

What makes this a really good for randomness, as it is well established that this avalance of electrons will happen. Also, it will happen often enough to produce a stream of events. It's just that we don't know exactly WHEN the avalance of electrons will happen. If you time-slice this to slots, a slot can be empty (no avalance) or full (electrons avalanching).

Linux tweaking:

udev

Anybody having multiple devices in their Linuxes knows, that you really cannot control which device name some specific device will get on reboot. To overcome that, udevd can be instructed to do things when it sees a device. My rules for TrueRNG include setting it to highest possible speed and creating a symlink-device so, that I can point to a known source of random. Also, I'm loosening access to that source of randomness to any users belonging to dialout-group. If I wouldn't do that, only root would have access to this fine random-source.

My /etc/udev/rules.d/99-TrueRNG.rules contains:

SUBSYSTEM=="tty", ATTRS{product}=="TrueRNG", SYMLINK+="TrueRNG", RUN+="/bin/stty raw -echo -ixoff -F /dev/%k speed 3000000"
ATTRS{idVendor}=="04d8", ATTRS{idProduct}=="f5fe", ENV{ID_MM_DEVICE_IGNORE}="1", GROUP="dialout", MODE="0664"

If you want to take your random-device for a spin, you can do something like:

dd if=/dev/TrueRNG of=random.bytes bs=64 count=1024

That would create a file of 64 KiB containing very very random bytes. In theory you can just cp data out of the character device, but since it has an infite flow, you'll need to cut it at one point.

rngd

Remember the part I said earlier about Linux using your keypresses and mouse movements as entropy source for randomness. Even with the USB-stick popped into a PC, that still remains the case. What needs to be done next is to offer a helping hand to the Linux kernel and make sure the entropy pool is always full.

My Fedora has package called rng-tools. It is packaged from Mr. Horman's https://github.com/nhorman/rng-tools. What's in there are the tools for pumping those precious truly random bits out of the USB-source to Linux kernel's entropy pool. As default, rngd will use /dev/hwrng as the source for randomness. Some Linuxes don't have that device at all, some Linuxes point that into CPU's random source. What's guaranteed, it will not point to your USB-stick! We need to change that.

Btw. you might be horrified by the fact, that something is fidding with your randomness. The exact bits transferred from USB to entropy pool won't be the actual bits getting out of /dev/random. Your keypresses and many other events are still a factor. Its still a good idea to not run randomness-monitoring malware or spyware in your Linux.

Systemd works so, that I did create a copy of /usr/lib/systemd/system/rngd.service into /etc/systemd/system/rngd.service. The contents of the copy in /etc/systemd/system/ can be freely modified and it has priority over the /usr/lib/systemd/system/ one. The only change I made was to have the ExecStart-line say as:

ExecStart=/sbin/rngd -f --rng-device=/dev/TrueRNG --fill-watermark=4000

When rngd-service would be started, it will use the USB-stick as source and make sure, there are at least 4000 bits of entropy in the pool.

Making sure rngd setup works

At any given point, you can query how many bits are available in the Linux entropy-pool:

cat /proc/sys/kernel/random/entropy_avail

Since my setup is working correctly, it will display a number greater than 4000 and smaller than 4096. The upper limit comes from /proc/sys/kernel/random/poolsize, which is a hard-coded number from Linux kernel source.

Hint: If you do the stupid thing like I did and set the /proc/sys/kernel/random/write_wakeup_threshold (using --fill-watermark) into 4096 (or above), your rngd will keep hogging CPU like there is no tomorrow. It is impossible for the pool to contain maximum number of bits at any given time. Give your system a break and set the threshold bit lower than max.

Finally

It's always nice to know for a fact, that random numbers are random. This fact can be verified and has been verified by number of other people.

Enjoy!

Over an year ago, I posted a piece regarding Let's Encrypt and specifically me starting to use their TLS certificates. This certificate operation they're running is bit weird, but since their price for a X.509 cert is right (they're free-of-charge), they're very popular (yes, very very popular) I chose to jump into their wagon.

In my previous post I had this flowchart depicting their (weird) chain of trust:

I'm not going to repeat the stuff here, just go read the post. It is simply weird to see a CHAIN of trust not being a chain. Technically a X.509 certificate can be issued by only one issuer, not by two.

Last month they announced, they're going this way:

What is this change and how will it affect me?

Read the announcement at https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html. There is a very good chance, it will not affect you. Its more like a change in internal operation of the system. The change WILL take place on 8th of July this year and any certificates issued by users like me and you will originate from a new intermediate CA using a certificate issued by ISRG's own CA. So, the change won't take place instantly. What will happen is, for the period of 90 days (the standard lifetime of a Let's Encrypt certificate) after 8th July, sites will eventually transition to these new certificates.

The part where it might have an impact to you is with legacy devices, browsers, operating systems, etc. Especially, if you're one of the unhappy legacy Android mobile users, it WILL affect you. Lot of concerned people are discussing this change at https://community.letsencrypt.org/t/please-reconsider-defaulting-to-the-isrg-root-its-unsupported-by-more-than-50-of-android-phones/91485. If you're smart enough to avoid those crappy and insecure 'droids not having any security patches by a mobile vendor who doesn't care about your security, then this will not affect you.

Testing - A peek into the future

The way Let's Encrypt does their business is weird. I have stated that opinion a number of times. However, as they are weird, they are not incompetent. A test site has existed for a long time at https://valid-isrgrootx1.letsencrypt.org/, where you can see what will happen after July 8th 2019.

On any browser, platform or device I did my testing: everything worked without problems. List of tested browsers, platforms and devices will include:

• Anything regular you might have on your devices
• Safari on multiple Apple iOS 12 mobile devices
• Chrome on Huawei Honor P9 running Android 8
• Microsoft Edge Insider (the Chrome-based browser)
• Safari Technology Preview (macOS Safari early release branch)
• Firefox 66 on Linux
• Curl 7.61 on Linux

An attempt to explain the flowchart

Yes, any topic addressing certificates, cryptography and their applications is always on the complex side of things. Most likely this explanation of mine won't clarify anything, but give me some credit for trying.

In an attempt to de-chiper the flowchart and put some sense to a chain-not-being-a-chain-but-a-net, let's take a look into the certificate chain details. Some things there are exactly the same, some things there are completely different.

Chain for an old certificate, issued before 8th July

Command: openssl s_client -showcerts -connect letsencrypt.org:443

Will output:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = www.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIHMjCCBhqgAwIBAgISA2YsTPFE5kGjwkzWw/oSohXVMA0GCSqGSIb3DQEBCwUA
...
gRNK7nhFsbBSxaKqLaSCVPak8siUFg==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
...
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = www.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
...

Chain for a new certificate, issued after 8th July

Command: openssl s_client -showcerts -connect valid-isrgrootx1.letsencrypt.org:443

Will output:

CONNECTED(00000003)
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFdjCCBF6gAwIBAgISA9d+OzySGokxL64OccKNS948MA0GCSqGSIb3DQEBCwUA
...
yEnNbd5O8Iz2Nw==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFjTCCA3WgAwIBAgIRANOxciY0IzLc9AUoUSrsnGowDQYJKoZIhvcNAQELBQAw
...
rUCGwbCUDI0mxadJ3Bz4WxR6fyNpBK2yAinWEsikxqEt
-----END CERTIFICATE-----
---
Server certificate
subject=CN = valid-isrgrootx1.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
...

But look the same!

Not really, looks can be deceiving. It's always in the details. The important difference is in level 1 certificate, having different issuer. Given cryptographny, the actual bytes transmitted on the wire are of course different as any minor change in a cert details will result in a completely different result.

As a not-so-important fact, the level 0 certificates are complately different, because they are for different web site.

Then again the level 1 certs are the same, both do have the same private key. There is no difference in the 2048-bit RSA modulus:

Certificate:
  Data:
    Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
    Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Subject Public Key Info:
      Public Key Algorithm: rsaEncryption
        RSA Public-Key: (2048 bit)
        Modulus:
          00:9c:d3:0c:f0:5a:e5:2e:47:b7:72:5d:37:83:b3:
          68:63:30:ea:d7:35:26:19:25:e1:bd:be:35:f1:70:
          92:2f:b7:b8:4b:41:05:ab:a9:9e:35:08:58:ec:b1:
          2a:c4:68:87:0b:a3:e3:75:e4:e6:f3:a7:62:71:ba:
          79:81:60:1f:d7:91:9a:9f:f3:d0:78:67:71:c8:69:
          0e:95:91:cf:fe:e6:99:e9:60:3c:48:cc:7e:ca:4d:
          77:12:24:9d:47:1b:5a:eb:b9:ec:1e:37:00:1c:9c:
          ac:7b:a7:05:ea:ce:4a:eb:bd:41:e5:36:98:b9:cb:
          fd:6d:3c:96:68:df:23:2a:42:90:0c:86:74:67:c8:
          7f:a5:9a:b8:52:61:14:13:3f:65:e9:82:87:cb:db:
          fa:0e:56:f6:86:89:f3:85:3f:97:86:af:b0:dc:1a:
          ef:6b:0d:95:16:7d:c4:2b:a0:65:b2:99:04:36:75:
          80:6b:ac:4a:f3:1b:90:49:78:2f:a2:96:4f:2a:20:
          25:29:04:c6:74:c0:d0:31:cd:8f:31:38:95:16:ba:
          a8:33:b8:43:f1:b1:1f:c3:30:7f:a2:79:31:13:3d:
          2d:36:f8:e3:fc:f2:33:6a:b9:39:31:c5:af:c4:8d:
          0d:1d:64:16:33:aa:fa:84:29:b6:d4:0b:c0:d8:7d:
          c3:93

New certificate from a GUI

Obviously, the same thing can be observed from your favorite browser. Not everybody loves doing most of the really important things from a command-line-interface.

Old one, having DST-root:

New one, having ISRG-root:

Completely different. Same, but different.

Confusion - Bug in Firefox

Chrome, correctly displaying DST-root for www.letsencypt.org:

Firefox failing to follow the certificate chain correctly for the same site:

This is the stuff where everybody (including me) gets confused. Firefox fails to display the correct issuer information! For some reason, it already displays the new intermediate and root information for https://letsencrypt.org/.

Most likely this happens because both of the intermediate CA certificates are using the same private key. I don't know this for sure, but it is entirely possible, that Mozilla TLS-stack is storing information per private key and something is lost during processing. Another explanation might be, that lot of Mozilla guys do work closely with Let's Encrypt and they have hard-coded the chain into Firefox.

Finally

Confused?

Naah. Just ignore this change. This is Internet! There is constantly something changing.

I've been receiving couple of these already:

Hello!

This is important information for you!

Some months ago I hacked your OS and got full access to your account one-of-my-emails@redacted
On day of hack your account one-of-my-emails@redacted has password: m7wgwpr7

So, you can change the password, yes.. Or already changed... But my malware intercepts it every time.

How I made it:
In the software of the router, through which you went online, was a vulnerability. I used it...
If you interested you can read about it: CVE-2019-1663 - a vulnerability in the web-based management interface of the Cisco routers. I just hacked this router and placed my malicious code on it. When you went online, my trojan was installed on the OS of your device.

After that, I made a full backup of your disk (I have all your address book, history of viewing sites, all files, phone numbers and addresses of all your contacts).

A month ago, I wanted to lock your device and ask for a not big amount of btc to unlock. But I looked at the sites that you regularly visit, and I was shocked by what I saw!!! I'm talk you about sites for adults.

I want to say - you are a BIG pervert. Your fantasy is shifted far away from the normal course!

And I got an idea.... I made a screenshot of the adult sites where you have fun (do you understand what it is about, huh?). After that, I made a screenshot of your joys (using the camera of your device) and glued them together. Turned out amazing! You are so spectacular!

I'm know that you would not like to show these screenshots to your friends, relatives or colleagues.
I think $748 is a very, very small amount for my silence. Besides, I have been spying on you for so long, having spent a lot of time!

Pay ONLY in Bitcoins!

Note: That is only the beginning of long rambling to make me convinced this is for real.

The good thing is, that I finally found my lost password. As stated in the scam mail, it is: m7wgwpr7
Wait a minute! Nobody should post their passwords publicly! No worries, I'll post all of mine. You can find them from https://github.com/danielmiessler/SecLists/blob/master/Passwords/Common-Credentials/10-million-password-list-top-1000000.txt As a careful Internet user, I only use passwords from that one million -list.

Also, thank you for informing me about a security flaw allowing remote code execution in my Cisco RV110W Wireless-N VPN Firewall, Cisco RV130W Wireless-N Multifunction VPN Router or Cisco RV215W Wireless-N VPN Router. I don't know exactly which one of those I have. I may need to re-read CVE-2019-1663 Detail again for details.

Good thing you mentioned, that you are capable of intercepting my attempts of changing my account password. I'm too scared to use Internet anywhere else than through my Cisco <whatever the model was> router. I'll never use my cell phone, office network or any public Wi-Fi for Internet access.

Nice job on finding the camera in one of my desktop PCs. I personally haven't found any in them yet!
Also I'm happy to know, that now somebody has backups of all my Windowses, Macs and Linuxes I use on regular basis. Including my NAS, that's nearly 20 terabytes of data! Transferring all that out of my Internet connections without me noticing anything is really a feat. Congrats on that one!

PS. Are you for real!

Whoa! I finally hit the long awaited 200 point mark in Stack Overflow. The good part about that is, Stack Exchange starts publishing a public badge they'll call "flair". They do this to make it possible for me to publicly boast about my prowess. To get some public boasting going on, my flair looks like this:

All of you Peeping Toms out there, I'll save you couple keystrokes of googling: My  public user profile is at https://stackexchange.com/users/1684769/hqjatu

Without thinking it too much, I just went for a SSH-connection on a Windows 10 box with the way it's typically done:

ssh -i .ssh/id_nistp521 user@linux-box

When the result was not a private key passphrase question, but a: Bad owner or permissions on .ssh/config, I started thinking about it. What! What? Why! It worked earlier, why won't it anymore?

For the record:
I am an avid Cygwin user. So, I get a very very Linuxish experience also on my Windows-boxes. So, don't be confused with the rest of the story. This is on a Windows 10, even if it doesn't appear to be so.

Back to the failure. I started poking around the permissions. SSH-clients are picky on private key permissions (also config file), so I thought that something was going on. Doing a ls -l:

-rw------- 1 jari None 547 Nov 12 17:25 config
-rw------- 1 jari None 444 Jul 15  2017 id_nistp521

Nope. Nothing wrong with that. Still, something HAD to be wrong, as this wasn't acceptable for the client. Doing a which ssh gave it away. I was expecting to see /usr/bin/ssh, not /cygdrive/c/Windows/System32/OpenSSH/ssh!!

So, who put an OpenSSH client to my Windows-directory? Why? When? What kind of sorcery is that? Environment variables:

Yes, there is a PATH-entry for OpenSSH before Cygwin. In the directory, there is a full set of OpenSSH-tools:

Version is 7.6:

OpenSSH_for_Windows_7.6p1, LibreSSL 2.6.4

The expected version in Cygwin is 7.9:

OpenSSH_7.9p1, OpenSSL 1.0.2p  14 Aug 2018

So, to fix this, I yanked the PATH-entry away. Now my SSH-connections worked as expected.

Little bit of googling around landed me on an article OpenSSH in Windows 10! in MSDN blogs. This is on January 2018 and apparently this stuff landed on my Windows in April update (build 1803). Also in article What’s new for the Command Line in Windows 10 version 1803, I found out that also tar and curl were added.

Ultimately this is a good thing. Now that I know this stuff is there as default, there is no need to go load a PuTTY or something for a random SSH-thing you just want to get fixed on a remote box.