A completely new type of spam has been flooding my mailbox. Ok, not flooding, but during past week I've got 7 different ones. The general idea for this spam delivery method is for the spam to originate from Google. How in detail the operation works, is to either exploit some innocent person's Google Account or create a ton of brand new Google Accounts to be used briefly and then thrown away. What the scammers do with the account is on Google Drive they'll create a presentation. There is no content in the presentation, it will be completely empty and then they'll share the document with me. Ingenious!

Shared presentation looks like this (hint: its completely blank):

The trick is in the comment of the share. If you add a new user to work on the same shared file, you can add own input. These guys put some spam into it.

When the mail arrives, it would contain something like this:

This approach will very likely pass a lot of different types of spam-filtering. People work with shared Google Drive documents all the time as their daily business and those share indications are not spam, its just day-to-day business for most.

Highlights from the mail headers:

Return-Path: <3FDxcYBAPAAcjvttlu0z-uvylws5kvjz.nvvnsl.jvt@docos.bounces.google.com>
Received-SPF: Pass (mailfrom) identity=mailfrom;
    client-ip=209.85.166.198; helo=mail-il1-f198.google.com;
    envelope-from=3fdxcybapaacjvttlu0z-uvylws5kvjz.nvvnsl.jvt@docos.bounces.google.com;
    receiver=<UNKNOWN>
DKIM-Filter: OpenDKIM Filter v2.11.0 my-linux-box.example.com DF19A80A6D5
Authentication-Results: my-linux-box.example.com;
    dkim=pass (2048-bit key) header.d=docs.google.com header.i=@docs.google.com header.b="JIWiIIIU"
Received: from mail-il1-f198.google.com (mail-il1-f198.google.com [209.85.166.198])
    (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
     key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
    (No client certificate requested)
    by my-linux-box.example.com (Postfix) with ESMTPS id DF19A80A6D5
    for <me@example.com>; Thu, 25 Mar 2021 09:30:30 +0200 (EET)
Received: by mail-il1-f198.google.com with SMTP id o7so3481129ilt.5
    for <me@example.com>; Thu, 25 Mar 2021 00:30:30 -0700 (PDT)
Reply-to: No Reply <p+noreply@docs.google.com>/code>

Briefly for those not fluent with RFC 821:
Nothing in the mail headers would indicate scam, fraud or even a whiff of spam. It's a fully legit, digitally signed (DKIM) email arriving via encrypted transport (TLS) from a Google-designated SMTP-server (SPF),

Given trusted source of mail, the only feasible attempt to detect this type of spam is via content analysis. Note: as an example of detecting and blocking unsolicited email, I've past written my thoughts how easy it is to block spam.

Well, until now it was. Darn!

Privacy, especially your/mine/everybody's, has been an always actual topic ever since The Internet begun its commercial expansion in the beginning of 90s. Being an important topic, it has been thought everybody should have the right for privacy. In EU, the latest regulation is called on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, or General Data Protection Regulation. In California, USA they'd call it California Consumer Privacy Act.

Privacy protection gone wrong

Both above mentioned regulations have good intention. Forcing websites to bombard all of their users with stupid questions are a complete waste of bandwidth! Everybody is tired of seeing popups like this while surfing:

Somehow the good intention turned on itself. Literally nobody wants to make those choices before entering a site. There should be a setting in my browser and all those sites should read the setting and act on it without bothering me.

Mobile phone tracking

Meanwhile, your cell service provider is using your data and you won't be offered a set of checkboxes to tick.

As an example, Telia’s anonymized location data helps Finnish Government fight the coronavirus (April 2020). This corporation has a product called Crowd Insights. Anybody with a reasonable amount of money (actual price of the service is not known) can purchase location data of actual persons moving around a city. There is a brief investigation by Finnish Chancellor of Justice stating that the service is legal and won't divulge any protected data. The decision (unfortunately, only in Finnish) Paikannustietojen hyödyntäminen COVID-19 –epidemian hillinnässä states, the service's data is daily or hourly and while a reasonable accurate location of a mobile device can be obtained, if that location data doesn't identify a person, obviously, it's not protected by any laws.

On the topic of COVID-19, Future of Privacy Forum has published an article A Closer Look at Location Data: Privacy and Pandemics, where they raise points from ethics and privacy perspective of such tracking. A good read, that!

Application of mobile movement tracking

Here is one. A newspaper article titled "Puhelinten sijainti­tiedot paljastavat, kuinka ihmismassojen liikkuminen yöllisessä Helsingissä muuttui" (https://www.hs.fi/kaupunki/art-2000006698336.html):

For non-Finnish -speaking readers: this article is about movement of people in center of Helsinki. For a reference point, there is Crowd Insights data from September 2019, time before COVID-19. Movement data from pandemic is from September 5th, October 10th and 17th. To state the obvious: in 2019 between Saturday 9pm and Sunday 9am people moved a lot. What's also visible is how this global pandemic changed this behaviour. In September 2020 there were no strict regulations for night clubs and bars, which is clearly visible in the data.

Anyway, this is the kind of data of you walking around in your hometown streets, your mobile in pocket, can be easily gathered. Doing the same walk without a cell network -connect mobile device wouldn't show in that data set.

What! Are they tracking my movements via my cell phone?

Short: Yes.

Long: Yes. Your cell network provider knows every second to which cell tower all devices in their network are connected to. They also know the exact location of that identified cell tower. This coarse location information can be used by somebody or not.

Everybody knows the Hollywood movie cliché where a phone call is being tracked and somebody throws the frase "Keep them talking longer, we havent' got the trace yet!". In reality they'll know your location if your phone is turned on. It doesn't have to have an ongoing call or message being received. This is how all cell networks are designed and that's how they have always operated. They know your coarse location at all times. How exact a "coarse" location is, depends. Who has access to the location information is protected by multiple laws. The point is: they have the information. At all times.

Example illustration from 3GPP spec TR 23.912 Technical report on Super-Charger:

I'm skipping most the TLA (Three-Letter Acronyms) from the pic, but the main concept is having the car (with a phone in it) moving around the network. A HLR (or Home Location Register) will always keep track, on which BSS (note: I think it's called Radio Network Subsysten, RNS in UMTS and LTE) the mobile device talks to. This BSS (or RNS) will send updates on any jumping between the serving cells.

To simplify this further: Just replace the car with a phone in your pocket and this fully applies to people bar-hopping in center of Helsinki.

Database of cell towers

As the cell tower locations are the key component when pinpointing somebody's location, we need to know which cell towers exist and exact locations of those. Unfortunately telcos think that's a trade secret and won't release such information to general public. At the same time, from our phones we can see the identifier of the cell tower a phone is connected to and some hints to neighbouring cells. I wrote about iPhone field test mode couple years back. What a phone also has is a GPS pinpointing the exact location where a particual cell tower and couple of its friends are located at. When added with the information of, a phone typically connecting to the tower with best signal, it is possible to apply some logic. Further gathering couple of data points more, it is possible to calculate a coarse location of a cell tower your phone connects to.

Being an iPhone user, I'm sorry to say an iPhone is not technically suitable for such information gathering. Fortunately, an Android being much more open (to malware) is. Necessary interfaces exist in Android-system to query for cell tower information with an app like Tower Collector. This kind of software it is possible to create records of cell tower information and send them to OpenCelliD to be further processed and distributed to other interested parties.

OpenCelliD website contains an interactive map:

The above example is from my home town of Lappeenranta, Finland. What it depicts is approximation of a LTE cell tower location having following attributes:

• MCC: 244
• MNC: 91
• LAC: 37322
• CID: 7984918

MCC and MNC indicate the telco. CellID Finder has following information for Finland at https://cellidfinder.com/mcc-mnc#F:

What a LAC (Location Area Code) and CID indicate cannot be decoded without a database like OpenCelliD. Wikipedia article GSM Cell ID gives some hints about LAC and CID. The page also lists other databases you may want to take a look.

COVID-19 tracking

Apple and Google being the manufactures of major operating systems for mobile devices combined forces and created Exposure Notifications. This technology does NOT utilize cell towers nor GPS it it. It works only on Bluetooth LE.

As mentioned in appropriate Wikipedia article, the protocol is called Decentralized Privacy-Preserving Proximity Tracing (or DP-3T for short).

Finally

The key takeaway from all this is:
Location of your mobile device is always known.

Your location, movements between locations and timestamp you did the moving (or not moving) are actively being used to track everybody of us. That's because the technology in mobile networks requires the information.

If this information is shared to somebody else, that's a completely different story.

From Wikipedia https://en.wikipedia.org/wiki/Advance-fee_scam:

An advance-fee scam is a form of fraud and one of the most common types of confidence tricks. The scam typically involves promising the victim a significant share of a large sum of money, in return for a small up-front payment, which the fraudster requires in order to obtain the large sum.

Any Internet user knows this loved scam is very common, used actively all the time and has number of aliases, including Nigerian scam and 419 scam.

I was on my computer minding my own business when a bot approached me in Discord (that chat-thing gamers use):

Basically what they're saying, I'd get ~7000 € worth in Bitcoins by going to their scam-site, registering as new user and applying the given promo code.

Ok. For those whose bullshit detector wouldn't start dinging already think this for a second:
Why would a complete stranger offer you 7k€ in Discord!!
no
they
wouldn't.

Being interested on their scam, I went for it. Clicked the link to their website, registered a new account, followed instructions and applied the promo code. Hey presto! I was rich!

I was a proud owner of 0.711 BTC. Serious money that!

Further following the instructions:
Obviously I wanted to access my newly found riches. Those precious Bitcoins were calling my name and I wanted to transfer them out to a wallet I controlled and could use them for buying new and shiny things.

Not minding the 0.005 BTC transaction fee, this is what happens when you try accessing your Bitcoin giveaway -price:

Now they're claiming my new account has been frozen, because they think I'm a bot. Thawing my funds would be easy, simply transfer ~300€ worth of my money to them! As I wanted to keep my own hard-earned money, I did not send them the requested 0.03 BTC. I'm 100% sure, they'll keep inventing "surprising" costs and other things requiring subsequent transfer of funds. I would never ever be able to access the fake-price they awarded me.

Nice scam!

Now Mr. Stott approached me with a comment. He suggested to check the new OpenSSH file format.

I was like "What?! New what? What new format!".

The obvious next move was to go googling the topic. And yes, indeed there exists two common formats for stored OpenSSH keys. Two pieces of articles I found most helpful were The OpenSSH Private Key Format and Openssh Private Key to RSA Private Key. Reading ssh-keygen man-page states:

-m key_format

Specify a key format for key generation, the -i (import), -e (export) conversion options, and the -p change passphrase operation.
The latter may be used to convert between OpenSSH private key and PEM private key formats.
The supported key formats are: “RFC4716” (RFC 4716/SSH2 public or private key), “PKCS8” (PKCS8 public or private key) or “PEM” (PEM public key).

The commonly used two formats can be identified by first line of the private key. Old format has the header of
-----BEGIN EC PRIVATE KEY-----. Obviously, those who are using RSA or Ed25519 keys, the word "EC" would be different. I've been using ECDSA for a while and am considering moving forward with Ed25519 when all of my clients and servers have proper support for it.

I've always "loved" (to hate) SSH's (non-)intuitive user experience. As suggested by all source, to convert my existing key to new PEM-format all I need is to whip up a key generator and use it to change the passphrase. Yeah.

As my OpenSSH-client is OpenSSH_8.3p1 31 Mar 2020, its ssh-keygen will default to output keys in the new format. Depending on your version, the defaults might vary. Anyway, if you're on a really old version, you won't be having the mentioned problem in the first place.

Warning: Changing private key passphrase will execute an in-place replace of the file. If you'll be needing the files in old format, best backup them first. Also, you can convert the format back if you want. It won't produce an exact copy of the original file, but it will be in old format.

For those not using passphrases in their private files: you can always enter the same passphrase (nothing) to re-format the files. The operation doesn't require the keys to have any.

Example conversion:

$ ssh-keygen -p -f my-precious-SSH-key
Enter old passphrase:
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

Now the first line of the private key stands at:
-----BEGIN OPENSSH PRIVATE KEY-----

Notice how the key type has been changed into "OPENSSH". The key is still an ECDSA-key, but has been stored in a different.

Testing with the new key indicates a success. Now more warnings, but connectivity is still there. After the conversion, my curves are still elliptic in the right places for the server to grant access! Nice.

For many years I've run my own systems at home. Its given most of you do much less system running than I. There are servers and network switches and wireless routers, battery-backed power supplies and so on. Most of that I've written about in this blog earlier.

There is many security aspects any regular Jane lay-person won't spend a second thinking of. One of them is: What hardware runs on my home network? In my thinking that question is in top 3 -list.

The answer to that one is very trivial and can be found easily from your own network. Ask the network! It knows.

ARP - Address Resolution Protocol

This is in basics of IPv4 networking. A really good explanation can be found from a CCNA (Cisco Certified Network Associate) study site https://study-ccna.com/arp/: a network protocol used to find out the hardware (MAC) address of a device from an IP address. Well, to elaborate on that. Every single piece of hardware has an unique identifier in it. You may have heard of IMEI in your 3G/4G/5G phone, but as your phone also supports Wi-Fi, it needs to have an identifier for Wi-Fi too. A MAC-address.

Since Internet doesn't work with MAC-addresses, a translation to an IP-address is needed. Hence, ARP.

Why would you want to watch ARPs?

Simple: security.

If you know every single MAC-address in your own network, you'll know which devices are connected into it. If you think of it, there exists a limited set of devices you WANT to have in your network. Most of them are most probably your own, but what if one isn't? Wouldn't it be cool to get an alert quickly every time your network sees a device it has never seen before. In my thinking, yes! That would be really cool.

OUIs

Like in shopping-TV, there is more! A 48-bit MAC-address uniquely identifies the hardware connected to an Ethernet network, but it also identifies the manufacturer. Since IEEE is the standards body for both wired and wireless Ethernet (aka. Wi-Fi), they maintain a database of Organizationally unique identifiers.

An organizationally unique identifier (OUI) is a 24-bit number that uniquely identifies a vendor, manufacturer, or other organization.
OUIs are purchased from the Institute of Electrical and Electronics Engineers (IEEE) Registration Authority by the assignee (IEEE term for the vendor, manufacturer, or other organization).

The list is freely available at http://standards-oui.ieee.org/oui/oui.csv in CSV-format. Running couple sample queries for hardware seen in my own network:

$ fgrep "MA-L,544249," oui.csv
MA-L,544249,Sony Corporation,Gotenyama Tec 5-1-2 Tokyo Shinagawa-ku JP 141-0001

$ fgrep "MA-L,3C15C2," oui.csv
MA-L,3C15C2,"Apple, Inc.",1 Infinite Loop Cupertino CA US 95014

As we all know, CSV is handy but ugly. My favorite tool Wireshark does pre-process the ugly CSV into something it can chew without gagging. In Wireshark source code there is a tool, make-manuf.py producing output file of manuf containing the information in a more user-friendly way.

Same queries there against Wireshark-processed database:

$ egrep "(54:42:49|3C:15:C2)" manuf
3C:15:C2 Apple Apple, Inc.
54:42:49 Sony Sony Corporation

However, arpwatch doesn't read that file, a minor tweak is required. I'm running following:

perl -ne 'next if (!/^([0-9A-F:]+)\s+(\S+)\s+(.+)$/); print "$1\t$3\n"' manuf

... and it will produce a new database usable for arpwatch.

Trivial piece of information: Apple, Inc. has 789 OUI-blocks in the manuf-file. Given 24-bit addressing they have 789 times 16M addresses available for their devices. That's over 13 billion device MAC-addresses reserved. Nokia has only 248 blocks.

Practical ARP with a Blu-ray -player

Let's take a snapshot of traffic.

This s a typical boot sequence of a Sony Blu-ray player BDP-S370. What happens is:

1. (Frames 1 & 2) Device will obtain an IPv4-address with DHCP, Discover / Offer / Request is missing the middle piece. Hm. weird.
2. (Frame 3) Instantly after knowing the own IPv4-address, the device will ARP-request the router (192.168.1.1) MAC-address as the device wants to talk into Internet.
3. (Frames 5 & 6) Device will ping (ICMP echo request) the router to verify its existence and availability.
4. (Frames 7-9) Device won't use DHCP-assigned DNS, but will do some querying of its own (discouraged!) and check if a new firmware is available at blu-ray.update.sony.net.
5. (Frame 12) Device starts populating its own ARP-cache and will query for a device it saw in the network. Response is not displayed.
6. (Frames 13 & 14) Router at 192.168.1.1 needs to populate its ARP-cache and will query for the Blu-ray player's IPv4-address. Device will respond to request.
7. Other parts of the capture will contain ARP-requests going back and forth.

Practical ARP with a Linux 5.3

Internet & computers do evolve. What we saw there in a 10 year old device is simply the old way of doing things. This is how ARP works in a modern operating system:

In this a typical boot sequence. I omitted all the weird and unrelated stuff and that makes the first frame as #8. What happens in the sequence is:

1. (Frames 8-11) Device will obtain an IPv4-address with DHCP, Discover / Offer /Request / Ack -sequence is captured in full.
2. (Frames 12-14) Instantly after knowing the own IPv4-address, the device will ARP-request the IPv4 address assigned into it. This is a collision-check to confirm nobody else in the same LAN is using the same address.
3. (Frame 15) Go for a Gratuitous ARP to make everybody else's life easier in the network.
   • Merriam-Webster will define "gratuitous" as:
     not called for by the circumstances :
     not necessary, appropriate, or justified :
     unwarranted
   • No matter what, Gratuitous ARP is a good thing!
4. (Frame 16) Join IGMPv3 group to enable multicast. This has nothing to do with ARP, though.

The obvious difference is the existence of Gratuitous ARP "request" the device did instantly after joining the network.

• A gratuitous ARP request is an Address Resolution Protocol request packet where the source and destination IP are both set to the IP of the machine issuing the packet and the destination MAC is the broadcast address ff:ff:ff:ff:ff:ff. A new device literally is asking questions regarding the network it just joined from itself! However, the question asking is done in a very public manner, everybody in the network will be able to participate.
• Ordinarily, no reply packet will occur. There is no need to respond to an own question into the network.
• In other words: A gratuitous ARP reply is a reply to which no request has been made.
• Doing this seems no-so-smart, but gratuitous ARPs are useful for four reasons:
  1. They can help detect IP conflicts. Note how Linux does aggressive collision checking by its own too.
  2. They assist in the updating of other machines' ARP tables. Given Gratuitous ARP, in the network capture, there are nobody doing traditional ARPing for the new device. They already have the information. The crazy public-talking did the trick.
  3. They inform switches of the MAC address of the machine on a given switch port. My LAN-topology is trivial enough for my switches to know which port is hosting which MAC-addresses, but when eyeballing the network capture, sometimes switches need to ARP for a host to update their MAC-cache.
  4. Every time an IP interface or link goes up, the driver for that interface will typically send a gratuitous ARP to preload the ARP tables of all other local hosts. This sums up reasons 1-3.

How can you watch ARPs in a network?

Simple: run arpwatch in your Linux-router.

Nice people at Lawrence Berkeley National Laboratory (LBNL) in Berkeley, California have written a piece of software and are publishing it (among others) at https://ee.lbl.gov/. This ancient, but maintained, daemon has been packaged into many Linux-distros since dawn of time (or Linux, pick the one which suits you).

As already established, all devices will ARP on boot. They will ARP also later during normal operations, but that's beside the point. All a device needs to do is to ARP once and it's existence is revealed. When the daemon sees a previously unknown device in your network, it will emit a notification in form of an email. Example:

Here, my router running arpwatch saw a Sony Blu-ray player BDP-S370. The ethernet address contains the 24-bit OUI-part of 54:42:49 and remaining 24-bits of a 48-bit MAC will identify the device. Any new devices are recorded into a time-stamped database and no other notifications will be made for that device.

Having the information logged into a system log and receiving the notification enables me to ignore or investigate the device. For any devices I know can be ignored, but anything suspicious I'll always track.

IPv6 and ARP

Waitaminute! IPv6 doesn't do ARP, it does Neighbor Discovery Protocol (NDP).

True. Any practical implementation does use dual-stack IPv4 and IPv6 making ARP still a viable option for tracking MAC-addresses. In case you use a pure-IPv6 -network, then go for addrwatch https://github.com/fln/addrwatch. It will support both ARP and NDP in same tool. There are some shortcomings in the reporting side, but maybe I should take some time to tinker with this and create a patch and a pull-request to the author.

Avoiding ARP completely?

Entirely possible. All a stealth device needs to do is to piggy-back an existing device's MAC-address in the same wire (or wireless) and impersonate that device to remain hidden-in-plain-sight. ARP-watching is not foolproof.

Fedora updated arpwatch 3.1 RPM

All these years passed and nobody at Fedora / Red Hat did anything to arpwatch.

Three big problems:

• No proper support for /etc/sysconfig/ in systemd-service.
• Completely outdated list of Organizationally Unique Identifier (OUIs) used as Ethernet manufacturers list displaying as unknown for anything not 10 years old.
• Packaged version was 2.1 from year 2006. Latest is 3.1 from April 2020.

Here you go. Now there is an updated version available, Bug 1857980 - Update arpwatch into latest upstream contains all the new changes, fixes and latest upstream version.

Given systemd, for running arpwatch my accumulated command-line seems to be:

/usr/sbin/arpwatch -F -w 'root (Arpwatch)' -Z -i eth0

That will target only my own LAN, both wired and wireless.

Finally

Happy ARPing!

Update 13th Sep 2020:
There is a follow-up article with a key format conversion infromation.

Ever since updating into OpenSSH 8.3, I started getting this on a connection:

$ ssh my-great-linux-server
load pubkey "/home/me/.ssh/id_ecdsa-my-great-linux-server": invalid format

Whaaaat!
Double what on the fact, that connection works. There is no change in connection besided the warning.

8.3 release notes won't mention anything about that (OpenSSH 8.3 released (and ssh-rsa deprecation notice)). My key-pairs have been elliptic for years and this hasn't bothered me. What's going on!?

Adding verbosity to output with a -vvv reveals absolutely nothing:

debug1: Connecting to my-great-linux-server [192.168.244.1] port 22.
debug1: Connection established.
load pubkey "/home/me/.ssh/id_ecdsa-ecdsa-my-great-linux-server": invalid format
debug1: identity file /home/me/.ssh/id_ecdsa-ecdsa-my-great-linux-server type -1
debug1: identity file /home/me/.ssh/id_ecdsa-ecdsa-my-great-linux-server-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.3

Poking around, I found this article from Arch Linux forums: [SOLVED] openssh load pubkey "mykeyfilepath": invalid format

Apparently OpenSSH-client now requires both the private AND public keys to be available for connecting. Mathematically the public key isn't a factor. Why would it be needed? I cannot understand the decision to throw a warning about assumed missing key. I do have the key, but as I won't need it in my client, I don't have it available.

Simply touching an empty file with correct name won't clear the warning. The actual public key of the pair needs to be available to make the ridiculous message go away.

After little bit of debugging points to the problem in ssh.c:

check_load(sshkey_load_public(cp, &public, NULL),
    filename, "pubkey");

Link: https://github.com/openssh/openssh-portable/blob/V_8_3_P1/ssh.c#L2207

Tracking the change:

$ git checkout V_8_3_P1
$ git log -L 2207,2207:ssh.c

.. points to a commit 5467fbcb09528ecdcb914f4f2452216c24796790 (Github link), which was made exactly two years ago in July 11th 2018 to introduce this checking of loaded public key and emitting a hugely misleading error message on failure.

To repeat:
Connecting to a server requires only private key. The public key is used only at the server end and is not mathematically required to establish encrypted connection from a client.

So, this change is nothing new. Still the actual reason for introducing the check_load()-call with most likely non-existing public key is a mystery. None of the changes made in the mentioned commit or before it explains this addition, nor there are no significant changes made in the actual public key loading. A check is added, nothing more.

Fast forward two years to present day. Now that the 8.3 is actually used by a LOT of people, less than a month ago the problem was fixed. Commit c514f3c0522855b4d548286eaa113e209051a6d2 (Github link) fixes the problem by simulating a Posix ENOENT when the public key was not found from expected locations. More details about that error are in errno (7) man page.

Problem solved. All we all need to do is wait for this change to propagate to the new clients. Nobody knows how long that will take as I just updated this.

The TLS-certificate for this blog was up for expiration. As I'm always eager to investigate TLS and test things, I'm doing an experiment of my own and went for more modern stuff. Also I've been wanting to obsolete TLS versions 1 and 1.1, so I went for an Elliptic-Curve private key. If you can read this, you probably operate reasonable modern hardware, operating system and browser. If you for some reason stop to see my writings, then you're out of luck. I don't support your obsoleted stuff anymore!

Here are the old and new certs side-by-side:

Obvious differences are:

• Expiry: 2+ years of lifetime left. Note: They sold the cert as a 5 year one, but I know about Apple's recent decision to shorten the lifetime of a TLS-cert. Read all about it from About upcoming limits on trusted certificates.
• Issuer: Sectigo ECC Domain Validation Secure Server CA, RapidSSL/DigiCert won't issue ECDSA on cheap certs
• Certificate chain: ECDSA and SHA-2 256-bits on certificate, intermediate-CA and root-CA
• Key-pair type: Now there is a 384-bit secp384r1 curve instead of plain-old-RSA

If you want, you can do almost the same with Let's Encrypt. Getting an ECDSA-cert out of Let's E used to be a tedious manual task, but I'm using acme.sh for my LEing. Its readme says:

Let's Encrypt can now issue ECDSA certificates.
And we support them too!
Just set the keylength parameter with a prefix ec-.

Example command to get a 384-bit ECDSA certificate from Let's Encrypt with acme.sh:

$ ./acme.sh --issue --dns dns_rackspace --keylength ec-384 -d example.com

That's no different than getting a RSA-certificate. The obvious difference in Let's Encrypt cert and my paid cert is in the certificate chain. In Let's E, chain's CA-types won't change from RSA, but your own cert will have elliptic-curve math in it.

Note:
Some people say ECDSA is more secure (example: ECC is faster and more secure than RSA. Here's where you (still) can't use it). ECDSA is modern and faster than RSA, that's for sure. But about security there is controversy. If you read Wikipedia article Elliptic-curve cryptography - Quantum computing attacks, there is note: "... suggesting that ECC is an easier target for quantum computers than RSA". There exists a theory, that a quantum computer might be able to crack your curve-math easier than your prime math. We don't know if that's true yet.

Bad what?

Neighbourhood. In The Internet. There is a chance you live in one. It is not likely, but entirely possible. Most ISPs offering services to you are NOT bad ones.

This idea of a "bad neighbourhood" comes from University of Twente. Back in 2013 Mr. Moura did his PhD thesis titled "Internet Bad Neighborhoods". Summary:

Of the 42,000 Internet Service Providers (ISPs) surveyed, just 20 were found to be responsible for nearly half of all the internet addresses that send spam. That just is one of the striking results of an extensive study that focused on “Bad Neighborhoods” on the internet (which sometimes correspond to certain geographical areas) that are the source of a great deal of spam, phishing or other undesirable activity.

In plain text for those allergic to scientific abstracts: Based on 2013 study, 0.05% of all Internet Service Providers are responsible for 50% of the trouble. (Some of you'll say 99% of all studies are crap, but this one falls into the 1%.)

How can you detect a "bad neighbourhood"?

Oh, that's easy! Somebody from such a neighbourhood will send you spam. They'll send you links to malware. They port-scan you. They try to guess your passwords in SMTPd, SSHd or whatever their scanning revealed about you.

The theory is: That person "lives" (not necessarily IRL live, but Internet live) near other such criminals and wanna-be-criminals. Such people will find safe harbour within ISPs or hosting providers with lax security and/or low morale. Offenders find other like-minded cohorts and commit their cyber crime from that "neighbourhood".

Ok, I know an IP-address of a spam sender, what next?

Today, most people are using their email from Google or Microsoft or whatever cloud service. Those companies hire smart people to design, develop and run software to attempt to separate bad emails from good emails. Since I run an email server of my own, I have the option (luxury!) of choosing what to accept. One acceptance criteria is the connecting client's IP-address.

Off-topic: Other accepting criteria I have is standards compliance. Any valid email from Google or Microsoft or other cloud service provider will meet bunch of standards, RFCs and de-facto practices. I can easily demand a connecting client to adhere those. A ton of spammers don't care and are surprised that I'll just kick them out for not meeting my criteria. Obviously it won't stop them, but it will keep my mailbox clean.

On a connection to my SMTPd I'll get the connecting client's IP-address. With that I do two things:

1. Check if its already blacklisted by spam blockers
   • I'm mostly using Zen by Spamhaus. Read https://www.spamhaus.org/zen/ for more.
2. Check if its already blacklisted by me based on an earlier spam that passed standards criteria and blacklisting by spam blockers

For a passing spam a manual process is started. From connecting client's IP-address first I'll determine the AS-number (see: Autonomous system (Internet) @ Wikipedia). Second, from the AS-number I'll do a reverse listing of CIDRs covered by that ASN.

Example illustration:

Credit: Above picture is borrowed from https://www.noction.com/knowledge-base/border-gateway-protocol-as-migration.

In the above picture, I'm Customer C on green color. The CIDR assigned to me by my ISP B is 192.168.2.1/24. The spammer attempting to send me mail is at Customer D on red color. As an example, let's assume the sending mail server will have IP-address of 192.168.1.10. Doing an ASN-query with that IPv4 will result in AS 64499. Then I'll do a second ASN-query to query for all the networks assigned for AS 64499. That list will contain one entry: 192.168.1.1/24. Now I can block that entire CIDR as a "bad" one. Effectively I have found one "Internet Bad Neighborhood" to be added into my blocking list. In reality I don't know if all IP-addresses in that network are bad, but I can assume so and won't get punished by doing so.

The only real "punishment" for me is the cardinal sin of assuming. I do assume, that everybody in that neighbourhood are as evil as the one trying to spam me. Unfortunately for the "good ones" in that network, blocking the entire network so is very very effective. The level of trouble originating from that neighbourhood comes down really fast.

Practical approach with Postfix

Doing all the queries and going to different databases manually is obviously tedious, time-consuming and can be easily automated.

A tool I've used, developed an ran for many years is now polished enough and made publicly available: Spammer block or https://github.com/HQJaTu/spammer-block

In the README.md, I describe my Postfix configuration. I do add a file /etc/postfix/client_checks.cidr which I maintain with the list CIDRs of known spammers. The restriction goes into Postfix main.cf like this:

smtpd_client_restrictions =
        permit_mynetworks
        permit_sasl_authenticated
        check_client_access cidr:/etc/postfix/client_checks.cidr

An example output of running ./spammer-block.py -i 185.162.126.236 going into client_checks.cidr would be:

# Confirmed spam from IP: 185.162.126.236
# AS56378 has following nets:
31.133.100.0/24         554 Go away spammer!    # O.M.C.
31.133.103.0/24         554 Go away spammer!    # O.M.C.
103.89.140.0/24         554 Go away spammer!    # Nsof Networks Ltd (NSOFNETWORKSLTD-AP)
162.251.146.0/24        554 Go away spammer!    # Cloud Web Manage (CLOUDWEBMANAGE)
185.162.125.0/24        554 Go away spammer!    # O.M.C.
185.162.126.0/24        554 Go away spammer!    # O.M.C.

That's it. No need to map a CIDR-file. Just remember to reload the postfix after changing the CIDR-list to make sure the new list is in effect of Postfix listener.

Problem with ipwhois-library

This is the reason I didn't publish the Spammer block -project year ago. My Python-tool is using ipwhois-library for the heavy lifting (or querying). It is a well-written handy library doing just the thing I need in this project. However, (yes even this one has a however in it) ipwhois is using RADb (https://www.radb.net/) for source of information. My experience is, that when specifically talking about list of CIDRs contained in an AS-number, RADb is not perfect source of information. To fix this, I wrote an improvement of ipwhois adding ipinfo.io as an alternate ASN data source. A developer can make a choice and use the one source suiting the project.

There is a catch. ipinfo.io doesn't allow free users more than 5 ASN-queries / day / IP-address. By creating an account and paying for their API-usage that limitation can be lifted. Also, their information about CIDRs in AS-number is accurate. This is a prime example why I love having options available.

Unfortunately for me and most users requiring ASN-queries, the author of ipwhois didn't approve my improvement. You may need to install my version from https://github.com/HQJaTu/ipwhois instead. Obviously, this situation is not ideal. Having a single version would be much better for everybody.

Finally

That's it. If you receive spam from unknown source, run the script and add results to your own block-list. Do that couple weeks and your amount of spam will reduce. Drastically!

Beware! Sometimes even Google or Microsoft will send you spam. Don't block reliable providers.

I'm open for any of your comments or suggestions. Please, don't hesitate to send them.

TLS-fingerprinting. Yes. Most of regular users don't even have a concept for it.

Any regular Jane User browsing around her favorite websites can be "easily" identified as a ... well ... regular user. That identification is done by number of factors:

• Announced User-Agent if HTTP-request
• JavaScript-code executed, what's announced and detected by running library code for available features
• Traffic patterns, how greedy the attempt to chomp a website is
• Behaviour, timing of interactions, machines tend to be faster than humans

When putting any regular user aside and concentrating on the malicious ones, things start to get tricky. A bot can be written to emulate human to pass the detection not to raise any concerns. The only known effective method is to place a CAPTCHA for the user and make them pass it. There are known cases, where human will guide the bot trough and pass control to bot for it to continue its mischief. Given that, there's always room for new options for identifing requests.

There are number of organizations and people (including Google, Edward Snowden and US Government) to encourage people to encrypt everything. An example of this would be The HTTPS-Only Standard by US Government. Early 2020 figures indicate, that the entire Internet is ~60% HTTPS, ~93% requests arriving to Google are using HTTPS instead of non-encrypted HTTP. We're getting there, about everything is encrypted.

Short history of TLS fingerprinting

The thing with TLS-encryption is, the way the encryption is implemented can be fingerprinted. This fingerprint can be added to list of factors used to determine if you are who you say you are. A really good presentation about history TLS/SSL fingerprinting can be found from DETECTION ENGINEERING: Passive TLS Fingerprinting Experience from adopting JA3, but to summarize it here:

• 2009: Apache module mod_sslhaf
• 2012: p0f, TCP/IP fingerprinting tool
• 2015: SquareLemon TLS fingerprinting
• and the good one from Salesforce:
  2017: JA3 for clients
• 2018: JA3S for servers
• a latecomer 2019: Cisco Joy

What's missing from the list is the techniques used by various government agencies and proprietary systems. It is obvious to me, they hire bunch of super-smart individuals and some of them MUST have found the fingerprinting and never telling us about it. Point here is, there has always been methods to fingerprint your encryption, you just didn't know about it.

In this blog post I'll be focusing on JA3 as it is gaining a lot of traction in the detecting Threat Actors -field of security. This post is not about dirty details of JA3, but if you want, you can study how it works. The source code is at https://github.com/salesforce/ja3 and a video of two out of three J.A.s presenting JA3 @ Shmoocon 2018 can be found at https://youtu.be/oprPu7UIEuk?t=407. Unlike source code (which has some prerequisites), the video alone should give most of you an idea how it works.

Theory of detecting Threat Actors (TAs)

Remember, this fingerprinting originates from intrusion detection. It can be used for other purposes, but a lot of the background is from IDS. So, when studying a system for Indicators of Compromise, the theory is as follows:

A Threat Actor to change various aspects of the attack. On the bottom of the pyramid-of-pain, there are things that are trivial to alter. You can change the hash of your attack tool (if running on a system trying to detect you), or approach the target from another IP-address or change your Command&Control server to a new domain. All of these changes would throw off any attempts to detect you. Food for thought: Typical OS-commands won't change, unless upgrades are being applied. If a tool changes constantly, it's a tell.

When placing TLS-fingerprinting on the same pyramid:

Looking into one's encrypted traffic obscures some of the handy points used to detect your activity. So, neeed to shift focus to the top of the pyramid. On top there are tools and attacker's ways of working. Really difficult to change those. That's exactly where TLS-fingerprinting steps in. What if your tools leave breadcrumbs? Enough for a vigilant detection-system to realize somebody is doing probing or already has compromised the system. At minimum there is weird traffic whose fingerprint doesn't look good warranting closer look by a human.

JA3 on Wireshark

My beef with JA3 has (so far) been the fact, that my favorite network analysis tool, Wireshark, doesn't support it. Now it does:

(I took a still frame from JA3 Shmoocon presentation video and pasted Wireshark logo on top of it)

There is a Wireshark dissector for JA3. It is written in LUA by an anonymous GitHub contributor who chose not to divulge any public details of him/herself and me. Original version didn't do JA3S, and I contributed that code to the project. The Wireshark plugin is available at https://github.com/fullylegit/ja3. To get it working in Wireshark my recommendation is to install the LUA-plugin into your personal plugins-folder. In my Windows, the folder is %APPDATA%\Wireshark\plugins. Note: The sub-folder plugins\ didn't exist on my system. I created it and then placed the required .lua-files into it.

In action a JA3-fingerprint would look like this:

What we see on the Wireshark-capture is a TLS 1.2 client hello sent to a server. That particular client has JA3-hash of "66918128f1b9b03303d77c6f2eefd128". There is a public JA3-database at https://ja3er.com/, if you do any fingerprinting, go check your matches there. Based on JA3er, the previously captured hash typically identifies a Windows Chrome 79. Not suspicious, that one.

Trying to evade JA3 fingerprinting

Security is always a chase. As we already learned, lot of (bad?) people knew about this fingerprinting for many many years. They may be already ahead, we don't really know that yet.

A good primer on how to make your fingerprint match on demand is a post JA3/S Signatures and How to Avoid Them.

Let's try some manual tinkering! A simple

curl --tlsv1.2 https://ja3er.com/

Will indicate de4c3d0f370ff1dc78eccd89307bbb28 as the JA3-hash, that's a known curl/7.6x. For example a Windows Firefox 72 gets a JA3-hash of b20b44b18b853ef29ab773e921b03422, so it should be easy to mimic that, right?

Ignoring all other parts of JA3, Windows Firefox 72 will on TLS 1.2 use a 18 cipher set in prioritized order of:

1. 0x1301/4865 TLS_AES_128_GCM_SHA256
2. 0x1303/4867 TLS_CHACHA20_POLY1305_SHA256
3. 0x1302/4866 TLS_AES_256_GCM_SHA384
4. 0xc02b/49195 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
5. 0xc02f/49199 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
6. 0xcca9/52393 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
7. 0xcca8/52392 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
8. 0xc02c/49196 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
9. 0xc030/49200 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
10. 0xc00a/49162 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
11. 0xc009/49161 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
12. 0xc013/49171 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
13. 0xc014/49172 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
14. 0x0033/51 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
15. 0x0039/57 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
16. 0x002f/47 TLS_RSA_WITH_AES_128_CBC_SHA
17. 0x0035/53 TLS_RSA_WITH_AES_256_CBC_SHA
18. 0x000a/10 TLS_RSA_WITH_3DES_EDE_CBC_SHA

Info:
Full list of all TLS/SSL cipher suites is at IANA: https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4 Beware! The list is a living thing and cipher suites are known to be deprecated for old TLS-versions. Also new ones might pop.

That 18 cipher list in OpenSSL-format used by curl is:

1. TLS_AES_128_GCM_SHA256
2. TLS_CHACHA20_POLY1305_SHA256
3. TLS_AES_256_GCM_SHA384
4. ECDHE-ECDSA-AES128-GCM-SHA256
5. ECDHE-RSA-AES128-GCM-SHA256
6. ECDHE-ECDSA-CHACHA20-POLY1305
7. ECDHE-RSA-CHACHA20-POLY1305
8. ECDHE-ECDSA-AES256-GCM-SHA384
9. ECDHE-RSA-AES256-GCM-SHA384
10. ECDHE-ECDSA-AES256-SHA
11. ECDHE-ECDSA-AES128-SHA
12. ECDHE-RSA-AES128-SHA
13. ECDHE-RSA-AES256-SHA
14. DHE-RSA-AES128-SHA
15. DHE-RSA-AES256-SHA
16. AES128-SHA
17. AES256-SHA
18. DES-CBC3-SHA

Info:
My Linux curl is built to use OpenSSL for TLS and OpenSSL has a different naming for ciphers than IANA and everybody else in the known universe.

Going for a curl with --cipher and listing all of the above ciphers colon (:) separated will rather surprisingly, NOT produce what we intended. We're expecting to get full JA3 starting with:
771,4865-4867-4866-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10,
but are getting a:
771,4866-4867-4865-4868-49195-49199-52393-52392-49196-49200-49162-49161-49171-49172-51-57-47-53-10-255,
instead.

TLS-version of 771 matches, first three ciphers match, in WRONG order. Then curl appends a 4868 into the list. 4868, or 0x1304 is TLS_AES_128_CCM_SHA256. We didn't ask for it, but because OpenSSL knows better, it will append it to the list.

Miserable failure!

Going forward, getting all the 18 ciphers right with OpenSSL can be done. It's just too much work for me to even attempt. And I haven't got to extensions or elliptic curve parameters yet! Uff.... way too difficult to forge the fingerprint. I might have a go with that on a Python. If I have time.

Finally

This is fun stuff!

Understanding how TLS works and why it can be used as a very good indicator of your chosen tool is critical to your success. Your Tor-client WILL have a distinct fingerprint. It will be different than your Chrome, iOS Safari, Python 3 or PowerShell. Anybody having access to your encrypted traffic (like your ISP or the server you're surfing to) will be able to detect with what you're arriving.

Embrace that fact!

This is what happened to me:

A missed call from Papua New-Guinea. Well... I don't know anybody there, so they shouldn't be calling me.

It doesn't take too much investigation to realize, IT'S A SCAM!

Example: Have you been getting unexpected overseas calls? from Australia
and How to identify and report Wangiri fraud to Vodafone from UK.

The Vodafone article says:

What’s Wangiri fraud?

It’s receiving missed calls from international numbers you don’t recognise on either a mobile or a fixed-line phone.
The fraudsters generating the missed calls hope that their expensive international numbers will be called back
so that they can profit.

Looks like that scam has been going on for years. The reason is obvious, it's way too easy! Making automated calls and hanging up when the other side starts ringing doesn't cost you anything. The seriously expensive number unsuspecting victims will call back will apparently play you some music while making you wait as long as you like. Every minute the criminals will get a slice of your money.

How is this possible? How can you change the number you're calling from? Well, easy! You can do it too: https://www.spooftel.com/
"SpoofTel offers you the ability to spoof caller ID and send SMS messages. You can change what someone sees on their call display when they receive a phone call to anything you like!"

Entire world is using ancient telecommunications protocol SS#7. If you're really interested, read The Wikipedia article about it. There are number of flaws in it, as it is entirely based on the assumption only non-criminals have access to global telecommunications network. It used to hold true at the time it was created, but after that. Not so much. And that unchangeable thingie we have to thank for this and multiple other scams and security flaws.

Emotet is a nasty piece of malware. It has been around The Net for number of years now and despite all the efforts, it is still stealing money from unsuspecting victims who log in into their online bank with their computers and suddenly lose all of their money to criminals.

Last month, I bumped into a "historical" Emotet-reference. A document contains the URLs for malicious distribution endpoints of documents and binaries used to spread the malware. It also contains IPv4-addresses for Command & Control servers. There are hundreds of endpoints listed, and every single one I tested was already taken down by ISPs or appropriate government officials. Surprisingly, only 20% of the URLs were for Wordpress. Given its popularity and all the security flaws, I kinda expected the percentage to match its market share, 35% of all the websites in the entire World run Wordpress. If you're reading this in the future, I'd assume the percentage to be higher.

As a coding exercise, I analysed the listed endpoints for all three variants (or Epochs as this malware's generations are called) of Emotet and created a heatmap of them. It would be really fun to get a list of all the infected computers and list of those computers where money was stolen from, but unfortunately for my curious mind, that data wasn't available.

So, no victims, only hijacked servers in this map:

Actual Google Maps -application I wrote is at https://blog.hqcodeshop.fi/Emotet-map/map.html, go investigate it there.

This is a simple project, but if anybody want's do learn data visualization with Google Maps JavaScript API, my project is at https://github.com/HQJaTu/emotet-malware-mapper. Note: You will need an API-key from Google for your own projects. My API-key is publicly available, but restricted. It won't work for you.

As analysis of the hijacked distribution points and C2 -servers, there is lot of heat in obvious places, Europe and North America. But as you can see, there are lots of servers in use all around the globe. That should give everybody an idea why fighting cybercrime is so difficult.

Update 30th Jan 2020:
Emotet seems to be picking up speed, there is a US CISA warning about Increased Emotet Malware Activity. Apparently it is #1 malware currently in the world.

Mr. Bruce Schneier is one of the influencers I tend to follow in The Net. He is a sought after keynote speaker and does that on regular basis. More on what he does on his site @ https://www.schneier.com/.

Quite few of his keynotes, speeches and presentations are made publicly available and I regularly tend to follow up on those. Past couple of years he has given a keynote having roughly the same content. Here is what he wants you, an average Joe Internet-user to know in form of lessons. To draw this with a crayon: Lessons are from Bruce Schneier, comments are mine.

These are things that are true and are likely to be true for a while now. They are not going to change anytime soon.

Truism 1: Most software is poorly written and insecure

The market doesn't want to pay for secure software.

Good / fast / cheap - pick any two! The maket has generally picked fast & cheap over good.

Poor software is full of bugs and some bugs are vulnerabilities. So, all software contains vulnerabilities. You know this is true because your computes are updated monthly with security patches.

Whoa! I'm a software engineer and don't write my software poorly and it's not insecure. Or ... that's what I claim. On a second thought, that's how I want my world to be, but is it? I don't know!

Truism 2: Internet was never designed with security in mind

That might seem odd 2019 saying that. But back in the 1970s, 80s and 90s it was conventional wisdom.

Two basic reasons: 1) Back then Internet wasn't being used for anything important, 2) There were organizational constraints about who could access the Internet. For those two reasons designers consciously made desicisions not to add security into the fabric of Internet.

Where still living with the effects of that decision. Whether its the domain name system (comment: DNS), the Internet routing system (comment: BGP), the security of Internet packets (comment: IP), security of email (comment: SMTP) and email addresses.

This truism I know from early days in The Net. We used completely insecure protocols like Telnet or FTP. Those were later replaced by SSH and HTTPS, but still the fabric of Internet is insecure. It's the applications taking care of the security. This is by design and cannot be changed.

Obviously there are known attempts to retrofit fixes in. DNS has extension called DNSSEC, IP has extensions called IPsec, mail transfer has a secure variant of SMTPS. Not a single one of those is fully in use. All of them are accepted, even widely used, but not every single user has them in their daily lives. We still have to support both the insecure and secure Net.

Truism 3: You cannot constrain the functionality of a computer

A computer is a general purpose device. This is a difference between computers and rest of the world.

No matter how hard is tried a (landline) telephone cannot be anything other than a telephone. (A mobile phone) is, of course, a computer that makes phone calls. It can do anything.

This has ramifications for us. The designers cannot anticipate every use condition, every function. It also means these devices can be upgraded with new functionality.

Malware is a functional upgrade. It's one you didn't want, it's one that you didn't ask for, it's one that you don't need, one given to you by somebody else.

Extensibility. That's a blessing and a curse at the same time. As an example, owners of Tesla electric cars get firmware upgrades like everything else in this world. Sometimes the car learns new tricks and things it can do, but it also can be crippled with malware like any other computer. Again, there are obvious attempts to do some level of control what a computer can do at the operating system level. However, that layer is too far from the actual hardware. The hardware still can run malware, regardless of what security measure are taken at higher levels of a computer.

Truism 4: Complexity is the worst enemy of security

Internet is the most complex machine mankind has ever built, by a lot.

The defender occupies the position of interior defending a system. And the more complex the system is, the more potential attack points there are. Attacker has to find one way in, defender has to defend all of them.

We're designing systems that are getting more complex faster than our ability to secure them.

Security is getting better, but complexity is getting faster faster and we're losing ground as we keep up.

Attack is easier than defense across the board. Security testing is very hard.

As a software engineer designing complex systems, I know that to be true. A classic programmer joke is "this cannot affect that!", but due to some miraculous chain, not ever designed by anyone, everything seems to depend on everything else. Dependencies and requirements in a modern distributed system are so complex, not even the best humans can ever fully comprehend them. We tend to keep some sort of 80:20-rule in place, it's enough to understand 80% of the system as it would take too much time to gain the rest. So, we don't understand our systems, we don't understand the security of our systems. Security is done by plugging all the holes and wishing anything wouldn't happen. Unfortunately.

Truism 5: There are new vulnerabilities in the interconnections

Vulnerabilities in one thing, affect the other things.

2016: The Mirai botnet. Vulnerabilities in DVRs and CCTV cameras allowed hackers to build a DDoS system that was raid against domain name server that dropped about 20 to 30% of popular websites.

2013: Target Corporation, an USA retailer, attacked by someone who stole the credentials of their HVAC contractor.

2017: A casino in Las Vegas was hacked through their Internet-connected fish tank.

Vulnerability in Pretty Good Privacy (PGP): It was not actually vulnerability in PGP, it was a vulnerability that arose because of the way PGP handled encryption and the way modern email programs handled embedded HTML. Those two together allowed an attacker to craft an email message to modify the existing email message that he couldn't read, in such a way that when it was delivered to the recipient (the victim in this case), a copy of the plaintext could be sent to the attacker's web server.

Such a thing happens quite often. Something is broken and its nobody's fault. No single party can be pinpointed as responsible or guilty, but the fact remains: stuff gets broken because an interconnection exists. Getting the broken stuff fixed is hard or impossible.

Truism 6: Attacks always get better

They always get easier. They always get faster. They always get more powerful.

What counts as a secure password is constantly changing as just the speed of password crackers get faster.

Attackers get smarter and adapt. When you have to engineer against a tornado, you don't have to worry about tornadoes adapting to whatever defensive measures you put in place, tornadoes don't get smarter. But attackers against ATM-machines, cars and everything else do.

An expertise flows downhill. Today's top-secret NSA programs become tomorrow's PhD thesis and next day's hacker tools.

Agreed. This phenomenon can be easily observed by everybody. Encryption always gets more bits just because today's cell phone is capable of cracking last decade's passwords.

If you want to see The Man himself talking, go see Keynote by Mr. Bruce Schneier - CyCon 2018 or IBM Nordic Security Summit - Bruce Schneier | 2018, Stockholm. Good stuff there!

This blog post is about TLS-protocol. It is a constantly evolving beast. Mastering TLS is absolutely necessary, as it is one the most common and widely used components keeping our information secure in The Net.

December 2018: Lot of obsoleted TLS-stuff deprecated

Nearly an year ago, Google as the developer of Chrome browser deprecated SHA-1 hashes in TLS. In Chrome version 72 (December 2018) they actually did deprecate TLS 1 and TLS 1.1 among other things. See 72 release notes for all the details.

Making TLS 1.2 mandatory actually isn't too bad. Given Qualsys SSL Labs SSL Pulse stats:

96,0% of the servers tested (n=130.000) in SSL Labs tester support TLS 1.2. So IMHO, with reasonable confidence we're at the point where the 4% rest can be ignored.

Transition

The change was properly announced early enough and a lot of servers did support TLS 1.2 at the time of this transition, so the actual deprection went well. Not many people complained in The Net. Funnily enough, people (like me) make noise when things don't go their way and (un)fortunately in The Net that noise gets carried far. The amount of noise typically doesn't correlate anything, but phrase where there is smoke - there's fire seems to be true. In this instance, deprecating old TLS-versions and SHA-1 hashing made no smoke.

Aftermath in Chrome

Now that we have been in post SHA-1 world for a while, funny stuff starts happening. On rare occasion, this happens in Chrome:

In Chrome developer tools Security-tab has a complaint: Connection - obsolete connection settings. The details state: The server signature users SHA-1, which is obsolete. (Note this is different from the signature in the certificate.)

Seeing that warning is uncommon, other than me have bumped into that one. An example: How to change signature algorithm to SHA-2 on IIS/Plesk

I think I'm well educated in the details of TLS, but this one baffled me and lots of people. Totally!

What's not a "server signature" hash in TLS?

I did reach out to number of people regarding this problem. 100% of them suggested to check my TLS cipher settings. In TLS, a cipher suite describes the algorithms and hashes used for encryption/decryption. The encrypted blocks are authenticated with a hash-function.

Here is an example of a cipher suite used in loading a web page:

That cipher suite is code 0xC02F in TLS-protocol (aka. ECDHE-RSA-AES128-GCM-SHA256). Lists of all available cipher suites can be found for example from https://testssl.sh/openssl-iana.mapping.html.

But as I already said, the obsoletion error is not about SHA-1 hash in a cipher suite. There exists a number of cipher suites with SHA-1 hashes in them, but I wasn't using one.

Second clue is from the warning text. It gives you a hint: Note this is different from the signature in the certificate ... This error wasn't about X.509 certificate hashing either. A certificate signature looks like this:

That is an example of the X.509 certificate used in this blog. It has a SHA-256 signature in it. Again: That is a certificate signature, not a server signature the complaint is about. SHA-1 hashes in certificates were obsoleted already in 2016, the SHA-1 obsoletion of 2018 is about rest of the TLS.

What is a "server signature" hash in TLS?

As I'm not alone with this, somebody else has asked this question in Stackexchange: What's the signature structure in TLS server key exchange message? That's the power of Stackexchange in action, some super-smart person has written a proper answer to the question. From the answer I learn, that apparently TLS 1 and TLS 1.1 used MD-5 and/or SHA-1 signatures. The algorithms where hard-coded in the protocol. In TLS 1.2 the great minds doing the protocol spec decided to soft-code the chosen server signature algorithm allowing secure future options to be added. Note: Later in 2019, IETF announced Deprecating MD5 and SHA-1 signature hashes in TLS 1.2 leaving the good ones still valid.

On the wire, TLS server signature would look like this, note how SHA-1 is used to trigger the warning in Chrome:

During TLS-connection handshake, Server Key Exchange -block specifies the signature algorithm used. That particular server I used loved signing with SHA-1 making Chrome complain about the chosen (obsoleted) hash algorithm. Hard work finally paid off, I managed to isolate the problem.

Fixing server signature hash

The short version is: It really cannot be done!

There is no long version. I looked long and hard. No implementation of Apache or Nginx on a Linux or IIS on Windows have setting for server signature algorithm. Since I'm heavily into cloud-computing and working with all kinds of load balancers doing TLS offloading, I bumped into this even harder. I can barely affect available TLS protocol versions and cipher suites, but changing the server signature is out-of-reach for everybody. After studying this, I still have no idea how the server signature algorithm is chosen in common web servers like Apache or Nginx.

So, you either have this working or not and there is nothing you can do to fix. Whoa!

IRS or Internal Revenue Service, the tax authority of United States of America seems to be an endless scapegoat of scams and malware. Couple days ago, I got an email from Germany stating, that I had received a tax refund from IRS. Nice! I thought, you need to pay your taxes to USA to be eligible for an IRS tax refund. After reading the mail, I knew this particular email was very important, it was sent from Germany after all. I instantly whipped up my credit card and redied myself for punching in the required numbers ... naah! That mail was an obvious scam. Actually, IRS is warning general public about such scams in their website. Again, this type of scam is nothing new, I had a look into finer details of such a scam back in 2016, see my blog post here about that.

An interesting side note here is, the e-mail address the scammers sent the email to, was the one which got harvested from DocuSign leak of 2018. See my blog post about that here.

As 85% of such scams, the email I got has a link in it. That number comes from Proofpoint Q2 2019 Threat Report:

Malicious URLs made up 85% of global combined malicious URL and attachment message volume, a slight increase from May, but overall in-line with the trend for 2019.

What happens when you click the link, there is a login:

Username and password was in the email I received. Username was my DocuSign email and a not-too-long-or-complex random password accompanied it. I didn't hesitate, I was curious to see what the scammers cooked up this time:

Yeah. A download. What could possibly go wrong with that!

To access the good bits, I moved away from web browser, to a Linux prompt and a curl. Initially, the copied link to download.php didn't respond. Wow! A security-minded scammer. That seems to be a trend nowadays. To get past that one, I took a peek into cookies in the web browser and added a session-cookie to the curl-request. Like this:
curl --verbose --header "Cookie: sid=-da-passwrod-" http://download.php

That did the trick, a document.zip landed on my lap. No funny business there, a regular Zip-file containing a single file:

Archive: document.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
  228703  Defl:N    22214  90% 09-30-2019 13:05 68c60f7a  011239-23489234.doc

As usual, jumping too fast for conclusions: Ah, a macro-loader, I thought. To confirm:

$ file 011239-23489234.doc
011239-23489234.doc: Rich Text Format data, version 1, unknown character set

Ok. Interesting. RTF-documents don't have macros. However, I remember reading an article "Using RTF Files as a Delivery Vector for Malware" earlier. Taking a peek with hexdump:

Confirmed! There is a binary-object embedded into RTF. I borrowed the idea from SANS diary "Getting the EXE out of the RTF" and wrote a single-liner Perl:

$ perl -ne 's/([0-9A-F]{2})/print chr(hex($1))/ge' 011239-23489234.rtf > scam.bin

This one will produce extra head and tail, but also the good bits will be in my new scam.bin. Cutting the extra stuff out:

It's an XML containing a VBscript scriptlet. There is some obfuscation in it, but the Base-64 -encoded string is something, that caught my attention instantly. Decoding it resulted in the real payload URL ... which unfortunately didn't load as the website was shut down by authorities and/or vigilant sysadmin of the site. So, I never got to download my promised b.exe. Well. It most likely wouldn't contain much interesting parts in it. The binary simply messes up with your Windows and loads more "goodies" into it and poking a backdoor. No surprises, the usual.

Typically this type of scam targets the not-so-bright ones. Scammers do that intentionally, to weed out "difficult" customers. Neither the email nor IRS website-replica weren't especially elaborate work. Any self-respecting computer user should instantly recognize the scam. For scammers its always worth trying. One out of million recipients might fall into this.

"Hey! Let's be careful out there."

Recently there as been an influx of SMS-messages informing that "your parcel has arrived". The message will contain a short URL bouncing you around the net couple times and landing you to a scam-site. Finnish CERT-FI actually released a warning about the scammers operation recently. You can read the announcement in Finnish here.

Scam

The fradaulent website did look like this:

I was in process of writing this blog post and POOOOOOOF! The domain went away.

Ultimately you will land on a form to enter your information into a subscription trap:

If there would be a honest scammer, these guys are. The fine print on the right hand corner explains how the scam will work. You won't be receiving your non-existent shipment, but you will be paying initial fee of a subscription. Nice!

As usual, I did some digging about the announced website and the company. The website is still running on a Hezner VM in Germany. Since these "honest" scammers are protecting your precious GDPR-covered PII, their website even has a valid TLS-certificate issued by Comodo. The obvious motivation for using encryption is the fact that most browsers will fight back for an user entering credit card information without encrypted traffic. So, a very elaborate scam, this.

Company

Since these scammers are honest, they inform the service "responsible" company being NMS Services Ltd. I doubt that much, given the company given seems to be some sort of blank shell. A company without any activity:

There is a good chance, the above ltd. has nothing to do with this scam.

Huge footprint

The "m.o." or modus operandi of these types of scams is to have a huge footprint. They have bunch of domains and companies as it's guaranteed their entire operation will be shut down by law enforcement as they scam along.

To get their traffic to the right place, the SMS has a very short URL in it. Something with gramkl.com in it. That's a jump-site operating at Google bouncing users to lilorole.com, which is a Cloudflare forwarder. The first form result is forwarded to t.viprsp.nl, a Cloudflare forwarder, which will ultimately land at end result healthybuddy.eu, the real harvesting website at Hezner.

What we have there in the above chain, is numer of top-level-domains and bunch of companies operating under different jurisdictions. Shutting all of them down is very very very difficult. If one is shut down, they'll just pull another rabbit out of their hat and go on scamming.

Alternates

In the TLS-certificate information I found following:

Subject: OU = Domain Control Validated, OU = PositiveSSL, CN = bestplayerwins.com
X509v3 extensions:
  X509v3 Subject Alternative Name:
    DNS:bestplayerwins.com, DNS:www.bestplayerwins.com

An alternate scam site running @ bestplayerwins.com is found. Again, this scam is a honest one, it will inform that they're bullshitting your with a subscription trap.

Just by investigating the site more, there are (unsurprisingly) other domains found:

bpwbill.com and ccbill.info sites look exactly like the initial scam site:

Again they have distributed their operation to multiple domains and TLDs to make it extremely difficult to shut the entire operation down.

I did not check the details of Gibraltar-registered Future Base Ltd because of high cost of the records. Again, there is a very good chance they're not affiliated with this operation at all.

Finally

With the words of Sergeant Phil Esterhaus from Hill Street Blues: