Wi-Fi Router Lifespan: A Threat to National Security?
Sunday, October 15. 2023
Wireless LAN, or Wi-Fi, is topic I've written a lot about. Router hardware is common. Most end-user appliance people use are wireless. Wi-Fi combined with proper Internet connection has tons of bandwidth and is responsive. From hacking perspective quite many of those boxes run Linux or a thing with hackable endpoints. Or alternatively, on the electronics board, there are interesting pins that a person with knowledge can lure the box do things manufacturer didn't expect to happen. Oh, I love hardware hacks! 
Routers are exploitable
Back in 2016 these routers were harnessed to a new use. From hacker's perspective, there exists a thing which works perfectly, but doesn't do the thing hacker wishes it to do. So, after little bit of hacking, the device "learns" a new skill. This new skill was to participate in criminal activity as a DDoS traffic generator. Geekflare article How to Secure Your Router Against Mirai Botnet Attacks explains as follows:
According to Paras Jha, one of the authors of the Mirai bot, most of the IoT devices infected and used by the Mirai Botnet were routers.
A word from national intelligence organization
Fast forward nine years to 2023. Things really haven't changed much. There exists even more Wi-Fi routers. The routers are manufactured in huge volumes, designed to make maximum profit for the manufacturer and are lacking a ton of security features. Combine these hackable routers with all the geopolitical tension in the World right now our beloved routers have become the interest of Finnish Security and Intelligence Service (or Supo, acronym in Finnish).
This month, in their annual National Security Overview 2023, in Threats to national security are continually evolving, they issued a warning: "Cyber espionage exploits unprotected consumer devices". Actually, they did pretty much the same thing back in March -21 with following statement: "Supo has observed that the intelligence services of authoritarian states have been exploiting dozens of network devices and servers of Finnish individuals and businesses in cyber espionage operations."
How?
Having a national intelligence service to warn about crappy network hardware is a big deal. They don't do the same warning about your toaster or dish washer or cheap Android phone. Same characteristics don't really apply to anything else. A device needs to be:
- On-line / Internet connected
- See Mr. Hyppönen's book: If It’s Smart, It’s Vulnerable
- And btw., all routers are computers. Any computer is considered as a "smart" device.
- Insecure
- Yeah. Even the expensive Wi-Fi routers have crappy manufacturer firmware in them. Cheap ones are especially vulnerable. And even the good ones expire after couple of years as manufacturer loses interest and ceases publishing new upgrades.
- Exist in masses
- Literally every home and business has at least one. I don't know the total number of homes and businesses in the World, but it must be a big number!
On those three characteristics, following things are true:
- Every single vulnerable device can be found easily.
- As the internet, there are 3,706,452,992 public IPv4 addresses. That seems like a big number, but in reality it isn't. (Actually, the scale of the number is same as the number of homes+businesses.)
- In 2013 the entire address space could be scanned in 44 minutes. (See Washington Post article Here’s what you find when you scan the entire Internet in an hour)
- Every single vulnerable device can be cracked wide open without human interaction by automated tools in a split second.
- Every single cracked device can be made to do whatever the attacker wants.
- Any typical scenario is some sort of criminal activity.
Why?
100% of all lay-persons I've talked to state "I don't care if I'm hacked. I'm not an important person and have nothing to hide." What they miss is, attacker wanting to use their paid connection while impersonating as them to commit crimes.
We also have learned not all attackers are after money, some of them are state actors doing espionage. There are multiple types of attackers ranging from teenagers smearing their school's website to cybercriminals stealing money to government sponsored spies stealing state secrets.
Now we're getting to the reason why intelligence services are issuing warnings!
Scaring consumers - There is profit to be made
Since these intelligence service warnings have existed for couple years, in May -23 a major ISP / Telco in Finland, DNA, issued a press release (this is in Finnish, sorry) stating following:
Does you home have over four year old router? An expert reveals why it can be a risk.
Translated quote:
As a rule of thumb I'd say: a four year old router for basic user is aged. Advanced users may replace their routers every two years.
Going open-source!
For clarity: I'm not disputing the fact an aged router with never being upgraded to latest available firmware wouldn't be a security risk. It is! As a hacker I'm disputing is the need to purchase a new one. Gen. pop. will never ever be able to hack their devices into running OpenWrt or DD-WRT, that's for sure. Instead, educating people with risks involved with cheap consumer electronics and offering advice on smart choices would be preferred.
Here is my advice:
- Router manufacturers (and ISPs) are commercial entities targeting to maximize their profit. Their intent is to sell you a new router even when the hardware of your old device is still ok.
- Part of profit maximizing is to abandon the device after couple years of its release. There exists manufacturers which have never released a single security patch. Profit maximizing is very insecure for you as consumer.
- Hardware doesn't expire as fast as software does. There are exeptions to this. Power supplies and radio frequency electronics takes the greatest wear&tear on your 24/7 enabled device, sometimes getting a new box is the best option.
- Your old hardware may be eligible for re-purposing with open-source options. Ask your local hacker for details.
- Open-source firmware gets upgrades for both features and security for any forseeable future. This can happen as open-source firmware unifies all various hardware under a single umbrella.
- Make a habit of upgrading your open-source firmware every now and then. New firmwares will be made available on a regular basis.
Personally, for the past 19 years I've only purchased Wi-Fi routers which have OpenWrt or DD-WRT -support. Typically, after unboxing the thing, factory firmware runs only those precious minutes to get a proper Linux running into them. This is what I recommend everybody else to do!
PS. Kudos to those manufacturers who skipped the part with creating something and abandoning firmware of their own and license open-source solutions. There aren't many of you. Keep it up!
Writing a secure Systemd daemon with Python
Sunday, March 5. 2023
Hetzner outgoing mail - SMTP blocked on TCP/25
Sunday, August 28. 2022
For ages spammers' tactic has been to use either free trial of a service or payment via "liberated" credit card. As public cloud service providers want to lure in new customers, there is a habit of offering non-paid services for a period of time. All major providers like AWS, Azure and Google Cloud Platform have such free-of-charge use available.
Also smaller companies have such offering. Smaller companies (are still quite big, but not as big as biggest ones) would include Hetzner and OVHcloud. Both have number of data centers around the Europe and for any postmaster, are a royal pain in the ass. You can get a really cheap VM on either and start spewing spam email into unsuspecting SMTP-servers, like the one I maintain.
This type of free-spam-for-all isn't sustainable, or hasn't been for years, but these companies still allow it to happen. In 2021 Hetzner made a change for better. Lot wasn't communicated, however, this information was published to its customers:
Microsoft Blacklist
Microsoft maintains their own internal blacklists, which are used to reject emails coming from specific IP addresses.
Note: Entire info is available @ Hetzner website.
As a customer of Hetzner having an operational VM, I bumped into a new undocumented restriction applied to new accounts. On my own box, everything still works as usual. Has been for years. To my surprise, outgoing TCP/25 on a new account having a new VM is blocked. What!? Investigating this, led me to Hetzner customer support site, as it has following options for support request:
There is a drop-list with different types of problems having a specific category of Sending mails not possible. Apparently somebody designed the support pipeline for that particular need.
Lack of official documentation made me look in The Net. There I found a confirmation of this policy change. In June 2021 Bjørn Erling Fløtten of Norway wrote in his blog post Ditching Windows and AWS, diving into Linux and Hetzner following story:
Step 8: Setting up an email server
...
(This reminds me of the good old days when a mail server by default usually would be an open relay. Now you definitely get a more secure basic install)
Now I could actually also telnet to port 25 from my Amazon instance, and the connection appeared in the Postfix log. I have no clue why it did not work on the first attempt.
Time to ask Hetzner to open port 25 outbound (SMTP port).
(There is no mentioning of port 25 blocking anywhere on Hetzner's pages. But it is of course natural to assume that port 25 is blocked, even though the firewall administration page of Hetzner's Cloud Console says "All outbound traffic is allowed. Add a rule to restrict outbound traffic.").
So I decided to open a Support ticket, which suggest:
"Server issue"
and
"Server issue: Sending mails not possible"
Choosing the latter gives this:
"Outgoing traffic to ports 25 and 465 are blocked by default on all Cloud Servers.
A request for unblocking these ports and enabling the send mail option is only possible after the first paid invoice."Fair enough, some thrust needs to be established.
Invoice information says:
"Invoice information
You currently receive 1 invoice a month.
The invoice will be created on the following day:
...th of the month"...
Bjørn: Hint! Get a free-of-charge TLS-certificate somewhere. Most browsers refuse to enter your site. Thanks, though, for sharing your experience. Now I know to first wait for the invoice and then request for the ports to be opened.
Hetzner: Hint! Would it be smart to document all of this somewhere? Thanks for this change in your policy, though. Anybody needing SMTP will eventually get access. Deactivating option to spam at will makes life of criminals looking for the-easy-way look for other alternatives. This new policy leaves plenty of details about perpetrator's identity for police to investigate.
Update 19th Sep 2022:
Hetzners rules are both of these need to be true:
- First invoice must be paid
- Account must be at least month old
By paying, your SMTP-ports won't open. They open only after bit of a wait.
Post-passwords life: Biometrics for your PC
Monday, July 4. 2022
Last year I did a few posts about passwords, example. The topic is getting worn out as we have established the fact about passwords being a poor means of authentiaction, how easily passwords leak from unsuspecting user to bad people and how you really should be using super-complex passwords which are stored in a vault. Personally I don't think there are many interesting password avenues left to explore.
This year my sights are set into life after passwords: how are we going to authenticate ourselves and what we need to do to get there.
Biometrics. A "password" everybody of us carries everywhere and is readily available to be used. Do the implementation wrong, leak that "password" and that human will be in big trouble. Biometric "password" isn't so easy to change. Impossible even (in James Bond movies, maybe). Given all the potential downsides, biometrics still beats traditional password in one crucial point: physical distance. To authenticate with biometrics you absolutely, positively need to be near the device you're about to use. A malicious cracker from other side of the world won't be able to brute-force their way trough authentication unless they have your precious device at their hand. Even attempting any hacks remotely is impossible.
While eyeballing some of the devices and computers I have at hand:
The pics are from iPhone 7, MacBook Pro and Lenovo T570. Hardware that I use regularily, but enter password rarely. There obviously exists other types of biometrics and password replacements, but I think you'll catch the general idea of life after passwords.
Then, looking at the keyboard of my gaming PC:
Something I use on daily basis, but it really puzzles me why Logitech G-513 doesn't have the fingerprint reader like most reasonable computer appliance does. Or generally speaking, if not on keyboard could my self assembled PC have a biometric reader most devices do. Why must I suffer from lack of simple, fast and reliable method of authentication? Uh??
Back-in-the-days fingerprint readers were expensive, bulky devices weren't that accurate and OS-support was mostly missing and injected via modifying operating system files. Improvements on this area is something I'd credit Apple for. They made biometric authentication commonly available for their users, when it became popular and sensor prices dropped, others followed suit.
So, I went looking for a suitable product. This is the one I ended up with:
A note: I do love their "brief" product naming!
It is a Kensington® VeriMark™ Fingerprint Key supporting Windows Hello™ and FIDO U2F for universal 2nd-factor authentication. Pricing for one is reasonable, I paid 50€ for it. As I do own other types of USB/Bluetooth security devices, what they're asking for one is on par with market. I personally wouldn't want a security device which would be "cheapest on the market". I'd definitely go for a higher price range. My thinking is, this would be the appropriate price range for these devices.
Second note: Yes, I ended up buying a security device from company whose principal market on mechanical locks.
Here is one of those lock slots on the corner of my T570:
From left to right, there is a HDMI-port, Ethernet RJ-45 and a Kensington lock slot. You could bolt the laptop into a suitable physical object making the theft of the device really hard. Disclaimer: Any security measure can be defeated, given enough time.
Back to the product. Here is what's in the box:
That would be a very tiny USB-device. Similar sized items would be your Logitech mouse receiver or smallest WiFi dongles.
With a Linux running lsusb, following information can be retrieved:
Bus 001 Device 006: ID 06cb:0088 Synaptics, Inc.
Doing the verbose version lsusb -s 1:6 -vv, tons more is made available:
Bus 001 Device 006: ID 06cb:0088 Synaptics, Inc.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 255 Vendor Specific Class
bDeviceSubClass 16
bDeviceProtocol 255
bMaxPacketSize0 8
idVendor 0x06cb Synaptics, Inc.
idProduct 0x0088
bcdDevice 1.54
iManufacturer 0
iProduct 0
iSerial 1 -redacted-
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0035
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xa0
(Bus Powered)
Remote Wakeup
MaxPower 100mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 5
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x83 EP 3 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 4
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x84 EP 4 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0010 1x 16 bytes
bInterval 10
Device Status: 0x0000
(Bus Powered)
So, this "Kensington" device is ultimately something Synaptics made. Synaptics have a solid track-record with biometrics and haptic input, so I should be safe with the product of my choice here.
For non-Windows users, the critical thing worth mentioning here is: There is no Linux support. There is no macOS support. This is only for Windows. Apparently you can go back to Windows 7 even, but sticking with 10 or 11 should be fine. A natural implication for being Windows-only leads us to following: Windows Hello is mandatory (I think you should get the hint from the product name already).
Without biometrics, I kinda catch the idea with Windows Hello. You can define a 123456-style PIN to log into your device, something very simple for anybody to remember. It's about physical proximity, you need to enter the PIN into the device, won't work over network. So, that's kinda ok(ish), but with biometrics Windows Hello kicks into a high gear. What I typically do, is define a rather complex alphanumeric PIN to my Windows and never use it again. Once you go biometrics, you won't be needing the password. Simple!
Back to the product. As these Kensington-people aren't really software-people, for installation they'll go with the absolutely bare minimum. There is no setup.exe or something any half-good Windows developer would be able to whip up. A setup which would execute pnputil -i -a synaWudfBioUsbKens.inf with free-of-charge tools like WiX would be rather trivial to do. But noooo. Nothing that fancy! They'll just provide a Zip-file of Synaptics drivers and instruct you to right click on the .inf-file:
To Windows users not accustomed to installing device drivers like that, it is a fast no-questions-asked -style process resulting in a popup:
When taking a peek into Device Manager:
My gaming PC has a biometric device in it! Whoo!
Obviously this isn't enough. Half of the job is done now. Next half is to train some of my fingers to the reader. Again, this isn't Apple, so user experience (aka. UX) is poor. There seems not to be a way to list trained fingers or remove/update them. I don't really understand the reasoning for this sucky approach by Microsoft. To move forward with this, go to Windows Settings and enable Windows Hello:
During the setup-flow of Windows Hello, you'll land at the crucial PIN-question:
Remeber to Include letters and symbols. You don't have to stick with just numbers! Of course, if that suits your needs, you can.
After that you're set! Just go hit ⊞ Win+L to lock your computer. Test how easy it is to log back in. Now, when looking at my G-513 it has the required feature my iPhone 7, MBP and Lenovo has:
Nicely done!
123RF Leak - Fallout
Sunday, July 3. 2022
Your and my data is leaked left and right all the time by organizations who at the time of you entering your precious PII (or Personally Identifiable Information) solemny swear to take really good care of it. Still they fail at it. By adapting the famous speech by John F. Kennedy: "Not because it is easy, but because it is hard." Even companies whose business is information security get hacked, or mis-configure their systems or human administrators fail at a minor thing causing major disasters, causing data to leak. Because it is very very hard to protect your data. Let alone companies whose business is not information security, they simply want to conduct their business on-line and not to focus in nurturing YOUR information. When a flaw in their system or procedures is found by malicious actors, data will be leaked.
I don't think this is about to change anytime soon. Unfortunately.
Back in 2020, a site called 123RF.com was pwned. How, or by whom is irrelevant. They failed protecting MY data. More details are available in Bleeping Computer article Popular stock photo service hit by data breach, 8.3M records for sale.
As anybody can expect, there are negative effects on such leaks. Today, such negativity manifests itself in my mailbox as follows:
A company, who has nothing to do with me is sending email to my unique 123RF-dedicated address, stating my attempt to request password recovery on their on-line service has failed. Ok, intersting. Not cool. 
However inconvenient this is to me, I fail to find the attacker's angle on this. By sending 8.3 million password recovery attempts to a public endpoint is a far fetch. They might by shear luck get access to somebody's account on the targeted site. Most likely not. But it's worth the try as it won't cost them anything and very easy to do.
This exact same has happened to me multiple times. An example from 2018 is my blog post DocuSign hacked: Fallout of leaked E-mail addresses. Given all the time and effort I put into creating mail system where I can have unique addresses to all possible use-cases it will be easy to identify the leaking source. Me and many others thought EU's GDPR was supposed to help every single Internet user with this, but still Wired is writing in their May 2022 article How GDPR Is Failing (sorry about the paywall, I'm a subscriber).
I'd like to end this by expressing my generic despair and agony not targeted towards anybody in particular while still targeted towards all stakeholders. It's a movie quote from the original -68 Planet of the Apes where at the end of the film Charlton Heston's character understands what had happened and how he ended up in that mess:
Ah, damn you! God damn you all to hell!
FTC-case: Cafepress leak
Thursday, June 30. 2022
Couple days ago I got an email to a really old address of mine. I happen to own number of domains and most of them are connected to the mail system on my home Linux server. Many many years ago I stopped using a single email address for the simple reason that I couldn't identify to whom I had given that address to. This is a design flaw in archaic asynchronous messging methods like snail mail, telephone calls or SMTP. You simply don't know who is contacting you and where did they obtain your information from.
To defeat this design flaw I have an unique email address for every single service I ever hand my information to. That particular address in question pre-dates my system and is roughly 20 years old. So, no way of knowing who originally leaked it or how many times the address has been leaked. As I don't use it anywhere anymore, an email sent there is with 99,9% confidence junk.
Why I'm writing about that email is obvious, it had very interesting content. Sender claimed to be (a company?) called CafePress. Something I've never head of. Another reason why I kept investigating this maybe-spam was the subject of "Notice of FTC Settlement - 2019 Data Breach". Ok. Federal Trade Commission of USA was addressing me via CafePress. Really interesting!
First the technical ones. To deduce if this was spam or not, I always check the mail headers. This baby had both SPF and DKIM verified correctly. Both technical measures are pretty much a must for anybody to accept the incoming set of bytes. Well, also spammers know this and typically use Google or Microsoft as spam platforms via their free-of-charge mail offerings.
As technical details checked out, next the actual sending server. It originated to something called https://www.sparkpost.com/. Again, something I've never heard of. Ultimately this mail passed all techical checks. My thinking started leaning towards this being the Real Deal.
Email contents:
Dear Valued Customer,
We are contacting you about the 2019 breach of your information collected by the prior owners of CafePress. This notice is about that breach, which you may have already been notified of.
We recently reached a settlement with the Federal Trade Commission, the nation's consumer protection agency, to resolve issues related to the 2019 data breach, and to make sure CafePress keeps your information safe. What happened?
Before November 2019, CafePress didn't have reasonable practices to keep your information safe. When the company had a security breach, the following information about you may have been stolen: your email address, password, name, address, phone number, answers to your security questions, and the expiration date and last four digits of your credit card.
What you can do to protect yourself
Here are some steps to reduce the risk of identity theft and protect your information online:
<blah blah removed>
Sincerely,
Chris Klingebiel
General Manager, CafePress
Hm. Interesting.
Somebody in a company I've never heard of leaked my data in 2019 by a hack. Well, why did they have my information on the first place!
Googling the topic for more details:
Article Commission orders e-commerce platform to bolster data security and provide redress to small businesses is available on FTC.gov. It was legit after all!
Press release mentionded following: "Residual Pumpkin Entity, LLC, the former owner of CafePress, and PlanetArt, LLC, which bought CafePress in 2020". Whoa! Even more companies which I never heard of.
Some highlights:
According to the complaint, a hacker exploited the company’s security failures in February 2019 to access millions of email addresses and passwords with weak encryption; millions of unencrypted names, physical addresses, and security questions and answers; more than 180,000 unencrypted Social Security numbers; and tens of thousands of partial payment card numbers and expiration dates.
Obviously this CafePress (or whoever it is) didn't do much of a job for protecting their stolen/bought/downloaded-from-DarkNet -data. It's a good thing FTC gave them a slap-on-the-wrist -punishment as the company is clearly crooked. Why an earth didn't FTC do more follow up on the origins of the data? I'd definitely love to hear how this CafePress got my data into their hands. I didn't volunteer it, that's for sure!
Really puzzling case this. I suppose it speaks volumes about the modern state of Internet.
Diverse spam
Thursday, May 5. 2022
It seems detecting and blocking email spam is easy. That's something I stated in my blog post back in 2020.
Here I'm introducing three types of spam I'm getting.
Spam from free email accounts
But wait! That's not new. We've seen email originating from Yahoo for many years.
What is new, the fact that increasing number of spammers are creating tons of accounts into Gmail or Outlook.com. As both free-of-charge services limit the amount of email one can send during 24 hours into 500, quite a lot of accounts are needed. As example, see Gmail's Limits for sending & getting mail.
The reason those two services are used for spamming is the good reputation and both are well accepted among postmasters. As an example I won't block neither of them and consider both being as reliable. As end result, majority of spam I receive is from either of those. To weed of bad ones from good ones, traditional spam-filtering will do the trick.
What's good is both Microsoft and Google are really good in detecting ill behaviour and stopping the activity. However, criminals are really good in spending their 500 mails effectively to make their behaviour as "natural" as possible. Neither service provider isn't fast/good enough to detect and stop the activity. Unfortunately.
SMS / iMessage spam
Getting spam via SMS isn't new, but getting junk from Apple iMessage is. Here is a sample:
For non-Finnish readers, the spam says I've been selected for a part time job. Daily salary range is from 10 to 200 euros.
It seems this weird scam is active all over Europe. See iPhone users lament for spam messages via iMessage. Unlike Android, iPhone doesn't have any mechanisms for detecting SMS-spam. Maybe it should?
Probably those spammers didn't even break anything (much). It is likely they've automated something to be able to send crap to everybody. I don't think iMessage protocol isn't broken nor Apple's authentication. All those jerks did was automate the thing to the hilt.
GPG spam
When spam arrives in a heavily encrypted email, then you'll know it must be very important! Here is one I got encrypted with my GPG-key:
What's weird is the choice of language. That sample is in Burmese, but I got one in Hindi and Telugu. All of which point heavily to Asia and specifically to Indian region. I have no idea why I got those as I definitely need Google Translate to decipher those.
I have no idea how my address leaked, but the one I'm using is from https://keyserver.pgp.com/, a directory where I choose to publish my PGP / GPG public key. Maybe somebody hacked that? Or just iterated all possible keys. I dunno.
Wrap-up
I see all this as a beginning. Email generally is losing emphasis and Discord / Teams / Slack are gaining on that. Spammers are getting really creative and I'm sure criminals will diversify more. Back-in-the-days Skype spam was a a real thing, see my article from 2016 on that.
pkexec security flaw (CVE-2021-4034)
Wednesday, January 26. 2022
EU Digital COVID Certificate trust-lists
Sunday, January 9. 2022
It seems vacdec is one of my most popular public projects. I'm getting pull requests, comments and questions from number of people.
Thank you for all your input! Keep it coming. All pull requests, bug fixes and such are appreciated.
Briefly, this is about decoding EU digital COVID certificate QR-codes. For more details, see my blog post from August -21 or the Python source-code at https://github.com/HQJaTu/vacdec.
Quote from my original post: "the most difficult part was to get my hands into the national public certificates to do some ECDSA signature verifying".
My original code has tool called fetch-signing-certificates.py to do the heavy lifting. It fetches a list of X.509 certificates which are used in signing the QR-code's payload. That list is stored into your hard drive. When you do any COVID certificate decoding, that list is iterated with purpose of finding the key used when the QR-code was generated. If the trust-list contains the cert, signature will be verified and a result from that will be given.
To state the obvious: That precious trust list will contain all signature certificates from all countries using the same EU COVID certificate QR-code system. The list of countries is long and includes all 27 EU-countries. Among participating non-EU countries are: Andorra, United Arab Emirates, Swizerland, Georgia, Iceland, Liechtenstein, Morocco, Norway, Singapore, San Marino, Ukraine, Vatican. Looking at the country-list, it is easy to see how EU COVID passport is a triumph of open-source!
On the flipside, getting signing certificates from all those countries requires some effort. A participating country first needs to generate and secure (emphasis: SECURE the private key) required number of ECDSA public/private key -pairs to all organizations/parties doing the COVID certificate issuing. Of those key-pairs COVID certificate signing X.509 certificates can be created and public certificate will be added to the common trust-list.
As an example, in my home country of Finland up until recently there used to be one (1 as in single) X.509 cert for entire nation of 5,6 million people, now there is two. Ultimately, pretty much everybody gets their COVID certs signed from a single cert. This is because in Finland issuing the certs is centralized. On the other hand bigger countries need to have multiple X.509 certs as their government organization may be different and less centralized. A good example is Germany, the most populous countru in Europe and a federal state consisting of 17 constituent states. Having single cert would not make sense in their government.
Moving on. Countries deliver their public certificates into some EU "central repository". It is not known how and where this is, but I'm assuming one has to exist. From this repository the participating countries are allowed the retrieve the trust-list for their verification needs. Subsequently the list is distributed by the country to those organizations and people needing to do COVID certificate verification. Remember "get my hands into the national public certificates"? Not all countries do the distributing very easily nor publicly. Some do, and that's where I get my data from.
Now that there is access of trust-lists, factor in time. That little devil is always messing everything up!
In August, when I first download the Austrian list, the entire list had 150+ valid signing certs. Early December -21 there was 192. Early January -22 there was already 236. Today, the Austrian list contains 281 signing certificates. However, Swedish list has only 267. Argh! I'd really really need access to that EU endpoint for trust lists.
As the Austrian list is larger, here are the per-country statistics of 9th January 2022, expect this to change almost daily:
| Country ISO 3166-2 code | Count signing certs | EU member country name | Participating country name | Test data exists |
|---|---|---|---|---|
| DE | 57 | Germany | x | |
| FR | 50 | France | x | |
| ES | 23 | Spain | x | |
| NL | 21 | Netherlands | x | |
| CH | 13 | Switzerland | x | |
| MT | 12 | Malta | x | |
| LU | 8 | Luxembourg | x | |
| GB | 6 | UK | ||
| LI | 6 | Liechtenstein | x | |
| NO | 6 | Norway | x | |
| IT | 4 | Italy | x | |
| NZ | 4 | New Zealand | ||
| LT | 3 | Lithuania | x | |
| LV | 3 | Latvia | x | |
| MC | 3 | Monaco | ||
| PL | 3 | Poland | x | |
| AL | 2 | Albania | ||
| AT | 2 | Austria | x | |
| BE | 2 | Belgium | x | |
| BG | 2 | Bulgaria | x | |
| CZ | 2 | Czech Republic | x | |
| DK | 2 | Denmark | x | |
| EE | 2 | Estonia | x | |
| FI | 2 | Finland | x | |
| FO | 2 | Faroe Islands | ||
| GE | 2 | Georgia | x | |
| HR | 2 | Croatia | x | |
| IE | 2 | Ireland | x | |
| IS | 2 | Iceland | x | |
| MK | 2 | North Macedonia | ||
| SE | 2 | Sweden | x | |
| SG | 2 | Singapore | x | |
| AD | 1 | Andorra | x | |
| AE | 1 | United Arab Emirates | x | |
| AM | 1 | Armenia | ||
| CV | 1 | Cabo Verde | ||
| CY | 1 | Cyprus | x | |
| GR | 1 | Greece | x | |
| HU | 1 | Hungary | ||
| IL | 1 | Israel | ||
| LB | 1 | Lebanon | ||
| MA | 1 | Morocco | x | |
| MD | 1 | Moldova | ||
| ME | 1 | Montenegro | ||
| PA | 1 | Panama | ||
| PT | 1 | Portugal | x | |
| RO | 1 | Romania | x | |
| RS | 1 | Serbia | ||
| SI | 1 | Slovenia | x | |
| SK | 1 | Slovakia | x | |
| SM | 1 | San Marino | x | |
| TG | 1 | Togo | ||
| TH | 1 | Thailand | ||
| TN | 1 | Tunisia | ||
| TR | 1 | Turkey | ||
| TW | 1 | Taiwan | ||
| UA | 1 | Ukraine | x | |
| UY | 1 | Uruguay | ||
| VA | 1 | Vatican | x | |
| 281 | 27 | 32 | 38 |
Any test data provided by participating country is at: https://github.com/eu-digital-green-certificates/dgc-testdata
Things to note from above table:
- Countries have their government arranged with lot of different variations.
- More certificates --> more things to secure --> more things to be worried about.
- In Germany somebody got access to a cert and issued QR-codes not belonging to actual persons.
- Lot of countries outside EU chose to use this open-source the system.
- Assume more countries to join. No need for everybody to invent this system.
- Lack of test data
- In EU, it is easy to make test data mandatory. Outside EU not so much.
- Way too many countries have not provided any sample certificates.
- Faroe Islands have 50k people, two certs.
- Okokok. It makes sense to have already issued two. If one cert leaks or gets "burned" for any reason, it is fast to re-issue the COVID certificates by using the other good cert.
- Country stats:
- Liechtenstein has 40k people, 6 certs! Whooo!
- Malta and Cabo Verde have 500k people. Maltese need 12 certs for all the signing! In Cabo Verde they have to do with only one.
- Vatican is the smallest country in the entire World. Single cert needed for those 800+ persons living there.
- Italy has 60M people, they manage their passes with only 4 certs. Nice!
- In total 59 countries participating this common system. Adoption is wide, system is well designed.
Also note how "EU" COVID passport has more participating countries outside EU.
True testament of open-source, indeed!
Decoding EU Digital COVID Certificate into human-readable format
Saturday, December 11. 2021
Passwords, December 2021 Follow up
Monday, December 6. 2021
This year, I've done quite a few posts about passwords. There are still couple days left before 2022 begins, so another piece about passwords is needed.
Most commonly used passwords
For reason not really known to me, at the end of each year there is a reccurring topic: Lists of most commonly used passwords. Here is one example, a Techspot article from November 2021 The most common passwords of 2021 are outright embarrassing. In the article, they're referring to another article by Nordpass Top 200 most common passwords. Nordpass seems to be the password manager software by NordVPN.
In the Nordpass-article, you can select a list to look at. In their back-end the front-app loads the data as JSON from per-country lists. Example: All countries list has URL of https://nordpass.com/json-data/top-worst-passwords/findings/all.json. The one I, as a Finn, was most interested in was the Finnish most used passwords from https://nordpass.com/json-data/top-worst-passwords/findings/fi.json.
Bit of processing with jq to extract top-30 list with use frequency:
jq -r '.[] | "\(.Rank): \(.Password) [\(.User_count)]"' all.json | head -30
Resulting:
1: 123456 [103170552]
2: 123456789 [ 46027530]
3: 12345 [ 32955431]
4: qwerty [ 22317280]
5: password [ 20958297]
6: 12345678 [ 14745771]
7: 111111 [ 13354149]
8: 123123 [ 10244398]
9: 1234567890 [ 9646621]
10: 1234567 [ 9396813]
11: qwerty123 [ 8933334]
12: 000000 [ 8377094]
13: 1q2w3e [ 8204700]
14: aa12345678 [ 8098805]
15: abc123 [ 7184645]
16: password1 [ 5771586]
17: 1234 [ 5544971]
18: qwertyuiop [ 5197596]
19: 123321 [ 5168171]
20: password123 [ 4681010]
21: 1q2w3e4r5t [ 4624323]
22: iloveyou [ 4387925]
23: 654321 [ 4384762]
24: 666666 [ 4329996]
25: 987654321 [ 4239959]
26: 123 [ 3606937]
27: 123456a [ 3493177]
28: qwe123 [ 3284938]
29: 1q2w3e4r [ 3197899]
30: 7777777 [ 3112046]
Ok. Now we learn using 123456 is really common. Nordpass has recorded 103 million times that password being used.
Q: Where does this password data come from?
Yeah, I know the above list contains really commonly used passwords.Or does it?
From which data do we know the above list to contain top-30 most commonly used passwords? Isn't it kinda suspicious for a password vault company to publish this kind of information? Do they know what passwords do you use? How do they know that 123456 is being used over 100 million times?
There is no answer to the set of questions. Nordpass don't say, so I need to speculate and guess.
Harvesting the passwords people do use
According to John Wetzel of Recorded Future passwords are leaked constantly:
Actually, dumps with leaked passwords are easily available in the net. Even I have millions of passwords from various leaks. That could be one source to measure bad passwords, see which ones are leaked the most.
Q: What if only bad passwords leak?
Either Nordpass cheats and they do know which passwords their customers use, OR there is a case of survivorship bias. The bad password ended up being hijacked, put to a database, leaked and picked up by number of people wanting to see what passwords are being used just because it was poor one to begin with. Back-in-the-days there were cases where users' passwords
I don't know if that's the case. Not disclosing the source makes me wonder if Nordpass's ethics is bad or if their password manger is bad allowing them to see what the password is.
Life after passwords
A lot of systems being used allow users to change password and while doing it put a super simple one. Also a big problem are lists of default passwords of devices sold making the password easily guessable or non-existent as they're commonly known.
In UK they're fighting against passwords hard time: Ban on default passwords in new UK law. Nice! As a lot of device manufacturers choose to go the cheapest way, some legislation will be needed to smack some sense to them. If a device has simple default password, they won't allow selling it. We definitely should have that ruling in effect in the EU-side too!
One mechanism to rid passwords is WebAuthN. As it doesn't seem to get the traction, yet one initiative was launched Decentralized Identity Foundation (DIF). What they're proposing is to kinda reverse the authentication problem and let you control your own data allowing you to define which service will verify you're you to a website you're logging into. That would solve bunch of problems if being commonly used. However, DIF is such a new proposal, we don't know yet if that's going to fly or not.
EU Digital COVID Certificate, December 2021 edition
Sunday, December 5. 2021
Since my August blog post about vacdec, the utility to take a peek into COVID-19 passport internals, I've got lots of feedback and questions. The topic, obviously, touches all of us. As this global pandemic won't end, I've been following different events, occurrences and incdients around digital COVID certificates.
TV and Print media bloopers
To anybody working with software, computers, data, encoding and transmitting data, especially in the era of GDPR, it's second nature to classify data. There is public data, confidential data, data containing GDPR-covered personal details, to classify by simple three criteria.
What's puzzling on how EU Digital COVID Certificate QR-codes where shown in media. Personally I contacted four different journalists in fashion of "Hey, you published all of your personal details in form of a QR-code. Please, retract and do not ever do that again." One journalist I talked in person and his comment was "I couldn't fathom this black-and-white blob should be kept secret!"
These leaks weren't by small and insignificant media. The codes I saw were portrayed from 9 o'clock evening TV news (cinematographer's cert), website of print media (journalist's own cert), Internet news video (producer's cert) and paper print of news (not sure whose cert that was).
As these bloopers aren't regularily seen anymore, it seems all of the media outlets have issued internal memos for not to display the QR-codes in a readable format anymore. I have facts of one media outlet's internal memo informing people not to publish real certs publicly. One positive change is most of them are using sample certs when they really want to portray a QR-code.
Brute-forcing ECDSA cryptography
In one instance, I bumped into a script-kiddies masterpiece. He was convinced of the possibility of brute-forcing the ECDSA-256 private key of one COVID Certificate issuer. I eyeballed the Python-code and indeed, it approached the problem by doing an nearly-forever loop of picking 256 random bits to match those bits against the public key. Nice! By carefully choosing the bits, that is how the thing works.
However, some suspicions rose. There was a comment in the discussion thread saying: "I don't think this code is working. My home computer has been crunching this for two weeks now and there doesn't seem to be any results."
Really! REALLY!!
These kids really have no clue.
Hints for brute-forcing ECDSA-crypto:
- If it was meant to be that easy, everybody was cracking the private key.
- Don't do random bits, instead go through 2^256 bits in sequence. By going random, there is no mechanism to check if that combination has been attempted already.
- Go parallel, split the sequence into chunks and do it simultaneously with multiple computers.
- Two to 256th power is: 115792089237316195423570985008687907853269984665640564039457584007913129639936. If for some reason you come to a conclusion of that particular number being on the larger scale, your conclusion is correct. IT IS!
- If some magic lottery bounces your way and you happend to find the correct sequence. For the love of god, do not tell anybody else of your findings. It takes you couple gazillion years to brute force the key, for the opposing party couple seconds to revoke it.
- Make sure to put in couple lottery tickets while at it. Chances are you'll become rich before finding the private key.
Forged Digital COVID Certificates
Somebody in Germany got their hands on the actual certificate issuing private key and did throw out couple fully valid and totally verifiable COVID certificates for different names. There were at least three different ones circulating that I know of. Here is one (notice the green color indicating validity):
This one was targeting Finnish customer pool and had first name of "Rokotepassieu" which translates as "Vaccination passport EU". Surname translated: "Contact me via Wickr" (for those who don't do Wickr, it's an AWS owned messaging app with highest standards on security).
According to Swedish government list of valid certificates, Germany has 57 sets of keys in use (disclaimer: at the time of writing this post). What German government had to do is revoke the public key which private key was misplaced. For thousands of people who had a valid COVID cert, they had to get theirs again with different signing key.
This kind of incident/leakage obviously caught attention from tons of officials. Making sellers' business dry up rather swiftly. Unfortunaly no details of this incident were disclosed. The general guess is for some underpaid person to be doing darknet moonlighting on the side with his access to this rare and protected resource.
Leaked Digital COVID Certificates
In Italy roughly thousand COVID certs issued to actual people were made available. I downloaded and processed couple thousand files in four different leaks just to find out most of the leaked certs were duplicates. I did combine and de-duplicate publicly available data. Here are the stats:
All leaked certs are still valid (disclaimer: at the time of writing this post). However, Italian government has only three sets of keys in place and replacing one would mean rendering 1/3 of all issued certificates as invalid. They have not chosen to do that. Most likely because the certs in circulation are harvested by somebody doing the checking with a malicious mobile app.
No surprises there. Most of them are for twice vaccinated people. Couple certs are found for three times vaccinated, test results and recovered ones. What's sad is the fact that such a leak exists. These QR-codes are for real human beings and their data should be handled with care. Not cool.
What next
This unfortunate pandemic isn't going anywhere. More and more countries are expanding the use of COVID certs in daily use. For any cert checking to make sense, the cert must be paired with person's ID-card. Crypto math with the certs is rock solid, humans are the weak link here.
MySQL Java JDBC connector TLSv1 deprecation in CentOS 8
Friday, November 12. 2021
Yeah, a mouthful. Running CentOS 8 Linux, in Java (JRE) a connection to MySQL / MariaDB there seems to be trouble. I think this is a transient issue and eventually it will resolve itself. Right now the issue is real.
Here is the long story.
I was tinkering with Databricks. The nodes for my bricks were on CentOS 8 and I was going to a MariaDB in AWS RDS. with MySQL Connector/J. As you've figured out, it didn't work! Following errors were in exception backtrace:
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)
Weird.
Going to the database with a simple CLI-command of (test run on OpenSUSE):
$ mysql -h db-instance-here.rds.amazonaws.com -P 3306 \
-u USER-HERE -p \
--ssl-ca=/var/lib/ca-certificates/ca-bundle.pem \
--ssl-verify-server-cert
... works ok.
Note: This RDS-instance enforces encrypted connection (see AWS docs for details).
Note 2: Term used by AWS is SSL. However, SSL was deprecated decades ago and the protocol used is TLS.
Taking a look on the wire:
Two details popped out instantly: TLSv1 and TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA cipher. Both deprecated. Both deemed highly insecure and potentially leaking your private information.
Why would anybody using those? Don't MySQL/MariaDB/AWS -people remove insecure stuff from their software? What! Why!
Troubleshooting started. First I found SSLHandShakeException No Appropriate Protocol on Stackoverflow. It contains a hint about JVM security settings. Then MySQL documentation 6.3.2 Encrypted Connection TLS Protocols and Ciphers, where they explicitly state "As of MySQL 5.7.35, the TLSv1 and TLSv1.1 connection protocols are deprecated and support for them is subject to removal in a future MySQL version." Well, fair enough, but the bad stuff was still there in AWS RDS. I even found Changes in MySQL 5.7.35 (2021-07-20, General Availability) which clearly states TLSv1 and TLSv1.1 removal to be quite soon.
No amount of tinkering with jdk.tls.disabledAlgorithms in file /etc/java/*/security/java.security helped. I even created a simple Java-tester to make my debugging easier:
import java.sql.*;
// Code from: https://www.javatpoint.com/example-to-connect-to-the-mysql-database
// 1) Compile: javac mysql-connect-test.java
// 2) Run: CLASSPATH=.:./mysql-connector-java-8.0.27.jar java MysqlCon
class MysqlCon {
public static void main(String args[]) {
try {
Class.forName("com.mysql.cj.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:mysql://db.amazonaws.com:3306/db", "user", "password");
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("select * from emp");
while (rs.next())
System.out.println(rs.getInt(1) + " " + rs.getString(2) + " " + rs.getString(3));
con.close();
} catch (Exception e) {
System.out.println(e);
e.printStackTrace(System.out);
}
}
}
Hours passed by, but no avail. Then I found command update-crypto-policies. RedHat documentation Chapter 8. Security, 8.1. Changes in core cryptographic components, 8.1.5. TLS 1.0 and TLS 1.1 are deprecated contains mention of command:
update-crypto-policies --set LEGACY
As it does the trick, I followed up on it. In CentOS / RedHat / Fedora there is /etc/crypto-policies/back-ends/java.config. A symlink pointing to file containing:
jdk.tls.ephemeralDHKeySize=2048
jdk.certpath.disabledAlgorithms=MD2, MD5, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=DH keySize < 2048, TLSv1.1, TLSv1, SSLv3, SSLv2, DHE_DSS, RSA_EXPORT, DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_DSS_EXPORT, DH_RSA_EXPORT, DH_anon, ECDH_anon, DH_RSA, DH_DSS, ECDH, 3DES_EDE_CBC, DES_CBC, RC4_40, RC4_128, DES40_CBC, RC2, HmacMD5
jdk.tls.legacyAlgorithms=
That's the culprit! It turns out any changes in java.security -file won't have any effect as the policy is loaded later. Running the policy change and set it into legacy-mode has the desired effect. However, running ENTIRE system with such a bad security policy is bad. I only want to connect to RDS, why cannot I lower the security on that only? Well, that's not how Java works.
Entire troubleshooting session was way too much work. People! Get the hint already, no insecure protocols!
OpenSSH 8.8 dropped SHA-1 support
Monday, October 25. 2021
WebAuthN Practically - iOS 15
Monday, September 20. 2021
As Apple has recently released iOS 15, and iPadOS 15 and macOS 12 will be released quite soon. Why that is important is for Apple's native support for WebAuthN. In my WebAuthN introduction -post there is the release date for the spec: W3C Recommendation, 8 April 2021. Given the finalization of the standard, Apple was the first major player to step forward and start supporting proper passwordless authentication in it's operating systems. For more details, see The Verge article iOS 15 and macOS 12 take a small but significant step towards a password-less future.
For traditional approach with USB-cased Yubikey authenticator, see my previous post.
Registration
Step 1: Enter the username you'd like to register as.
Step 2: Go for Register
Step 3: Your browser will need a confirmation for proceeding with registration.
In Apple's ecosystem, the private key is stored into Apple's cloud (what!?). To allow access to your cloud-based secerts-storage, you must enter your device's PIN-code and before doing that, your permission to proceed is required.
Note: The option for "Use Security Key" is for using the Yubikey in Lightning-port. Both are supported. It is entirely possible to login using the same authenticator with a USB-C in my PC or Mac and Lightning with my iPhone or iPad.
Step 4: Enter your device PIN-code
Step 5: You're done! Now you have successfully registered.
Best part: No passwords! Private key is stored into Syncing Platform Authenticator. Btw. weird name that for WebAuthN in Apple-lingo. Ok, to be honest, WebAuthN is a mouthful too.
This was couple steps simpler than with Yubikey. Also there is the benefit (and danger) of cloud. Now your credential can be accessed from your other devices too.
Login
Step 1: Enter the username you'd like to log in as.
Step 2: Go for Login
Step 3: Your browser will need a confirmation for proceeding with login. A list of known keys and associated user names will be shown.
Step 4: Enter your device PIN-code
Step 5: You're done! Now you have successfully logged in.
Best part: No passwords!
That's it. Really.
Finally
I don't think there is much more to add into it.
In comparison to Yubikey, any of your Apple-devices are authenticators and can share the private key. Obviously, you'll need iOS 15 or macOS 12 for that support.
