Hacker's ramblings

This week seems to be especially hard on my Linuxes. Doing a regular pacman -Syu started spitting crap on me:

error: failed to commit transaction (conflicting files)
man-db: /usr/bin/accessdb exists in filesystem
man-db: /usr/bin/apropos exists in filesystem
man-db: /usr/bin/catman exists in filesystem
man-db: /usr/bin/convert-mans exists in filesystem
man-db: /usr/bin/lexgrog exists in filesystem
man-db: /usr/bin/man exists in filesystem
man-db: /usr/bin/mandb exists in filesystem
man-db: /usr/bin/manpath exists in filesystem
man-db: /usr/bin/whatis exists in filesystem
man-db: /usr/lib/man-db/globbing exists in filesystem
man-db: /usr/lib/man-db/libman-2.7.6.1.so exists in filesystem
man-db: /usr/lib/man-db/libman.so exists in filesystem
man-db: /usr/lib/man-db/libmandb-2.7.6.1.so exists in filesystem
...

A simple query for what's wrong:

# pacman -Qkk man-db
man-db: 363 total files, 0 altered files

So, nothing wrong with it. It just loves busting my balls!

Using a bigger hammer:

# pacman -S --force man-db
...
:: Processing package changes...
(1/1) upgrading man-db      [######################] 100%
New optional dependencies for man-db
    gzip [installed]
:: Running post-transaction hooks...
(1/2) Creating temporary files...
(2/2) Arming ConditionNeedsUpdate...

Now my pacman -Syu works. Weird case, that.

I opened my mail, and I had 730 new e-mails there! Whaat!

One of my Azure boxes has (for reason unknown to me), following crontab-entry on root's crontab:

* * * * * [ \( ! -f /etc/opt/omi/creds/omi.keytab \) -o
   \( /etc/krb5.keytab -nt /etc/opt/omi/creds/omi.keytab \) ] &&
   /opt/omi/bin/support/ktstrip /etc/krb5.keytab /etc/opt/omi/creds/omi.keytab

/opt/omi/bin/support/ktstrip keeps failing, because /etc/krb5.keytab is missing. And that command is run every single minute on my machine. So, every single minute I get a new information about the failure. Nice!

The sequence of events is totally unclear to me. I haven't touched anything, but this morning an influx of e-mails stated pouring in.

OMI, or Open Management Infrastructure is something Linux-images in Azure have, so it shouldn't be anything dangerous.

The obvious fix as to remove that stupid line.

Since most of the revenue earned by my empolyer comes from Candy Crush, a lot of swag around the office is somehow related. Very understandable, that.

This time we all got a Monopoly-game, but not the regular one everybody has been playing. Ok, to be entirely clear on this, threre are a minimum of 37 different language editions of the game, so in which country you spent your childhood, make it depend which streets the game you used to play has. Anyway, this is the Candy Crush -edition of Monopoly:

Also very understandably we did discuss different aspects of the game, one of them being about the winning tactics.

I started thinking, if it would be possible to just go to The Net and get some source code for a Monopoly-simulator and try to program some rules to a bot, which would then play against other bots and see which approach would be more likely to win in the long run.

Quite easily, I found a Python-version of the game by Juan Manuel Contreras from GitHub. Unfortunately, the work by him is quite unfinished, so I chose to fork it and continue from that point. I didn't touch the .py-files, only the Jupyter notebook -version of the game.

As a result of some weekend coding, I have:

• Class for user game logic
• Lots of game functionality, moving around, collecting rent, buying, selling, going bankrupt
• Winning the game

The resulting notebook is quite bug-free, but there are still some game rules, that aren't properly followed. Especially the situation where a player defaults, all the assets need to go to creditor. If creditor is another player, then to that, if bank, there needs to be an auction where other players may purchase the defaulting user's properties. Doing auctions between AI-players is not that easy, so none of that is implemented.

The same AI-module would be needed in a situation where a player lands on an unowned property. According to rules, that needs to be auctioned if the player doesn't want to buy it. Not so easy to do with Python.

Also, the Chance-cards are not implemented, so it is impossible to get a Get-out-of-jail -card. Yet.

By running the game in the sim for couple hundred times, I found that with 5 players, it is very likely to get into a situation where nobody can build houses/hotels. Also, if there are not many houses/hotels, the other players are unlikely to land on those properties and the game will run forever with all players accumulating wealth indefinitely. To get around that, I'm detecting such a situction and artificially doubling rents. That rule is not in the original Monopoly-rules. How human players would solve this situation is to trade/barter properties between each other to gain building rights. A thing not so easy to implement with any programming language.

That Python Monopoly -project was just a fun thing to do. I'm not sure if I'll continue it any further. The code is out there, if you're interested.

Over half an year later, I realized, that I never published my article about operating system and software setup of my Mini-PC router. This is a follow-up post about the Qotom-hardware I wrote earlier. So, its probably about time to do that!

To get the ball rolling on new CentOS installation, a good start is to download it, Rufus it into an USB-stick and install the minimal setup into router-PC. The CentOS installation is so well documented and trivial process, I won't go into any details of it. Read something like Installing Red Hat Enterprise Linux 7.4 on all architectures for details of that.

Goal

Every project needs a goal. In any kind of engineering there is a specification and criteria, that the goal has been met.

The goal of this project is to create a Linux-server capable of securing a local network from the Internet and allow traffic to pass from the LAN to the wild-wild-net.

Spec:

• There is a working CentOS Linux running on the MiniPC
• ISP's cable modem is configured as bridge, no double NATting done
  • MiniPC gets a public IP-address from ISP
  • MiniPC can be accessed from the Net via the IP-address
• Configurations persist a reboot on the MiniPC
• MiniPC issues dynamic IP-addresses to LAN-clients
• MiniPC acts as a caching nameserver to LAN-clients
  • Any requests from the Net are not served
• Wireless access point is configured not do do any routing, aka. it is in access point mode
• The setup is secure with attack surface minimized
• LAN IP-address range is 192.168.1.0/24

Definition of done:

• Internet works!
  • MiniPC can connect to net
  • MiniPC can be connected from net and LAN via SSH
  • Wired clients can connect to net via Ethernet cable without any manual configuration
  • Wireless clients can connec to the net via Wi-Fi without any manual configuration

Step 1: Packages

After minimal installation, the set of tools and packages required includes:
net-tools bind-utils screen tcpdump policycoreutils-python setools

• net-tools: mostly for netstat, using route or ifconfig is deprecated
• bind-utils: for dig and nslookup
• screen: for full-screen window manager
• tcpdump: taking a look into Ethernet and TCP/IP-packages, when something goes wrong, getting detailed view is very important
• policycoreutils-python setools: for managing SELinux

Step 2: Remove NetworkManager

Packages to install: -none needed-

Why a server would have GNOME NetworkManager installed on a server is beyond me. I simply cannot comprehend what CentOS-people are thinking, when they as default treat my server as a laptop. But the main thing is, that this piece of shit needs to go! The quicker, the better!

DANGER!

When you actually run the yum-command to remove NetworkManager, your system will lose all network connectivity. So, please, do run this at a console, not via SSH-connection.

DANGER!

Run command as root on console:

yum erase NetworkManager

Now your system's networking is royally messed up.

Step 3: Setup NICs

Packages to install: -none needed-

Check that NetworkManager created and left ifcfg-files into /etc/sysconfig/network-scripts/. If the appropriate ifcfg-files (one for each interface) are gone, you need to start learning how to write one fast. A good starting point on that would be RedHat Enterprise Linux 7 product documentation, Networking Guide, section 2.2 Editing Network Configuration Files.

LAN interface

Out of the two Ethernet-interfaces, 50/50 coin-flip ended as enp3s0 LAN and enp1s0 WAN. For any practical purposes, it really doesn't matter which one is which, but I'm describing here my setup. If you're using some other hardware, your interface names won't match those.

For any sensible use of your LAN-side, this interface should be connected to a network switch, so that your local network can be shared by your PC, Playstation, TV, Wi-Fi access point or whatever you have there running. Of course you can run it with only one host connected directly to the router.

This is critical: Your LAN-interface MUST have a static IP-address for it. It really cannot act as LAN-side of a router without one.

I chose my LAN to be private IP-range 192.168.1.0/24, so I edited /etc/sysconfig/network-scripts/ifcfg-enp3s0 to contain:

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV6INIT=yes
NAME=enp3s0
UUID=-don't-touch-this-
DEVICE=enp3s0
ONBOOT=yes

NETWORK=192.168.1.0
BROADCAST=193.168.1.255
USERCTL=no

IPADDR=192.168.1.1
PREFIX=24
IPV4_FAILURE_FATAL=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no

WAN interface

For WAN, there is no need to touch enp1s0 setup (much). When your WAN-interface (enp1s0) starts, it will obtain an IP-address from your ISP. With that, it will also get your ISP's DNS-address an overwrite your precious manual changes in /etc/resolv.conf. You don't want that to happen. So, prevent that and edit /etc/sysconfig/network-scripts/ifcfg-enp1s0 and add:

PEERDNS="no"

Well, that was easy!

IP-forwarding

For routing to work, it requires Linux kernel to have IP-forwarding enabled. It will allow network packets to travel between interfaces.

Enable IP-forwarding immediately:

sysctl -w net.ipv4.ip_forward=1

Enable IP-forwarding on boot:

sysctl net.ipv4.ip_forward > /etc/sysctl.d/1_ip_forward.conf

Finalize network setup

When your network interface configurations are ok, restart everything by running following as root:

systemctl enable network
systemctl restart network

Now your system:

• has both interfaces on-line
• is reachable from a machine on your wired-LAN using a static IP-address other than 192.168.1.1.
  Note: your router doesn'ht have DHCPd running yet, so you need to figure out how to configure a static IP-address to your device
• still gets an IP-address from your ISP from your external interface
• can reach IP-addresses via both external and internal interfaces

If these criteria are not met, there is simply no point in proceeding. Your system won't work as a router without those prerequisites.

Finally, make sure that your IPtables-rules have effect. Your box is connected to Internet and it can be accessed/bombarded from there, so run following to secure your setup:

systemctl restart firewalld

Now your system is ready to become a router.

Step 4: Firewalld

Packages to install: -none needed-

Zones

Out-of-box CentOS has firewalld enabled. It has only one zone defined for public wild-wild-net, also TCP/22 SSH is open for the world. This needs to be run as root. First split off LAN into own zone home:

# firewall-cmd --zone home --change-interface enp3s0 --permanent

Check the zones and their assigned interfaces:

# firewall-cmd --get-active-zones
home
  interfaces: enp3s0
public
  interfaces: enp1s0

Setup network address translation (NAT) and allow traffic to flow from your LAN to outside world. Any relevant traffic is allowed to flow in from Internet back to your LAN. Commands to run:

# firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -o enp1s0 -j MASQUERADE
# firewall-cmd --permanent --direct --add-rule ipv4 filter FWDI_home_allow 0 -o enp1s0 -j ACCEPT
# firewall-cmd --permanent --direct --add-rule ipv4 filter FWDI_public_allow 0 -o enp3s0 -m state --state RELATED,ESTABLISHED -j ACCEPT

Enable the DNS-server we'll setup later, also disable any outgoing DNS-queries from your LAN (a security measure):

# firewall-cmd --permanent --zone home --add-service dns
# firewall-cmd --permanent --direct --add-rule ipv4 filter FWDI_home_deny 0 -p udp -m udp --dport 53 -j REJECT

At this point do a reload:

# firewall-cmd --reload

... and test your firewall setup from router:

• You still must be able to access Internet from your router
• Your LAN does work at this point. A client with a static IP must be able to access Internet.

Step 5: Named

Packages to install: bind-chroot

You can continue to use your ISP's nameserver, but I never do that. It makes much more sense to have a caching nameserver running at your own router. This allows your box to go directly to Internet root servers and do all the name queries for you. In many countries ISPs intentionally drop some domains out or are forced by government to do that. Running your own resolver makes sure that you get all the records as is and in case of changes you can flush caches whenever you want and don't have to wait for a record to expire.

Out-of-box the BIND 9.9.4 does not server anybody else than localhost. To fix this, find following two lines in /etc/named.conf:

listen-on port 53 { 127.0.0.1; };
allow-query     { localhost; };

Edit them to contain:

listen-on port 53 { 127.0.0.1; 192.168.1.1; };
allow-query     { localhost; 192.168.1.0/24; };

Finally, change your system's default name resolver by editing /etc/resolv.conf to contain a single line:

nameserver 127.0.0.1

Start the server and enable it to start on boot:

systemctl start named-chroot
systemctl enable named-chroot

Now you're ready to test the setup. Just host www.google.com or your favorite site. Successful reply will include IP-address(es) for your query.

Now your caching nameserver is ready and most probably you'll never have to touch it again.

Step 6: DHCP

Packages to install: dhcp

Edit /etc/dhcp/dhcpd.conf and have it contain:

ddns-update-style interim;
ignore client-updates;
authoritative;
default-lease-time 14400;
max-lease-time 86400;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.1;
option domain-name "my.own.lan";
option domain-name-servers 192.168.1.1;
subnet 192.168.1.0 netmask 255.255.255.0 {
        range 192.168.1.50 192.168.1.99;
}

That piece of configuration will use your router as DNS for the clients and issue them addresses from range .50 - .99.

Start the server and enable it to start on boot:

systemctl start dhcpd
systemctl enable dhcpd

At this point, configure your client to use DHCP for IP-addressing. You must get an IP from the above range, also DNS-resolve and NAT should work, but that's the next step to do. Test it all.

Step 7: Testing it

Make sure:

1. A client in your LAN gets an IP-address from DHCP
2. A client in your LAN can ping your router at 192.168.1.1
3. A client in your LAN can ping something in the Internet, like Google's name server at 8.8.8.8
4. A client in your LAN resolves names, for example: nslookup www.google.com returns IP-addresses
5. A client in your LAN can access https://www.google.com/ via a web-browser

That's it! What else would you need?

Done!

Congratulations on your new router!

What I did next was set up my own DNS-zone so that my hosts had FQDNs. But that's beyond this blog post. Read something like How To Configure BIND as a Private Network DNS Server on CentOS 7 by DigitalOcean for that.

UUUJEA!

Com Hem has been rolling out a native SLAAC/DHCPv6-based IPv6 to it's entire customer base, and they took a very important step 2 in their project. They started honoring Prefix Delegation -requests! To a non-network person that means absolutely nothing, but to a network administrator that is really a game changer!

Normally I don't use much of the features my Sagemcom cable-TV -router has, it's just set to bridge:

Since I failed earlier to get a prefix from my ISP, I was toying around with my router and set it to router-mode, and to my great surprise:

The thing issued my LAN a /64 IPv6-network! Nice.
After putting everything back and eye-balling the DHCPv6 lease file on my Linux-router:

lease6 {
  interface "enp1s0";
  ia-pd c4:d0:0a:85 {
    starts 1515858667;
    renew 302400;
    rebind 483840;
    iaprefix 2a04:ae00::/56 {
      starts 1515858667;
      preferred-life 604800;
      max-life 2592000;
    }
    option dhcp6.status-code success;
  }
}

Yes, it contains an ia-pd -section! The iaprefix from the file is mine, only mine, my precious address space!
I have no idea how long they have been honoring my PD-requests, but on December 17th they didn't.

A generally accepted IPv6-deployment principle is to follow RFC 5375 [IPv6 Unicast Address Assignment Considerations]'s suggestions and issue a minimum of /64 (18E IPv6 addresses) to customers. Since this /64 is completely useless for your own LAN, a second suggestion is to issue a /48 (1,2 million E IPv6 addresses) or /56 (4700E IPv6 addresses) prefix for consumer's own LANs.
Note: 18E is SI-prefix for exa, aka 10^18. A /64 IPv6 network has 18,446,744,073,709,551,616 unique addresses in it.

The numbers are astronomically big and it's quite easy to get confused and lose the perspective what they actually mean. A more concrete approach is, that by being issued a /56, I can now split my prefix into 256 separate /64 networks as I please. It's not like I need 256 LANs, I'm totally happy get even 1 of them to work!

Now I have my hands full to configure and test everything on my LAN. I need to make sure, that native-IPv6 works for wired and wireless toys I have here.

When first information about a severe security flaw in Intel CPUs came out, it was immediately obvious, that this one is huge. This is in the scale of a meteorite hitting The Earth.

Here is just one example of what can happen, when fixed Linux kernels are deployed. I've been pumping data points from my weather station to Weather Underground for couple of years. Now this is how they are able to display my data:

If you don't see any graphs or data points there, that's my point here! WU does not work!

Yesterday I got an e-mail from them explaining the problems they're facing:

The interruption of service was related to the hot patches applied to our servers to correct the recent Intel hardware security flaws. We can say with 100% confidence that the data you share with us is completely safe. The patches required systems to be rebooted and, as these systems came back online, many of them did not boot up cleanly.

That is most definitely not the entire truth. The quality of their service at the time of writing this blog post is simply crap! Looks like Meltdown mitigation hit them harder than they anticipated.

 

I came back from Christmas holidays on Tuesday 2nd January and quite soon that morning our corporate chat channels were buzzing about this "Intel CPU flaw". Lot of the information came from article The mysterious case of the Linux Page Table Isolation patches. At the end, the author of the post suggests "Invest in popcorn, 2018 is going to be fun". For something written on first day of the year 2018, that is pretty well said!

More and more details leaked on Wednesday 3rd. The Register article Kernel-memory-leaking Intel processor design flaw forces Linux, Windows redesign suggested said, that Linux kernel team named the flaw as Forcefully Unmap Complete Kernel With Interrupt Trampolines, aka FUCKWIT. So, they must have been mighty annoyed by this one. Urban Dictionary has a rather hilarious explanation of it.

Given the time difference between US an Europe, the information was published late on 3rd January, in form of website Meltdown and Spectre - Vulnerabilities in modern computers leak passwords and sensitive data. On Thursday 4th European time, entire Internet was full of news and speculation of these security flaws. Nobody has really confirmed this, but it looks like Intel knew about this flaw back in June 2017, for example Ubuntu-project confirms, that they received information about the flaw back in November 9th.

So, what is fuss this all about?

Since nobody really understands Kernel page-table isolation or Speculative execution, a much simper approach is needed to get a grasp of this problem (yes, you Super-Nerds understand it, I know!).
Xkcd #1938 has the best summary of this flaw that I've seen:

Mr. Daniel Miessler also has pretty good summary of the problem in his post A Simple Explanation of the Differences Between Meltdown and Spectre:

Ultimately you have two choices:

1. Have a computer, which has a serious exploitable security flaw
2. Lose some of your CPU-power

Most cases door #1 is chosen for you by your operating system vendor or cloud service provider.

A real measured effect of the fix

This is what Epic Games measured for their game Fortnite:

While looking at the graph, please remember, that Epic Games didn't choose to have the fix installed. This decision was made for them thanks to cloud computing.

There are number of similar reports all around the web, including Tim Gostony's tweet "Impact of the patch for the Intel bug on my AWS EC2 instances running Linux". It shows much les bump in the CPU-power than Epic Games' graph, but still it is there and it can be noticed.

What now?

Ok, everybody's every CPU is affected, but luckily for Meltdown mitigation you wouldn't normally run malware on your computer anyway. For Spectre, it gets scary. It mostly affects shared computers where somebody else is capable of stealing data from your instance.

The #1 thing I'll be waiting most is a physical CPU product that can be purhcased from a store which has a fix for this. Nobody has even promised anything about that. The existing software-based fixes are horrible. For example I was running Progress Quest on a Windows 93 session on a i7 laptop. After the fix, simply starting Windows 93 on a web browser it takes almost all of the CPU instead of idling like it did before the fix.

Amazon Web Services (AWS) in their original statement they said that "This is a vulnerability that has existed for more than 20 years in modern processor architectures like Intel, AMD, and ARM across servers, desktops, and mobile devices." I'm hoping, that it will take less than a decade to get it fixed.

As usual, I got an e-mail from my NAS-box stating, that it has a new firmware upgrade QTS 4.3.3.0404 build 20171213.

It had the usual release notes telling the changes. What really caught my attention was:

QTS 4.3.3 is the final available firmware update for the following models: TS-419U II

WHAAT! Just out of the blue, my model was obsoleted piece of junk!

The actual full list of models is: TS-112P, TS-212P, TS-212-E, HS-210, TS-112, TS-212, TS-121, TS-221, TS-421 TS-120, TS-220, TS-420, TS-420U, TS-421U TS-412, TS-412U, TS-419U, TS-419U+, TS-419U II, TS-119P II, TS-219P II, TS-419P II, TS-119P+, TS-219P+, TS-419P+, TS-119P, TS-219P, TS-419P, TS-119, TS-219, TS-419

So, only the recent QNAP boxes were maintained from this point on. Darn! My take on their decision to stop maintaining all the old models is, that initially they barely maintained them at all. In fact, QNAP got burned seriously on not acting: 0-day: QNAP NAS Devices suffer of heap overflow. In less than two years they managed to get that one fixed. They received information on 1st Feb 2016, stalled on the fix and after 12 months somebody else stumbled into the same flaw and after QNAP failed to receive the information about it, he released into public. QNAP managed to the fix out at 14th Feb 2017.

To me that action (read: lack of it) means, that they did not have a protocol in place for a situation where a security flaw would be found in one of (read: all of) their main products sold to general public. While spewing out unfounded allegations here, I'm pretty confident, that it wouldn't have made any difference if the security flaw was in their internal systems. Also, I'm sure, they did not act on the initial report as the author was well-behaving and extended his grace period on QNAP's request. Unfortunately to QNAP, their security reporting system wasn't maintained and it didn't work at the time of second finding, so the information leaked quite soon.

After all this commotion, they chose to create processes, assign personnel to it and start maintaining their products, they suddenly realized, that IT'S HARD WORK! Oh really! Rest of the world knows it already. But that's what you need to do when you are in device manufacturing business. World is full of non-maintained IoT-junk, as this Twitter-feed points out.

Ok, enough rant. Now I have a decision to make. What to do with a perfectly good NAS-box. Suggestions are welcome.