Hacker's ramblings

This was pretty funny one. I was about to start a Windows Azure 30-day trial on Windows 8 with Internet Explorer 10, but it failed on payment options.

I waited for 10 minutes, but no avail. It was pretty obvious that a failure was imminent after 30 seconds of nothingness. The payment just hangs forever without doing anything. They simply never tested it on IE10. On any other browser I tried it works just ok.

Most Linux distros have udev. It has been around a while and is pretty much the way of handling physical devices in your box.

In The Old Age™ making a device to be something was very simple. /dev was in regular filesystem and could have permissions/symlinks/whatever set by admins. During modern era creating a symlink or setting permissions is bit more complex. The steps are:

1. Identify the device
2. Figure out the identifying attributes from udev
3. Choose an operation / operations to be executed when the device is found
• This can be during boot or plug'n'play / USB
  
4. Bring it all together in a configuration file readable by udev
   

An example:

External USB-drive/-stick can have pretty much any drive letter assigned into it by SCSI-subsystem during plugin. It can be /dev/sde today and /dev/sdf tomorrow. Trying to figure out the drive letter each time it is plugged in is both tedious and unnecessary. With (simple?) udev-wrangling you can have a /dev/myownusb to access it every time the drive is plugged in. Steps:

1. Identify
1. lsusb is your friend, from the output it is possible to determine that:
   Bus 001 Device 007: ID 1941:8021 My C00l USB-drive
   
2. Today USB-bus 001 device 007 is the drive. What if you plug it into a different USB-port next time? We need to find identifying attribute/attributes to make configuring possible.
1. If we assume that the drive is /dev/sdf this time, all the udev-attributes can be displayed with a:
   udevadm info --query=all --name=/dev/sdf --attribute-walk
   
2. It will reveal a drive serial number in a format similar to:
   ATTRS{serial}=="0000002CE09310500C1B"
3. The operation we'd like to be done when such a USB-device with a matching serial number is plugged into the computer is a symlink.
4. The final step to get this configured would be to create a file into /etc/udev/rules.d/ with a suitable name.
1. I chose my configuration to be /etc/udev/rules.d/99-mylocalrules.
2. The file will contain a single line with identifying information and the operation. Example:
   SUBSYSTEMS=="usb", ATTRS{serial}=="0000002CE09310500C1B", KERNEL=="sd?1", SYMLINK+="myownusb"
   
3. That literally reads: Whenever a new device is introduced into USB-subsystem with suitable serial number and having a partition, the 1st partition will be symlinked into udev with name "myownusb"
   

To get the rule into effect you need to run:

udevadm trigger 

It is not necessary to unplug an already working drive. Just confirm that it worked:

ls -l /dev/myownusb

... or similar. Then just mount:

mount /dev/myownusb /mnt/myownusb

Another example:

I have a weather station connected into my Linux via USB-cable. There is no point of accessing it as a root, but out-of-the-box that's the only way to go. I need to chgrp the device after every boot for regular users to gain access into it.

With above process my identifying factor is the USB ID of the device and operation is to chgrp the device with a suitable group to allow access for those users belonging into the group. The rule is:

SUBSYSTEMS=="usb", ATTR{idVendor}=="1941", ATTR{idProduct}=="8021", GROUP="110"

Yet again the udev-rule reads: Whenever a new device is introduced into USB-subsystem with vendor ID of 0x1941 and product ID of 0x8021 the newly created udev-device will have a group with id 110. I prepared a group with groupadd and confirmed that it exists:

# getent group 110
WH-1080usb:*:110:itsme

After a udevadm trigger the result can be confirmed:

# ls -l /dev/bus/usb/001/007
crw-rw-r--. 1 root WH-1080usb 189, 6 Jun 19 10:07 /dev/bus/usb/001/007

The long(ish) path into the device comes from the lsusb output, it reads:

Bus 001 Device 007: ID 1941:8021 Dream Link WH1080 Weather Station

... and can be also translated as /dev/bus/usb/001/007. Simple, huh?

Fedora Linux guys replaced the ancient Initd with Systemd in Fedora 16. Bold move. I understand it had to be done. It speeds up booting and does a bunch of other things Initd can't or won't.

The classic story when introducing something new is that it has bugs. This particular time I struggled to get my Magic Sysrq key working on boot. Looks like Fedora people failed (at least) two times with it: Bug 760254 in Fedora 16 and Bug 924433 in Fedora 18 describe these shortcomings.

In short, the trouble with this new thing is that your changes won't take effect on boot. Most Linux admins never touch any of the sysctl(8)-settings and continue living successfully. Then there are rest of us, who tinker&tune their boxes to match their requirements. In Fedora Linux there is a directory of /etc/sysctl.d/ into a sysadmin may create a file with own settings to either override existing settings from /usr/lib/sysctl.d/ or set completely new values, which have only their kernel default set.

An example:
To set the Sysrq-key into "dangerous"-mode allowing all possible operations, the value of file /proc/sys/kernel/sysrq needs to be "1". It can be achieved with a file in /etc/sysctl.d/ containing following:

kernel.sysrq = 1

In Fedora the default value according to /usr/lib/sysctl.d/50-default.conf and manual inspection after boot is "16". So, the big trouble is to get the value of "1" stick. After a couple of reboots I realized that it is possible to test the functionality without booting the computer. As a root, simply run:

systemctl restart systemd-sysctl.service

... and watch what happens. The rather complex name of the service is something I couldn't figure out without Fedora discussion forums.

Anyway, after many many failures I concluded that my own settings need to be executed before the file 50-default.conf. To make things easier, systemd-sysctl.service first gathers a full list of files to be processed, then alphabetizes them and finally executes the settings in order. So I made my file to be /etc/sysctl.d/01-myownsettings.conf, which seemed to do the trick! There is a logic behind that, but it is just tricky to figure out.

My Fedora 19 got an update for KVM. It was a no biggie, nothing really happened at the time.

Then one of the virtual guests got a new kernel (RHSA-2013:0911-1). I rebooted the guest and BANG! My KVM hung the entires machine. I have the Magic SysRq enabled, but nothing. The box was completely hung.

The "funny" part happened after I forced a reboot from the button. The box wouldn't boot! My LDAP was corrupted. All I got was a "BDB0060 PANIC: fatal region error detected; run recovery" -message.

There is the /usr/bin/db_recover -tool, but it just said FUBAR. I didn't get the actual phrase, but surely you'll get the meaning. Then, what next? I was lucky enough to have a 3 week old slapcat of my entire LDAP. That was plenty of luck for me. But the morale of the story is, that I'll need to start dumping the LDAP or change the back-end format into something more recoverable.

Linux-distros are having a race for the most popular one. Part of the setup is to have the latest Linux kernel, and any other part of the software library a distro has. This effect yields into a "nice" effect where running the bleeding edge distros (Fedora, ArchLinux, Debian sid, etc.) every now and then something breaks.

The latest race is with PHP programming language which is nearing the 5.5.0 release. It has an RC3 version already out. Now the problem is that distro-guys start using the latest stuff, but PHP has incompatible changes in it. Plenty of things made with PHP 5.4 or 5.3 or ... won't run.

One of them is phpLDAPadmin. For some incomprehensible reason they are using stuff, which has been flagged as obsoleted years ago. So, it won't work. Luckily somebody at Debian made a fix. That makes my system's LDAP-admin working again. Hopefully somebody does the same with with all of the PEAR packages.

This is what you get when running alpha version of Fedora 19.

Windows 7 and 8 IPv6-stack has a nice feature. Every now and then when waking up from sleep/hibernate there is no autoconfigured IPv6-address. It looks like this when checking for IPv6-address with netsh interface ipv6 show address:

Interface 12: WiFi

Addr Type  DAD State   Valid Life Pref. Life Address
---------  ----------- ---------- ---------- ------------------------
Other      Preferred     infinite   infinite fe80::227:10ff:1234:5678%12

Note how there is only link-local address and no "proper" IPv6-address. On Windows 7 there is a fix. Just disable the Teredo-interface:

netsh interface teredo set state disabled

After that IPv6-autoconfiguration works much better. That doesn't do the trick for Windows 8, tough. I have no idea what would help. My only fix is to discard autoconfiguration and manually set an IPv6-address. This setting won't survive a reboot, so it will keep using autoconfig under normal conditions. But anyway, this is what I do to fix my laptop:

netsh interface ipv6 add address "WiFi" 2001:1234:5678:0:227:10ff:1234:5678

After this, netsh interface ipv6 show address will function properly:

Interface 12: WiFi

Addr Type  DAD State   Valid Life Pref. Life Address
---------  ----------- ---------- ---------- ------------------------
Manual     Preferred     infinite   infinite 2001:1234:5678:0:227:10ff:1234:5678
Other      Preferred     infinite   infinite fe80::227:10ff:1234:5678%12

If anybody knows how to fix this, please drop me a comment.

I got a new Supermicro server. Pretty much the next day there was a writing about computer security. The article is in Finnish, and it contains interviews from

• Stonesoft, the Finnish firewall / intrusion detection company bought by McAfee / Intel
  
• F-Secure, the Finnish virus detection company
  
• CERT-FI, the Finnish national computer security incident response team
  
• SSH, the Finnish security company
  

It contains allegations from USA about Chinese Huawei using their hardware for spying American companies. There is also allegation that Supermicro had intentionally left specific IP-addresses open in their Base Management Controller. I didn't find a single trace about that, however, there is a security warning by SEC Consult about Barracuda-products having something similar what they describe in the article.

Taiwanese Supermicro is not totally innocent. I found that they had/have a flaw in their documentation. They failed to mention that their IPMI implementation has two admin-accounts. That is pretty rare when it comes to networked appliance. Typically one admin-account will do fine for most of us. Intentional or a honest mistake? Nobody knows. And those who do, won't tell.

Otherwise, I found two separate issues about remote management security issues. First one is about using a non-standard chiper to encrypt the IPMI-traffic. It simply lets you pass without encryption or authentication. Nice!

The second one is from almighty Bruce Schneier. It describes findings about general lack of security in those very critical systems used to manage the servers. Consumer products are safe, they don't have BMC / IPMI -chips in them.

I have one Asus ASMB4 and one Hewlett Packard iLO 2 on-line at the time of writing. The Asus BMC doesn't survive two weeks in the wild Internet. There is an unknown flaw to shoot it out of the Net somehow. That's why I put a firewall appliance in front of it. The HP iLO 2 seems to survive in the wild, however it has a very sluggish response time and absolutely requires a firewall also. It seems to be a target for lots of incoming traffic. Those BMC-boards are tiny both in physical size and computing capabilities, it doesn't take much to overload them.

This is a cool move! At least Arch Linux and Fedora Linux have done this. A typical root directory listing will look like this:

lrwxrwxrwx.   1 root root     7 May 30 23:17 bin -> usr/bin
lrwxrwxrwx.   1 root root     7 May 30 23:17 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 May 30 23:17 lib64 -> usr/lib64
lrwxrwxrwx.   1 root root     8 May 30 23:17 sbin -> usr/sbin

The /bin, /lib and /sbin have been in the root directory since first Unix was released in 1969. Any modern Linux distro won't have and won't need that many partitions than the legacy *nixes used to have. The reasoning is in this discussion thread.

Arch Linux guys have a pretty complex update process. The classic pacman -Syu won't do it for you this time.

On my previous post about Samba 4, I stumbled on printers.

I had a weird warning in my log:

../source3/param/loadparm.c:3121(lp_do_parameter)
  Ignoring unknown parameter "printer admin"
        printer admin = root

The fix to do is drop:

printer admin = root

from smb.conf. Instead it needs to be done run-time, like this:

smbpasswd -a root
net rpc rights grant root SePrintOperatorPrivilege

First a password is required for root-user. I have one in LDAP, but for some reason a local password is required too. After that permissions for printer administration are granted separately. That pretty much concentrates privilege handling out of any text-files.

The other issue was printer sharing to Windows. It had an easy fix. I deleted the existing printer from Windows and added it again. CUPS had renamed the printer and it was not available with the same name. A basic CUPS / Samba setup with cupsaddsmb does the trick.

When doing XML-editing, I always use a suitable editor for that. Recently I've been using oXygen XML editor. It has all the features I need, I like it and naturally I bought a license. On a minus side, it is Java-software, and lately I've been disliking Java very much.

A while ago, they released a new version of 15.0. They appropriately informed me about the new version and said to check the upgrade availability. They have a nice reminder -form to check what you purchased from them the last time.

There is one thing they fail to mention. If you purchase today, and don't want to pay extra $100 for software upgrade service, and they release a new version tomorrow, you won't be eligible for a free upgrade. That's how they perceive you, a paying customer, a stupid lamb not to have paid them for a service they don't tell any details about.

There is a huge number of software companies operating on different basis. First you purchase their software. At that point they give you (typically) 12 month upgrade-period free-of-charge. Then at that point, they ask if you'd like their product that much to start paying for a service. You can agree or decline. If you agree, you'll be hoping that they release often enough to get your money's worth. On the other hand, you can choose to purchase updates whenever you feel like doing it. The software company respects you and operates on a honest basis.

I'll be taking my business elsewhere. Any recommendations for a XML-editor?

My Fedora 19 project continues... An attempt to get Samba working. They upgraded into version 4 and obviously my version 3 smb.conf had issues. See my article about getting Samba to use LDAP as userbase backend.

The obvious problem was, that it didn't work.

A log entry from the failue:

../source3/lib/smbldap.c:575(smbldap_start_tls)
  Failed to issue the StartTLS instruction: Connect error
../source3/passdb/pdb_ldap.c:6531(pdb_ldapsam_init_common)
  pdb_init_ldapsam: WARNING: Could not get domain info, nor add one to the domain. We cannot work reliably without it.
../source3/passdb/pdb_interface.c:177(make_pdb_method_name)
  pdb backend ldapsam:ldap://my.server did not correctly init (error was NT_STATUS_CANT_ACCESS_DOMAIN_INFO)

I confirmed the existing settings:

passdb backend = ldapsam:ldap://my.server
ldap ssl = start tls

After a nice while of reading manual pages, an attempt to fix:

passdb backend = ldapsam:ldaps://my.server
ldap ssl = off

Yielded an improvement:

../source3/lib/smbldap.c:998(smbldap_connect_system)
  failed to bind to server ldaps://my.server with dn="uid=root,ou=People,dc=my,dc=domain" Error: Can't contact LDAP server
        TLS error -8179:Peer's Certificate issuer is not recognized.
../source3/passdb/pdb_ldap.c:6531(pdb_ldapsam_init_common)
  pdb_init_ldapsam: WARNING: Could not get domain info, nor add one to the domain. We cannot work reliably without it.
../source3/passdb/pdb_interface.c:177(make_pdb_method_name)
  pdb backend ldapsam:ldaps://my.server did not correctly init (error was NT_STATUS_CANT_ACCESS_DOMAIN_INFO)

This, however, was an easy fix. It was a simple SElinux issue:

semanage fcontext -a -t cert_t /etc/openldap/cacerts
restorecon -R -v /etc/openldap/cacerts

To my amazement SElinux context does not change on a local unix-socket request. When Samba makes the request to get user information, the LDAPd certificate store needs to have proper SElinux type for the directory. OpenLDAP does not make such checks and works fully.

Also allowing requests to home directories too:

setsebool -P samba_enable_home_dirs 1

After all this, I was happy to get my Samba-shares working again. CUPS-printing does not. But I'll fix that on some day.

Fedora 19 ships with Apache 2.4. After the install completed it very soon become obvious that my previous Apache 2.2 setup didn't work without changes. At the time I just took a bigger hammer and kept banging it until it started. Needless to say, it didn't have all my virtual hosts and services.

Now I spent a while getting all the necessary things working. The most common issue I had was:

authz_core:error AH01630: client denied by server configuration

This pretty weird error is caused by the fact that as default all access is denied and the classic:

Order allow,deny
Allow from all

... does not actually do anything on 2.4, instead it needs to be written as:

Require all granted

The change in the configuration actually makes the setup much clearer, but the obvious problem is that it is not compatible with the previous versions. In the conf.modules.d/00-base.conf there is a directive to load the compat-module:

LoadModule access_compat_module modules/mod_access_compat.so

I put the line into comment and started running my Apache with new style setup only.

My second biggest issue was with the services I'm running with allow access from my own LAN, but require LDAP authentication when traffic was not originating from my LAN. Apache 2.2 example would be:

Order Deny,Allow
Deny from all

Allow from my.lan
Allow from 2001:1234:5678::/64

AuthType Basic
AuthName www.my.lan
AuthBasicProvider ldap
AuthLDAPURL "ldap://server:389/ou=People,dc=example,dc=com?uid?sub?(objectClass=*)"
Require valid-user

Satisfy Any

The solution is very simple, just list the requirements and 2.4 somehow magically knows what you mean:

Require host my.lan
Require ip 2001:1234:5678::/64

AuthType Basic
AuthName www.my.lan
AuthBasicProvider ldap
AuthLDAPURL "ldap://server:389/ou=People,dc=example,dc=com?uid?sub?(objectClass=*)"

Require valid-user

Otherwise my migration was pretty smooth.

I needed to transfer really big files (10+ GiB) between couple of servers. Since I was not in a hurry I decided to go with FTP. It performs very well when doing large file transfers, see File Transfer Protocol Performance Study for EUMETSAT Meteorological Data Distribution by students of Institute of Mathematics and Computer Science University of Latvia. Only Rsync could do better, and that is when the target is to transfer large number of files instead of small number of large files. In reality all the time it took me to set up the servers, I'd done transferring my files with any protocol. Really, any. But since I'm expecting to need this setup sometimes later, I went ahead with it.

Anyway, I chose to go with FTPd and since my Fedora 19 beta has one in built in repo, I went with Proftpd. Initial problems:

• It didn't work well (read: not even the simplest things you'd expect work) with IPv6, it took me 3 hours to figure this out.
  Fix: I used IPv4 addresses instead of FQDNs. Everything started to work.
  
• No anonymous FTP-access
• No anonymous FTP uploads
  

IPv6-issues

There is plenty of code like this:

pr_log_debug(DEBUG0,
  "Unable to handle PASV for IPv6 address '%s', rejecting command",
  pr_netaddr_get_ipstr(session.c->local_addr));
pr_response_add_err(R_501, "%s: %s", cmd->argv[0], strerror(xerrno));

So after I found about that, I just stopped using it.

Anonymous FTP-access

FTP isn't a secure protocol. It was once by 80s standards, but in 2013 Internet... well, no. So the best bet is not to use security at all! I'll place my security on the firewall, use anonymous logins and reduce attack surface by shutting the daemon down when I'm not expecting to need it.

Getting the anonymous FTP to work turned out to be semi-tricky. The daemon needs a -DANONYMOUS_FTP in the sysconfig. Also I needed to re-direct the anonymous FTP root directory into a dedicated partition instead of the out-of-the-box /var/ftp. My enforcing SElinux didn't like that. I had appropriate owner and group setup for the directories and files, but it turned out that my mount-directory had weird SElinux type and I decided to go with pretty generic var_t as directory type, then anonymous user was able to log in. The next thing was to make sure that actual FTP-stuff had SElinux type of public_content_t for access to directories and files to work. The final thing was to convince my SElinux to allow using of appropriate TCP-ports for FTPd:

setsebool -P ftpd_connect_all_unreserved=1 

After that I had somewhat working anonymous FTP box.

Passive FTP from Windows

I didn't have any fancy software for testing the client side. All I had was a trustworthy ftp.exe on a Windows PowerShell. However, it operated only on active-FTP -mode, which in my opinion is completely obsoleted. It simply does not work at all with proper firewall setup, so I'd better using passive-FTP -mode. Well, then... how to lure the ancient FTP-client to use it? It wasn't obvious or easy, but luckily somebody else had the same problem and solved it. The solution is to use QUOTE-command for passing the FTP-protocol required PASV, like this:

Connected to 192.168.0.1.
220 FTP Server ready.
User (192.168.0.1:(none)): ftp
331 Anonymous login ok, send your complete email address as your password
Password:
230-
                *** Welcome to this anonymous ftp server! ***

     You are user 1 out of a maximum of 10 authorized anonymous logins.
     The current time here is Wed Jun 05 15:25:31 2013.
     If you experience any problems here, contact : root@localhost


230 Anonymous login ok, restrictions apply.
ftp> quote pasv
227 Entering Passive Mode (192,168,0,1,236,242).

That made passive mode work ok.

Allowing anonymous uploads

The last bit was trickiest, it was almost impossible to give Proftpd what it wanted. To my surprise it wasn't about directory permissions or SElinux. It was simply about configuration issue with <Limit> acting stupidly. Out-of-the-box, the config file apparently allows anonymous uploads. The problem is that it doesn't work. There is a line like this:

<Directory uploads/*>

But it should be like this:

<Directory /uploads>

I don't know why it must be like that, but after tinkering it for a very long time, that turned to be the key to my success. I also changed a <Limit READ> into <Limit LIST READ> to prevent getting directory listings from uploads-directory.

That concluded my setup. Just opening suitable IP-addressses and TCP-ports 20 and 21 made my files fly.

Steam has this reputation of games re-installing 3rd party prerequisites on every single time you launch a game. Stupid, huh? Find discussion about that on Steam's forums.

The official Steam explanation is that there are so many versions of D3DX-library. Well, in my mind that really doesn't answer the question why there is a re-install on every game launch. You'd think that the library needs to be installed only once. Luckily somebody found out a fix for that. The trick is to edit/delete the .VDF-file which is pretty much the install script for a game.

Here is an example 34870_install.vdf:

"InstallScript"
{
    "Run Process"
    {
        "DirectX"
        {
            "process 1"        "%INSTALLDIR%\\Redist\\dxwebsetup.exe"
            "Description"      "Microsoft DirectX Redist Package"
            "NoCleanUp"        "1"
        }
    }
    "Run Process"
    {
        "VCRedist"
        {
            "process 1"        "%INSTALLDIR%\\Redist\\vcredist_x86_2010_sp1.exe"
            "Description"      "Microsoft VC Redist Package"
            "NoCleanUp"        "1"
        }
    }
}

It pretty much states that both DirectX and VisualC redistributable packages are installed every single time the game launches. A simple rename/delete for the file will fix the problem. Game will start without problems and won't do anything stupid.

The issue has been reported by multiple people for the past 3 years. Shame on you Steam!

For some unknown reason a customer's backup job hung. A ps axuww looked like this after a failed kill 3150 attempt:

3150 ?        Z      0:00 [mysqldump] <defunct>

There was no other way of getting rid of that other than a reboot. It didn't help, though. It automatically re-started the backup job and did the same thing again.

All customer's services were disabled, there was Suspend domain until backup task is completed -setting enabled. At that point I was really screwed. The backup job turned undead and all customer's services were out of action. Nice going! The message that was given from Parallels Panel at control panel was: "The subscription is temporarily suspended until the backup or restore process is complete".

Parallels forums had an article about unsuspending a domain. I did the same thing for a subscription. Apparently at the time of writing in Plesk 9, there were no subscriptions. So I ran this:

/usr/local/psa/bin/subscription -u customer's domain name here.com  -status enabled

It helped! It un-suspended the services. The backup is not working yet, but the services were running again. I need to further investigate the backup issue and get back about that.