Linux - 4

I've had my run of bad things with Intel soft-RAID earlier. The constant RAID-verify -runs made me want stop using it. As its my Windows-box, I just wanted something that is hardware-based, reasonably fast and affordable. My choice is HighPoint RocketRAID 620.

For a switch-over -project I had a simple plan:

1. Clonezilla the existing RAID into an another drive which I could plug into a motherboard for the duration of the move
2. Un-configure the Intel soft-RAID at the motherboard
3. Plug in the RocketRAID-card
4. Change the hard-drive cables from motherboard RAID-connectors into RocketRAID-card
5. Configure a new RAID-1 mirror wit RocketRAID
6. Clonezilla the data back to the newly created RAID-1 volume
7. Be happy and continue computing

Guess what. Things fell trough at point #6. I was using an USB-bootable Clonezilla live on my first data move and obviously was planning to use it for the second one too. Whichever Linux-distro they use as the base for Clonezilla, they don't have the driver for the RAID-card. Darn!

The next best thing is a commercial distro for Clonezilla, Parted Magic. They used to be free (as in beer and speech), but they went commercial. The price is $9 USD for a single download, so I got it. And guess what again! They don't support Highpoint RocketRAID either.

I did ask about it in their support forums (closed to registered users only, sorry). And they replied:

We do not do "random" out-of-tree drivers because commonly these are supported by their vendors in a haphazard way. E.g. in the HighPoint case the latest driver is 3 versions behind our kernel version.

Luckily the vendor is providing the partial source code for the driver. There is a binary-part of in it and it is kind-of open-source. The biggest problem seems to be, that it doesn't build on any reasonably modern Linux.

By googling, I found that somebody else had the same process of thought and there was a Github project for the upgraded driver. Unfortunately that too was 3 years old and wouldn't build. Also it was for the vendor driver 1.1, and they already had 1.2 out.

In this imperfect world everything that you need to be done properly, you need to do by yourself. So, here it is: https://github.com/HQJaTu/rr62x
You can help yourselves with that one.

This is how it looks on my dmesg:

[ 85.518732] rr62x: module license 'Proprietary' taints kernel.
[ 85.518737] Disabling lock debugging due to kernel taint
[ 85.519709] rr62x:RocketRAID 62x SATA controller driver v1.2 (Jul 1 2012)
[ 85.735773] rr62x:adapter at PCI 3:0:0, IRQ 16
[ 85.950487] rr62x:[0 0 ] start port.
[ 85.950488] rr62x:[0 0 ] start port hard reset (probe 1).
[ 86.150712] rr62x:[0 1 ] start port.
[ 86.150712] rr62x:[0 1 ] start port hard reset (probe 1).
[ 89.093649] rr62x:[0 0 ] start port soft reset (probe 1).
[ 89.841048] rr62x:[0 1 ] start port soft reset (probe 1).
[ 90.501075] rr62x:[0 0 ] port started successfully.
[ 90.501078] rr62x:[0 0 0] device probed successfully.
[ 90.791364] rr62x:[0 1 ] port started successfully.
[ 90.791369] rr62x:[0 1 0] device probed successfully.
[ 90.806570] scsi host13: rr62x
[ 90.806870] scsi 13:0:0:0: Direct-Access HPT DISK_13_0 4.00 PQ: 0 ANSI: 5
[ 90.809711] sd 13:0:0:0: [sdd] 2930114560 512-byte logical blocks: (1.50 TB/1.36 TiB) [ 90.809847] sd 13:0:0:0: [sdd] Write Protect is off
[ 90.809852] sd 13:0:0:0: [sdd] Mode Sense: 2f 00 00 00
[ 90.809909] sd 13:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 90.832339] sdd: unknown partition table
[ 90.832903] sd 13:0:0:0: [sdd] Attached SCSI disk

I've tested that with Linux 4.0.4 and 3.19.3. It builds and works on both. Any comments, Github forks, pull requests, etc. are welcome. I will get back to the actual disk cloning project later, the driver won't  help unless it is used properly in an operating system.

Something funny happened to my Fedora 20 installation. Stuff that used to work stopped doing so. I understand, that I'm dealing with "bleeding edge" Linux-distro, but it kind of starts to annoy me, when things don't get back like they were.

Here is one:

# systemctl status iscsi.service --all
iscsi.service - Login and scanning of iSCSI devices
   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled)
   Active: inactive (dead)
           start condition failed at Thu 2014-07-17 12:23:18 EEST; 1min 24s ago
           none of the trigger conditions were met
     Docs: man:iscsid(8)
           man:iscsiadm(8)

Jul 17 12:23:17 box systemd[1]: Started Login and scanning of iSCSI devices.
Jul 17 12:23:18 box systemd[1]: Started Login and scanning of iSCSI devices.

Aow come on! "none of the trigger conditions were met"!! I don't know what changed with the precious trigger conditions. Basic checks didn't reveal anything noteworthy:

# systemctl is-active iscsi.service
active
# systemctl list-dependencies iscsi.service
iscsi.service

There are some other people suffering from the same issue. Fedora 20 ypbind won’t start at boot. NIS logins unavailable immediately after boot (ypbind service NOT started). and Bug 1027114 - nfs mount fails during boot because network is not ready point me to the direction of service trying to start too early. There is a document explaining how to run services only after network is up (Running Services After the Network is up), but the point is: somebody goofed! This used to work.

I tried a classic off/on -trick:

# systemctl disable iscsi.service
# systemctl enable iscsi.service

... no avail. Still after a boot I have a list of failing services:

# systemctl --failed
UNIT             LOAD   ACTIVE SUB    DESCRIPTION
arpwatch.service loaded failed failed Arpwatch daemon which keeps track of ether
dhcpd.service    loaded failed failed DHCPv4 Server Daemon
iscsi.service    loaded failed failed Login and scanning of iSCSI devices

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

3 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Finally I chose to add the After=network-online.target -line to my service description. First the service description needs to be copied from /usr/lib/systemd/system/ into /etc/systemd/system/. The rule is very simple: don't edit files under /usr/ and it applies also here. Systemd has a built in fall-back mechanism. If your own tailor made file doesn't exist in /etc/, it will take the default one. It is good to have that, but better if no such tinkering wasn't needed at all. What if there is an update to the service-description? I have to manually update my copied file.

So, I did a:

# cd /etc/systemd/system/
# cp /usr/lib/systemd/system/iscsi.service .

and edited the file. I added one line to Unit-description:

[Unit]
After=network-online.target

To be absolutely clear: I didn't touch the other lines.

Since systemd doesn't pick up the changes very easily, I did a:

# systemctl disable iscsi.service
# systemctl enable iscsi.service

after status informed, that my change was in place:

# systemctl status iscsi
iscsi.service - Login and scanning of iSCSI devices
   Loaded: loaded (/etc/systemd/system/iscsi.service; enabled)

Notice how, the service description is now under /etc/.

Anyway, that did the trick for all of my failing services. Now they start nicely on bootup.

Something funny happened to my Arch Linux, it changed the IPv6 address rather surprisingly. When I failed to SSH into it, I went to the console.

On none of my machines I have the RFC 4941 privacy extensions enabled. The first thing to do was to confirm, that the IP-address was not based on my Ethernet Address:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:cf:1c:36 brd ff:ff:ff:ff:ff:ff
    inet6 2001::e2:349d:bcc6:c9fc/64 scope global noprefixroute dynamic
       valid_lft 84267sec preferred_lft 12267sec

There is absolutely no reference to the MAC-address 52:54:00:cf:1c:36 of in the EUI-64 address 00e2:349d:bcc6:c9fc. Given the calculator result of 5054:00ff:fecf:1c36. Apparently something modifies the autoconfiguration address.

First culprit could be Linux kernel:

# cat /proc/sys/net/ipv6/conf/all/use_tempaddr
0

I even iterated all the configurations:

find /proc/sys/net/ipv6/conf/ -name use_tempaddr -print -exec cat {} \;

just to make sure, that privacy is off.

That was puzzling. What the heck is going on in the machine!! After all, it used to work ok before. What could be doing that?

Lot of googling around landed me on Arch Linux ArchWiki IPv6 page. It says: "dhcpcd includes in its default configuration file since version 6.4.0 the option slaac private, which enables "Stable Private IPv6 Addresses instead of hardware based ones", implementing RFC 7217". Hum? I've never heard of RFC 7217, nor "A Method for Generating Semantically Opaque Interface Identifiers with IPv6 Stateless Address Autoconfiguration (SLAAC)", but there it was.

I confirmed /etc/dhcpcd.conf and yes, it did contain a:

slaac private

in it. Why?! I didn't ask for it. I simply put a comment (#) in front of the line and did a:

systemctl restart dhcpcd@eth0.service

To refresh the situation. No change. A reboot. Yes change!

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:cf:1c:36 brd ff:ff:ff:ff:ff:ff
    inet6 2001::5054:ff:fecf:1c36/64 scope global noprefixroute dynamic
       valid_lft 86399sec preferred_lft 14399sec

Now my IPv6-address was exactly what it used to be. Weird incident, though. There is a perfectly good mechanism for privatizing addressses, if wanted to. But now somebody had created another and some higher power made a choice for me to start using it. Really weird! Well, I'm hoping that my addresses won't change any more.

One of my boxes is running an Arch Linux. Out of the box it is really a slim one. The install runs only a blink and as a result the operating system won't have anything that is not absolutely necessary to boot the thing for the first time. Given any of the other distros who require gigabytes and gigabytes of storage for crap you won't ever need this is a refreshing change. Every Arch Linux user needs to "build their own excitement" (originally about Gentoo from obsoleted http://www.usenix.org.uk/pictures/despair-linux/gentoo.jpg).

Recently they maintainers have been fiddling around too much with network interface naming conventions. When I installed it was eth0, then it changed to ens3, and guess what happened when I last updated! Yuupp! Back to eth0, but with a twist. Now the eth0 won't come up on boot. Crap!

The Arch Linux discussion forum's section Networking, Server, and Protection has a discussion with topic [SOLVED] Update broke netctl (I guess?). It discusses the problem with a sys-subsystem-net-devices-ens3.device. However, in my box none of the repair instructions were accurate.

Later I realized that my dmesg has following lines in it:

systemd[1]: Expecting device sys-subsystem-net-devices-eth0.device...
systemd[1]: Expecting device sys-subsystem-net-devices-ens3.device...

Ok. On bootup it waits for two, as in not one, network interfaces to become alive. The problem is that my box only has one. A check for the ghost-interface:

systemctl status sys-subsystem-net-devices-ens3.device
* sys-subsystem-net-devices-ens3.device
   Loaded: loaded
   Active: inactive (dead)

Yields pretty much what I know. It is inactive and dead. A manual fix would be to start the DHCP-client manually with a:

systemctl start dhcpcd@eth0.service

... after which the network starts functioning again, but which does not fix the problem. On bootup the interface won't work!

What I did to fix this was to disable dhcpcd for both interfaces:

systemctl disable dhcpcd@ens3.service
systemctl disable dhcpcd@eth0.service

And enabled it to the proper one:

systemctl enable dhcpcd@eth0.service

This does seem to help, but on bootup it still complains "Dependency failed for dhcpcd on ens3". I don't know exactly where the old interface keeps popping up.

In the end, this does work, but it simply takes a bit longer to boot than it used to. Any suggestions to improve booting are welcome.

Most people don't much care about their Linux-boxes' security. You install it, you run it, you use it and occasionally run some system updates into it. Not me. When I have a box running against the wild wild Net, I absolutely positively plan to make the life of anybody cracking into one of my boxes as difficult as possible (with some usability left for myself). See Mr. Tan's article about Security-Functionality-Usability Trade-Off.

So, my choice is at the Functionality - Security -axis with less on the Ease-of-use. The rationale is that, a web application needs to run as safely as possible and can have the ease-of-use in it. The system administrator is a trained professional, he doesn't need the easy-part so much. However, there is a point, when things are set up too tight:

Image courtesy of Dilbert by Scott Adams

So, I voluntarily run software designed and implemented by NSA, SElinux. I even run it in the the Enforcing-mode which any even remotely normal system administrator thinks as being totally insane! Any small or even a tiny slip-up from the set security policy will render things completely useless. Mordac steps in and stuff simply does not work anymore.

On my Fedora-box there was a bug in BIND, the name server and an update was released to fix that. After running the update, the DNS was gone. As in, it didn't function, it didn't respond to any requests and the service didn't start. All it said was:

# systemctl status named-chroot.service --full
named-chroot.service - Berkeley Internet Name Domain (DNS)
   Loaded: loaded (/usr/lib/systemd/system/named-chroot.service; enabled)
   Active: failed (Result: timeout)

Any attempt to start the service resulted in a 60 second wait and a failure. dmesg-log had nothing about the issue, nor BIND's own log had nothing about the issue in it. So I started suspecting a SElinux-permission issue. My standard SElinux debugging always starts with a:

cat /var/log/audit/audit.log | audit2allow -m local

... to see if SElinux's audit logger is logging any permission-related audit faults. Indeed it did:

require {
        type named_conf_t;
        type named_t;
        class dir write;
}

#============= named_t ==============
allow named_t named_conf_t:dir write;

That reads:
A process running in named_t security context is trying to access a directory with named_conf_t security context to gain a write access, but is denied while doing so.
It is obvious that the process in question must be the BIND name server. No other process has the named_t security context in it. When starting up, BIND name server was about to write into its own configuration directory, which is a big no no! When you write, you write only to designated directories, nowhere else (remember: running in enforcing-mode is insanity).

That is definitely a reason for a daemon not to start or to timeout while starting. Further investigation showed that also Fedora's SElinux policy had been updated a week ago: selinux-policy-3.12.1-74.19.fc19.

At this point I had all the pieces for the puzzle, it was simply a matter of putting it all together. The recently released SElinux policy has a bug in it, and nobody else was there to fix it for me.

The exact audit-log line is:

type=AVC msg=audit(1395481575.712:15239): avc:
denied  { write } for
pid=4046 comm="named" name="named" dev="tmpfs" ino=14899
scontext=system_u:system_r:named_t:s0
tcontext=system_u:object_r:named_conf_t:s0 tclass=dir

So, my chrooted BIND-damon was trying to write into a tmpfs. There aren't that many of those in a system. I've even touched the tmpfs-subject earlier when I wrote a systemd-configuration into my own daemon. To find the tmpfs-usage, I ran:

# mount | fgrep tmpfs
tmpfs on /var/named/chroot/run/named type tmpfs

BIND's chroot-environment has one. That is very likely the culprit. That can be confirmed:

# ls -Z /var/named/chroot/run/
drwxrwx---. named named system_u:object_r:named_conf_t:s0 named

Yep! That's it. The directory has incorrect security context in it. To compare into system's non-chrooted one:

# ls -Zd /run/
drwxr-xr-x. root root system_u:object_r:var_run_t:s0   /run/

There is a difference between named_conf_t and var_run_t. You can write temporary files into latter, but not to the first one. The fix is very simple (assuming, that you speak fluent SElinux):

semanage fcontext -a -t var_run_t "/var/named/chroot/run(/.*)?"
restorecon -R -v named/

The two commands are:
First, re-declare a better security-context for the directory in question and then start using the new definition. Now my BIND started and was fully operational! Nice.

My investigation ran further. I needed to report this to Fedora-people. I looked into the policy-file of /etc/selinux/targeted/contexts/files/file_contexts and found the faulty line in it:

/var/named/chroot/var/run/named.*       system_u:object_r:named_var_run_t:s0

That line almost works. The directory in question has only two files in it. One of them even has a matching name. The problem, obviously, is that the another one does not:

# ls -l /var/named/chroot/run/named/
total 8
-rw-r--r--. 1 named named   5 Mar 22 12:02 named.pid
-rw-------. 1 named named 102 Mar 22 12:02 session.key

See Bug 1079636 at Red Hat Bugzilla for further developments with this issue.

OpenSuse 13.1 was released November 2013. During Christmas holidays I started a project of upgrading my previous installation.

Since I'm running on a MacBook 1,1 it was obvious that I was looking for trouble. Previously I had rEFIt running just to get a GRUB 2 -prompt. This time I decided to start from a clean slate. Literally. I ran
dd if=/dev/zero of=/dev/sda
for the 10 first MiB of the drive to make sure, that it definitely has no trace of any of my previous settings. Since rEFIt has been abandoned years ago, I went for the replacement project rEFInd. I approached the author Mr. Roderick W. Smith and he was very helpful, but no matter what I did, I could not get rEFInd running on my very old 32-bit Mac. So, I had two options left: to go back to abandonware or see what would happen without a Boot Manager.

I failed on the installer settings-dialog, by trying to out-smart OpenSuse logic. My completed installation didn't boot. On 2nd try I simply went with the flow. As Mr. Smith instructed me, I didn't touch the most critical thing: MBR is not the way to go on a Mac! Here are my settings:

And guess what, it worked! OpenSuse 13.1 installer has enough logic to create a bootable Linux-installation to a completely blank hard drive. Nice!

The installer was pretty smart. Wi-Fi network was configured properly, it worked out-of-the-box. Apple-keys work: screen brightness, volume, etc. work as is. Also the typical trouble-makers sleep (on RAM) / hibernate (to disk), battery info, sound, and what not. There were only two minor issues: iSight does not work without the Apple proprietary firmware and the keyboard Apple-keys don't do anything usable.

To get the iSight camera working, see ift-extract -tool at Apple Built-in iSight Firmware Tools for Linux. It can dig the guts out of Mac OS X iSight-driver and equip your Linux with a functioning camera. The keyboard is a trivial one. Like previously, I just keyfuzz'ed the keys into something useful. See the OpenSuse 12.3 installation blog entry for details.

There is one thing you may want to check. If you enable SSHd, like I always do on all servers. As default /etc/sysconfig/SuSEfirewall2.d/services/sshd defines as TCP/22 to be open. That is the general idea, but apparently there is so much SSHd bombing going on, that I always tar pit my installations. For some strange reason Suse engineers chose not to allow that in a specific service definition file, but it has to be in the classic /etc/sysconfig/SuSEfirewall2 file, section FW_SERVICES_ACCEPT_EXT="0/0,tcp,22,,hitcount=3,blockseconds=60,recentname=ssh"
I urge everyone of you to rename the services/sshd into something else and add the above line. This makes bombing your SSH-port so much more difficult. And it does not affect your own login performance, unless you choose to bomb it yourself.

You may want to check OpenSuse's hardware compatibility list for details about Apple Laptops. The HCL has info about what works and what doesn't.

In general OpenSuse folks did a very good job with this one. There was a real improvement on ease installation. Thanks Roderick W. Smith for his help during my installation and thanks to Novell for giving this great distro for free!

The ancient tar is de-facto packing utility in all *nixes. Originally it was used for tape backups, but since tape backups are pretty much in the past, it is used solely for file transfers. Pretty much everything distributed for a *nix in the net is a single compressed tar-archive. However, there is a hidden side-effect in it. Put a colon-character (:) in the filename and tar starts mis-behaving.

Example:

tar tf 2014-02-04_12\:09-59.tar
tar: Cannot connect to 2014-02-04_12: resolve failed

What resolve! The filename is there! Why there is a need to resolve anything?

Browsing the tar manual at chapter 6.1 Choosing and Naming Archive Files reveals following info: "If the archive file name includes a colon (‘:’), then it is assumed to be a file on another machine" and also "If you need to use a file whose name includes a colon, then the remote tape drive behavior can be inhibited by using the ‘--force-local’ option".

Right. Good to know. The man-page reads:

Device selection and switching:
--force-local
              archive file is local even if it has a colon

Let's try again:

tar --force-local tf 2014-02-04_12\:09-59.tar
tar: You must specify one of the `-Acdtrux' or `--test-label'  options

Hm.. something wrong there. Another version of that would be:

tar -t --force-local f 2014-02-04_12\:09-59.tar

Well, that hung until I hit Ctrl-d. Next try:

tar tf 2014-02-04_12\:09-59.tar --force-local

Whooo! Finally some results.

I know that nobody is going to change tar-command to behave reasonably. But who really would use it over another machine (without a SSH-pipe)? That legacy feature makes things overly complex and confusing. You'll get my +1 for dropping the feature or changing the default.

This puzzled me for a while. It is almost impossible to install the root certificate from own CA into openSUSE Linux and make it stick. Initially I tried the classic /etc/ssl/certs/-directory which works for every OpenSSL-installation. But in this case it looks like some sort of script cleans out all weird certificates from it, so effectively my own changes won't last beyond couple of weeks.

This issue is really poorly documented. Also searching the Net yields no usable results. I found something usable in Nabble from a discussion thread titled "unify ca-certificates installations". There they pretty much confirm the fact that there is a script doing the updating. Luckily they give a hint about the script.

To solve this, the root certificate needs to be in /etc/pki/trust/anchors/. When the certificate files (in PEM-format) are placed there, do the update with update-ca-certificates -command. Example run:

# /usr/sbin/update-ca-certificates
2 added, 0 removed.

The script, however, does not process revocation lists properly. I didn't find anything concrete about them, except manually creating symlinks to /var/lib/ca-certificates/openssl/ -directory.

Example of verification failing:

# openssl verify -crl_check_all test.certificate.cer
test.certificate.cer: CN = test.site.com
error 3 at 0 depth lookup:unable to get certificate CRL

To get this working, we'll need a hash of the revocation list. The hash value is actually same than the certificate hash value, but this is how you'll get it:

openssl crl -noout -hash -in /etc/pki/trust/anchors/revoke.crl

Then create the symlink:

ln -s /etc/pki/trust/anchors/revoke.crl \
  /var/lib/ca-certificates/openssl/-the-hash-.r0

Now test the verify again:

# openssl verify -crl_check_all test.certificate.cer
test.certificate.cer: OK

Yesh! It works!
Funny how openSUSE chose a completely different way of handling this... and then chose not to document it enough.

Microsoft announced version 3.5 of Linux Integration Services for Hyper-V. An ISO-image is available for download at Mirosoft's site.

In one of my earlier articles I was wondering if it really matters when Hyper-V indicates the Linux guest status as degraded and tells that an upgrade is required. This version does not change that. Looks like they just added some (weird) new features and improved set of virtulization features for Windows Server 2012 R2, but didn't touch the network code. However, there is a promise of TRIM-support for 2012 R2.

So, the bottom line is: not worth upgrading.

Speedtest.net has pretty much gained The-place-to-test-your-connection-speed -status. It's like Google for doing web searches. There simply is no real competition available.

Mr. Matt Martz (while throwing hot coals) did study their JavaScript-code enough to write their client-API with Python.

The installation into proper directory (recommended: /usr/local/bin/) with proper permissions is this simple:

wget -O speedtest-cli \
 https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
chmod +x speedtest-cli

The built-in automatic detection of nearest server does not work for me very well. Their recommended nearest server is not in the country I live (Finland), but on Russian side. The network connections over their border aren't that good and it simply does not yield reliable measurements. Not to worry, the CLI-version can do following:

speedtest_cli --list | fgrep Finland
864) Nebula Oy (Helsinki, Finland) [204.35 km]

Now that we know the server ID of a proper point, we can do:

speedtest_cli --server 864

It will yield:

Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Finland Oyj (80.80.80.80)...
Hosted by Nebula Oy (Helsinki) [204.35 km]: 14.782 ms
Testing download speed........................................
Download: 92.69 Mbit/s
Testing upload speed..................................................
Upload: 4.32 Mbit/s

Nice!

Again, thanks Matt for sharing your work with all of us.

PLD RescueCD is my new favorite Linux rescue CD. It has a ton of stuff in it, even the ipmitool from OpenIPMI-project. One of these days, it so happened that I lost my IPMI network access due to own mis-configuration. I just goofed up the conf and oops, there was no way of reaching management interface anymore. If the operating system on the box would have been ok, it might have been possible to do some fixing via that, but I chose not to. Instead I got a copy of PLD and started working.

The issue is, that PLD RescueCD comes as ISO-image only. Well, erhm... nobody really boots CDs or DVDs anymore. To get the thing booting from an USB-stick appeared to be a rather simple task.

Prerequisites

• A working Linux with enough root-access to do some work with USB-stick and ISO-image
• syslinux-utility installed, all distros have this, but not all of them install it automatically. Confirm that you have this or you won't get any results.
  
• GNU Parted -utility installed, all distros have this. If yours doesn't you'll have to adapt with the partitioning weapon of your choice.
  
• An USB-stick with capacity of 256 MiB or more, the rescue CD isn't very big for a Linux distro
• WARNING! During this process you will lose everything on that stick. Forever.
• Not all old USB-sticks can be used to boot all systems. Any reasonable modern ones do. If you are failing, please try again with a new stick.
  
• PLD RescueCD downloaded ISO-file, I had RCDx86_13_03_10.iso
• You'll need to know the exact location (as in directory) for the file
• The system you're about to rescue has a means of booting via USB. Any reasonable modern system does. With old ones that's debatable.
  

Assumptions used here:

• Linux sees the USB-stick as /dev/sde 
• ISO-image is at /tmp/
• Mount location for the USB-stick is /mnt/usb/
• Mount location for the ISO-image is /mnt/iso/
• syslinux-package installs it's extra files into /usr/share/syslinux/
• You will be using the 32-bit version of PLD Rescue

On your system those will most likely be different or you can adjust those according to your own preferences.

Information about how to use syslinux can be found from SYSLINUX HowTos.

Steps to do it

1. Insert the USB-stick into your Linux-machine
   
2. Partition the USB-stick
• NOTE: Feel free to skip this if you already have a FAT32-partition on the stick
• Steps:
1. Start GNU Parted:
   parted /dev/sde
2. Create a MS-DOS partition table to the USB-stick:
   mktable msdos
   
3. Create a new 256 MiB FAT32 partition to the USB-stick:
   mkpart pri fat32 1 256M
   
4. Set the newly created partition as bootable:
   set 1 boot on
5. End partitioning:
   quit
   
3. Format the newly created partition:
   mkfs.vfat -F 32 /dev/sde1
   
4. Copy a syslinux-compatible MBR into the stick:
   dd if=/usr/share/syslinux/mbr.bin of=/dev/sde conv=notrunc bs=440 count=1
   
5. Install syslinux:
   syslinux /dev/sde1
   
6. Mount the USB-stick to be written into:
   mount /dev/sde1 /mnt/usb/
   
7. Mount the ISO-image to be read:
   mount /tmp/RCDx86_13_03_10.iso /mnt/iso/ -o loop,ro
   
8. Copy the ISO-image contents to the USB-stick:
   cp -r /mnt/iso/* /mnt/usb/
   
9. Convert the CD-boot menu to work as USB-boot menu:
   mv /mnt/usb/boot/isolinux /mnt/usb/syslinux
10. Take the 32-bit versions into use:
    cp /mnt/usb/syslinux/isolinux.cfg.x86 /mnt/usb/syslinux/syslinux.cfg
    
11. Umount the USB-stick:
    umount /mnt/usb
    
12. Umount the ISO-image:
    umount /mnt/iso
    
13. Un-plug the USB-stick and test!

Result

Here is what a working boot menu will look like:

Like always, any comments or improvements are welcome. Thanks Arkadiusz for your efforts and for the great product you're willing to share with rest of us. Sharing is caring, after all!

This is something I've been trying to crack ever since I installed Fedora 19 alpha into my router. My HTTP-streams do not work. At all. Depending on the application and its retry-policy implementation some things would work, some won't. Examples:

• Playstation 3 updates: Updates load up to 30% and then nothing, this one I mistakenly thought was due to PS3 firmware update
  
• YLE Areena: No functionality after first 10-40 seconds
  
• Netflix: Poor picture quality, HD pretty much never kicks in, super-HD... dream on.
  
• Spotify: Works ok
• Ruutu.fi: Endless loop of commercials, the real program never starts
• Regular FTP-stream: Hang after first bytes
  

My Fedora 19 Linux is a router connecting to Internet and distributing the connection to my home LAN via NAT. The IPtables rule is:

iptables -t nat -A POSTROUTING -o em1 -j MASQUERADE

I found an article with title IPTables: DNAT, SNAT and Masquerading from LinuxQuestions.org. It says:

"SNAT would be better for you than MASQUERADE, but they both work on outbound (leaving the server) packets. They replace the source IP address in the packets for their own external network device, when the packet returns, the NAT function knows who sent the packet and forwards it back to the originating workstation inside the network."

So, I had to try that. I changed my NAT-rule to:

iptables -t nat -D POSTROUTING -o em1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o em1 -j SNAT --to-source 80.my.source.IP

... and everything starts to work ok! I've been using the same masquerade-rule for at least 10 years without problems. Something must have changed in Linux-kernel.

I did further studies with this problem. On a remote server I did following on a publicly accessible directory:

# dd if=/dev/urandom of=random.bin bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 1.76654 s, 5.9 MB/s

It creates a random file of 10 MiB. For testing purposes, I can load the file with wget-utility:

# wget http://81.the.other.IP/random.bin
Connecting to 81.the.other.IP:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M) [application/octet-stream]
Saving to: `random.bin'

100%[=================>] 10,485,760  7.84M/s   in 1.3s

2013-10-08 17:06:02 (7.84 MB/s) - `random.bin' saved [10485760/10485760]

No problems. The file loads ok. The speed is good, nothing fancy there. I change the rule back to MASQUERADE and do the same thing again:

# wget http://81.the.other.IP/random.bin
Connecting to 81.the.other.IP:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760 (10M) [application/octet-stream]
Saving to: `random.bin'

10% [=======>          ] 1,090,200   --.-K/s  eta 85m 59s

After waiting for 10 minutes, there was no change in the download. wget simply hung there and would not process without manual intervention. Its official: masquerade is busted.

Me finding a bug in Linux kernel is almost impossible. I'm not a kernel developer, or anything, but anything I try finds nothing from the net. So I had to double check to rule out following:

• Hardware:
• Transferring similar file from router-box to client works fully. I tested a 100 MiB file. No issues with my LAN or the client computer.
• Transferring similar file from outside-server to router-box works fully. I tested a 100 MiB file. No issues with my Internet connection.
  
• When not NATing, everything works ok. Based on this I don't suspect any hardware issues.
• There is no difference in my home if using WLAN or Ethernet. The problem is related to my POSTROUTING-setting.
  
• IPv4:
• I have a SixXS IPv6-tunnel at my disposal. Transferring a 100 MiB file from outside-server via IPv6 to the same a IPv4 NATed client works fully. No issues.
• My original claim is that MASQUERADE is broken, SNAT works. Functioning IPv6 connection supports that claim.

To further see if it would be a Fedora-thing, or affecting entire Linux, I took official Linux 3.11.4 source code and Fedora kernel-3.11.3-201.fc19.src.rpm and ran a diff:

# diff -aur /tmp/linux.orig/linux-3.11.4/net/ipv4/netfilter \
/tmp/linux.fc19/linux-3.11/net/ipv4/netfilter

Nothing. No differences encountered. Looks like I have to file a bug report to Fedora and possibly Netfilter-project. Looking at the change log of /net/ipv4/netfilter/ipt_MASQUERADE.c reveals absolutely nothing, the change must be somewhere else.

In my Fedora 19 I've been wondering why my NTPd does not start on boot. It used to do so couple of Fedora installations ago. This is not a big deal, so I've been mostly ignoring it. Today I dug up some energy to investigate.

The reason was much simpler than I tought. On my very short checklist were:

1. Confirm that systemd has ntpd.service enabled, it was.
   
2. Confirm that ntpd.service has a dependency to start the service after network interfaces are up, it was chained to do a single ntpdate update and start the daemon after it.
3. Needed interfaces have not been blocked and/or needed interfaces have been enabled in config, everything was out-of-the-box: all network interfaces allowed.
   

The daemon even had the panic-threshold disabled in the config, so it wouldn't choke on startup if time was badly off for some reason. I found no reason for the daemon to start.

However, doing a search for ntpd in /usr/lib/systemd/system revealed what was going on. chronyd.service has Conflicts=ntpd.service in the service description. WTF?! What the hell is chronyd?

According to http://chrony.tuxfamily.org/ it is "a pair of programs which are used to maintain the accuracy of the system clock on a computer". Sounds like a NTPd to me. Running netstat confirmed the fact:

# netstat -nap | fgrep :123
udp        0      0 0.0.0.0:123             0.0.0.0:*                           666/chronyd        
udp6       0      0 :::123                  :::*                                666/chronyd        

The daemon does bind to NTP-ports. To get chronyd running properly, all I had to do was add proper time source and allowed updates from my LAN with allow-directives.

That's it!

My Fedora 19 failed to boot if I had an entry for an iSCSI-mount in /etc/fstab. During boot the system just fell to emergency mode. To get the box to boot, I simply did a "stupd man's solution", and commented the line out. This is what happens if I have the standard line in fstab:

My fstab line is:

/dev/qnap  /mnt/qnap  ext4  defaults  1 0

It took me a while to get back to the issue and investigate, it was that bad. This is the clue I found on Fedora project's documentation about iSCSI. They said, that any iSCSI-volumes should be mounted with a special flag _netdev. I changed to that, and hey presto! During bootup, it first does something and then mounts the iSCSI-drive. I merged those two occurrences into a single photo:

It works! I'm so happy about this. For clarity, the fstab-line is:

/dev/qnap  /mnt/qnap  ext4  _netdev  0 0

During my ventures in the Linux-land, I constantly package and re-package RPMs. Sometimes to introduce new functionality to existing package or to simply get a newer version that distro vendor is prepared to offer. Number of times I've created packaging to software that is not in the distro at all.

Another thing I love using are symlinks. I can have newer and older package of a software and can simply switch with updating the symlink into correct version. When I combined those two, it bit me in the ass.

I had quite simple script-blocks to handle the symlink:

%post
cd /my/package/directory/
%{__rm} -f my.cool.symlink-name
ln -s package/library/my.cool my.cool.symlink-name

%preun
%{__rm} -f /my/package/directory/my.cool.symlink-name

On install, that worked, but on update/freshen there was no symlink left. I was puzzled, why is that? Little bit of googling revealed two pieces of information: RPM spec-file documentation about scripts, especially the Install/Erase-time Scripts -part and 2nd the Fedora Project's packaging information, especially the scriptlet ordering. I'll abbreviate the Fedora's ordering here omitting the non-interesting parts:

1. %pre of new package
   This is the part where my script confirms that the symlink exists.
   
2. (package install)
3. %post of new package
4. %preun of old package
   During update/freshen, this is the part where my script removes the symlink created in 1.)
   Crap!
   
5. (removal of old package)
6. %postun of old package

Further reading of RPM spec-docs said "the argument passed to version 1.0's scripts is 1". Ok, nice to how, but now what? How can I utilize the information? What is the exact syntax for the script? The only usable information I found was in the Fedora packaging instructions, there was an example:

%preun
if [ $1 = 0 ] ; then
  /sbin/install-info --delete %{_infodir}/%{name}.info %{_infodir}/dir || :
fi

So this was the thing I had to try. My solution is to change the %preun-block:

%preun
if [ $1 -lt 1 ] ; then
  # This is really an un-install, not deleting previous version on update
  %{__rm} -f /my/package/directory/my.cool.symlink-name
fi

I did that and upgraded the package. Poooof! The symlink was gone like there was no change at all. WHY? I upgraded the revision number of the package and upgraded again. NOW it worked! Nice.

There is a simple explanation what happened. It says in the Fedora project's order-list that "%preun of old package". OLD package! It works starting from the next update, but not on the first one.

Anyway I was delighted to get that one sorted.