Linux

In my Fedora 19 I've been wondering why my NTPd does not start on boot. It used to do so couple of Fedora installations ago. This is not a big deal, so I've been mostly ignoring it. Today I dug up some energy to investigate.

The reason was much simpler than I tought. On my very short checklist were:

1. Confirm that systemd has ntpd.service enabled, it was.
   
2. Confirm that ntpd.service has a dependency to start the service after network interfaces are up, it was chained to do a single ntpdate update and start the daemon after it.
3. Needed interfaces have not been blocked and/or needed interfaces have been enabled in config, everything was out-of-the-box: all network interfaces allowed.
   

The daemon even had the panic-threshold disabled in the config, so it wouldn't choke on startup if time was badly off for some reason. I found no reason for the daemon to start.

However, doing a search for ntpd in /usr/lib/systemd/system revealed what was going on. chronyd.service has Conflicts=ntpd.service in the service description. WTF?! What the hell is chronyd?

According to http://chrony.tuxfamily.org/ it is "a pair of programs which are used to maintain the accuracy of the system clock on a computer". Sounds like a NTPd to me. Running netstat confirmed the fact:

# netstat -nap | fgrep :123
udp        0      0 0.0.0.0:123             0.0.0.0:*                           666/chronyd        
udp6       0      0 :::123                  :::*                                666/chronyd        

The daemon does bind to NTP-ports. To get chronyd running properly, all I had to do was add proper time source and allowed updates from my LAN with allow-directives.

That's it!

My Fedora 19 failed to boot if I had an entry for an iSCSI-mount in /etc/fstab. During boot the system just fell to emergency mode. To get the box to boot, I simply did a "stupd man's solution", and commented the line out. This is what happens if I have the standard line in fstab:

My fstab line is:

/dev/qnap  /mnt/qnap  ext4  defaults  1 0

It took me a while to get back to the issue and investigate, it was that bad. This is the clue I found on Fedora project's documentation about iSCSI. They said, that any iSCSI-volumes should be mounted with a special flag _netdev. I changed to that, and hey presto! During bootup, it first does something and then mounts the iSCSI-drive. I merged those two occurrences into a single photo:

It works! I'm so happy about this. For clarity, the fstab-line is:

/dev/qnap  /mnt/qnap  ext4  _netdev  0 0

During my ventures in the Linux-land, I constantly package and re-package RPMs. Sometimes to introduce new functionality to existing package or to simply get a newer version that distro vendor is prepared to offer. Number of times I've created packaging to software that is not in the distro at all.

Another thing I love using are symlinks. I can have newer and older package of a software and can simply switch with updating the symlink into correct version. When I combined those two, it bit me in the ass.

I had quite simple script-blocks to handle the symlink:

%post
cd /my/package/directory/
%{__rm} -f my.cool.symlink-name
ln -s package/library/my.cool my.cool.symlink-name

%preun
%{__rm} -f /my/package/directory/my.cool.symlink-name

On install, that worked, but on update/freshen there was no symlink left. I was puzzled, why is that? Little bit of googling revealed two pieces of information: RPM spec-file documentation about scripts, especially the Install/Erase-time Scripts -part and 2nd the Fedora Project's packaging information, especially the scriptlet ordering. I'll abbreviate the Fedora's ordering here omitting the non-interesting parts:

1. %pre of new package
   This is the part where my script confirms that the symlink exists.
   
2. (package install)
3. %post of new package
4. %preun of old package
   During update/freshen, this is the part where my script removes the symlink created in 1.)
   Crap!
   
5. (removal of old package)
6. %postun of old package

Further reading of RPM spec-docs said "the argument passed to version 1.0's scripts is 1". Ok, nice to how, but now what? How can I utilize the information? What is the exact syntax for the script? The only usable information I found was in the Fedora packaging instructions, there was an example:

%preun
if [ $1 = 0 ] ; then
  /sbin/install-info --delete %{_infodir}/%{name}.info %{_infodir}/dir || :
fi

So this was the thing I had to try. My solution is to change the %preun-block:

%preun
if [ $1 -lt 1 ] ; then
  # This is really an un-install, not deleting previous version on update
  %{__rm} -f /my/package/directory/my.cool.symlink-name
fi

I did that and upgraded the package. Poooof! The symlink was gone like there was no change at all. WHY? I upgraded the revision number of the package and upgraded again. NOW it worked! Nice.

There is a simple explanation what happened. It says in the Fedora project's order-list that "%preun of old package". OLD package! It works starting from the next update, but not on the first one.

Anyway I was delighted to get that one sorted.

Previously I've studied the init.d replacement systemd.
Update 4th Jun 2017: See the new version

To my surprise, my contraption from the previous article didn't survive a reboot. WTF?! It turned out that in Fedora 19 the /var/run/ is a symlink into /run/ which has been turned into tmpfs. Goddamnit! It literally means, that it is pointless to create /var/run/<the daemon name here>/ with correct permissions in RPM spec-file. Everything will be wiped clean on next reboot anyway.

So, I just had to study the systemd some more.

This is my version 2 (the Unit and Install -parts are unchanged):

[Service]
Type=forking
PrivateTmp=yes
User=nobody
Group=nobody
# Run ExecStartPre with root-permissions
PermissionsStartOnly=true
ExecStartPre=-/usr/bin/mkdir /var/run/dhis
ExecStartPre=/usr/bin/chown -R nobody:nobody /var/run/dhis/
# Run ExecStart with User=nobody / Group=nobody
ExecStart=/usr/sbin/dhid -P /var/run/dhis/dhid.pid
PIDFile=/var/run/dhis/dhid.pid

The solution is two-fold. First an ExecStartPre-directive is required. It allows to run stuff before actually executing the deamon. My first thing to do is create a directory, the minus sign before the command says to ignore any possible errors during creation. The target is mainly to ignore any errors from the fact that creation would fail due to the directory already existing. Anyway, all errors are ignored regardless of the reason.

The second command to run is to make sure that permissions are set correctly for my daemon to create a PID-file into the directory created earlier. That must succeed or there will be no attempt to start the daemon. chowning the directory will fail if the directory does not exist, or any other possible reason.

Sounds nice, huh? Initially I couldn't get that working. It was simply due to reason, that the entire Service-part is run as the user pointed by the User=nobody and Group=nobody -directives. That user was intentionally chosen, because it has very limited permission anywhere or anything. Now it cannot create any new directories into/var/run/. Darn!

This where the solution's 2nd part comes in. Somebody designing the systemd thought about this. Using the PermissionsStartOnly-directive does the security context switch at the last moment before starting the daemon. This effectively changes the default behavior to allow running Service-part as root, except for the daemon. Exactly what I was looking for! Now my daemon starts each and every time. Even during boot.

Another thing which I noticed, is that when I edit a systemd service-file, the changes really don't affect before I do a systemctl --system daemon-reload. It was a big surprise to me, after all in traditional init.d everything was effective immediately.

PS.
Why cronie does not create a PID-file? I had an issue in CentOS where I had not one, but two cron-daemons running at the same time. This is yet another reason to go systemd, it simply works better with ill-behaving deamons like cronie.