Operating systems - 6 - Hacker's ramblings

Improving Nuvoton NCT6776 lm_sensors output

Monday, November 16. 2015

Problem

My home Linux-box was outputting more-or-less useless lm_sensor output. Example:

coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +36.0°C (high = +80.0°C, crit = +98.0°C) Core 0: +34.0°C (high = +80.0°C, crit = +98.0°C) Core 1: +31.0°C (high = +80.0°C, crit = +98.0°C) Core 2: +36.0°C (high = +80.0°C, crit = +98.0°C) Core 3: +33.0°C (high = +80.0°C, crit = +98.0°C) nct6776-isa-0290 Adapter: ISA adapter Vcore: +0.97 V (min = +0.00 V, max = +1.74 V) in1: +1.02 V (min = +0.00 V, max = +0.00 V) ALARM AVCC: +3.33 V (min = +2.98 V, max = +3.63 V) +3.3V: +3.31 V (min = +2.98 V, max = +3.63 V) in4: +1.01 V (min = +0.00 V, max = +0.00 V) ALARM in5: +2.04 V (min = +0.00 V, max = +0.00 V) ALARM in6: +0.84 V (min = +0.00 V, max = +0.00 V) ALARM 3VSB: +3.42 V (min = +2.98 V, max = +3.63 V) Vbat: +3.36 V (min = +2.70 V, max = +3.63 V) fan1: 0 RPM (min = 0 RPM) fan2: 703 RPM (min = 0 RPM) fan3: 0 RPM (min = 0 RPM) fan4: 819 RPM (min = 0 RPM) fan5: 0 RPM (min = 0 RPM) SYSTIN: +36.0°C (high = +0.0°C, hyst = +0.0°C) ALARM sensor = thermistor CPUTIN: -60.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermal diode AUXTIN: +35.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor PECI Agent 0: +26.0°C (high = +80.0°C, hyst = +75.0°C) (crit = +88.0°C) PCH_CHIP_TEMP: +0.0°C PCH_CPU_TEMP: +0.0°C PCH_MCH_TEMP: +0.0°C

That's all great and all, but what the heck are in 1, 4-6 and fan 1-5? Are the in 1, 4-6 readings really reliable? Why are there sensors with 0 RPM readings? CPUTIN indicating -60 degrees, really? PCH-temps are all 0, why?

Investigation

In order to get to bottom of all this, let's start from the chip in question. lm_sensors -setup identified it as NCT6776. For some reason Nuvoton doesn't have the data sheet anymore, but by little bit of googling, a PDF with title NCT6776F / NCT6776D Nuvoton LPC I/O popped up.

Analog inputs:

Following information can be found:

It contains following analog inputs:

AVCC
VBAT
3VSB
3VCC
CPUVCORE
VIN0
VIN1
VIN2
VIN3

The good thing is, that first 5 of them are clearly labeled, but inputs 0 through 3 are not. They can be pretty much anything.

Revolution Pulse counters:

When it comes to RPM-readings, following information is available:

That lists following inputs:

SYSFANIN
CPUFANIN
AUXFANIN0
AUXFANIN1
AUXFANIN2

Looks like all of those have connectors on my motherboard.

Temperature Sources:

For the temperature measurements, the chip has:

The analog temperature inputs are:

SMIOVT1
SMIOVT2
SMIOVT3
SMIOVT4
SMIOVT5
SMIOVT6

According to the above table, they're mapped into AUXTIN, CPUTIN and SYSTIN.

Also on top of those, there is PECI (Platform Environment Control Interface). A definition says "PECI is a new digital interface to read the CPU temperature of Intel® CPUs". So, there aren't any analog pins for that, but there are readings available, when questioned.

Configuration

A peek in to /etc/sensors3.conf at the definition of the chip shows:

chip "w83627ehf-*" "w83627dhg-*" "w83667hg-*" "nct6775-*" "nct6776-*" label in0 "Vcore" label in2 "AVCC" label in3 "+3.3V" label in7 "3VSB" label in8 "Vbat" set in2_min 3.3 * 0.90 set in2_max 3.3 * 1.10 set in3_min 3.3 * 0.90 set in3_max 3.3 * 1.10 set in7_min 3.3 * 0.90 set in7_max 3.3 * 1.10 set in8_min 3.0 * 0.90 set in8_max 3.3 * 1.10

And that's all. I guess that would be ok for the generic case, but in my particular box that list of settings doesn't cover half of the inputs.

Solution

Configuration changes

I added following settings for temperature into "chip "w83627ehf-*" "w83627dhg-*" "w83667hg-*" "nct6775-*" "nct6776-*""-section:

label in0 "Vcore" set in0_min 1.1 * 0.9 set in0_max 1.1 * 1.15 label in1 "+12V" compute in1 @ * 12, @ / 12 set in1_min 12 * 0.95 set in1_max 12 * 1.1 label in2 "AVCC" set in2_min 3.3 * 0.95 set in2_max 3.3 * 1.1 label in3 "+3.3V" set in3_min 3.3 * 0.95 set in3_max 3.3 * 1.1 label in4 "+5V" compute in4 @ * 5, @ / 5 set in4_min 5 * 0.95 set in4_max 5 * 1.1 ignore in5 ignore in6 label in7 "3VSB" set in7_min 3.3 * 0.95 set in7_max 3.3 * 1.1 label in8 "Vbat" set in8_min 3.3 * 0.95 set in8_max 3.3 * 1.1

The obvious problem still stands: what are the undocumented in 1, 4, 5 and 6? Mr. Ian Dobson at Ubuntuforums.org discussion about NCT6776 claims, that in1 is for +12 VDC power and in4 is for +5VDC power. I cannot deny nor confirm that for my board. The Novoton-chip only provides the inputs, but there is absolutely no way of telling how the manufacturer chooses to connect them to various parts of the MoBo. I took the same assumption, so all that was necessary, was to multiply the input data by 12 and 5 to get a proper reading. I don't know what in5 and in6 are for, that's why I remove them from the display. All the other ones are min and max boundaries for the known readings.

The fan settings are machine specific, in my case:

label fan2 "CPU fan" set fan2_min 200 label fan4 "HDD fan" set fan4_min 200 ignore fan1 ignore fan3 ignore fan5

As I only have fans connected to 2 out of 5, I'll ignore the not connected ones. For the connected, I set a lower limit of 200 RPM.

Temperatures are motherboard-specific. In my case, I did following additions:

label temp1 "MB" set temp1_max 38 set temp1_max_hyst 35 label temp3 "CPU" label temp7 "CPU?" ignore temp2 ignore temp8 ignore temp9 ignore temp10

The easy part is to remove the values not displaying anything. The hard part is to try to figure out what the measurements indicate. Based on the other readings, temp3 is CPU combined somehow. The other sensor is displaying rougly same values for each core I have there. However, the temp7 is for PECI, but it doesn't behave anything like CPU-temps. It should, but it doesn't. That's why I left a question mark after it.

Resulting output

After the additions, following output is available:

coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +48.0°C (high = +80.0°C, crit = +98.0°C) Core 0: +48.0°C (high = +80.0°C, crit = +98.0°C) Core 1: +40.0°C (high = +80.0°C, crit = +98.0°C) Core 2: +43.0°C (high = +80.0°C, crit = +98.0°C) Core 3: +39.0°C (high = +80.0°C, crit = +98.0°C) nct6776-isa-0290 Adapter: ISA adapter Vcore: +1.22 V (min = +0.99 V, max = +1.26 V) +12V: +12.29 V (min = +11.42 V, max = +13.25 V) AVCC: +3.33 V (min = +3.14 V, max = +3.63 V) +3.3V: +3.31 V (min = +3.14 V, max = +3.63 V) +5V: +5.04 V (min = +4.76 V, max = +5.52 V) 3VSB: +3.42 V (min = +3.14 V, max = +3.63 V) Vbat: +3.38 V (min = +3.14 V, max = +3.63 V) CPU fan: 912 RPM (min = 200 RPM) HDD fan: 897 RPM (min = 200 RPM) MB: +35.0°C (high = +38.0°C, hyst = +35.0°C) sensor = thermistor CPU: +37.0°C (high = +80.0°C, hyst = +75.0°C) sensor = thermistor CPU?: +37.0°C (high = +80.0°C, hyst = +75.0°C) (crit = +88.0°C)

Before taking the readings, I ran sensors -s to set the min/max values.

Now my output starts making sense and I can actually monitor any changes.

PS.
At the time of writing this article, website http://www.lm-sensors.org/ was down for multiple days in a row. I can only hope, that project personnel solves the issue with the web site and it is up at the time you're seeing this.

by Jari Turkia in Linux at 18:20 | Comments (3) | Share in LinkedIn

Replacing physical drive for LVM - pvcreate Can't open /dev exclusively

Sunday, November 8. 2015

This is part 2 of my hard drive upgrade. My previous part was about failure to partition a replaced hard drive with GNU Parted: It was just emitting an error of: "The resulting partition is not properly aligned for best performance"

When I had the drive partitioned properly, I failed to proceed with my setup in a yet another mysterious error. My drives are always using LVM, so that I get more control over the filesystem sizes. To get the new partition into LVM, it needs to be associated with a Volume Group (VG). First step is to inform LVM about new physical drive:

# pvcreate /dev/sda1 Can't open /dev/sda1 exclusively. Mounted filesystem?

Oh really? It's definitely not mounted, but ... somebody is stealing my resource. The root of this problem is obviously on the fact, that there used to be a PV on that partition, but I replaced the drive and partitioned it. It is entirely possible, that LVM likes to fiddle with my new partition somehow.

The device mapper knows about the partition:

# dmsetup ls Box_vg1-LogVol_wrk2 (253:9)

That's kind of bad. I guess it likes to hold on into it. Further check of:

# pvdisplay

... indicates, that LVM doesn't know about the partition (yet), but Linux kernel does.

An attempt to fix:

# dmsetup remove Box_vg1-LogVol_wrk2

And new attempt:

# pvcreate /dev/sda1 Can't open /dev/sda1 exclusively. Mounted filesystem?

No change. Perhaps a strace will provide helpful details of the problem:

# strace pvcreate /dev/sda1 ... stat("/dev/sda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 1), ...}) = 0 stat("/dev/sda1", {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 1), ...}) = 0 open("/dev/sda1", O_RDWR|O_EXCL|O_DIRECT|O_NOATIME) = -1 EBUSY (Device or resource busy) ...

Reading a fragment of OPEN(2) man page:

OPEN(2)
open, openat, creat - open and possibly create a file

O_EXCL Ensure that this call creates the file: if this flag is
specified in conjunction with O_CREAT, and pathname already
exists, then open() will fail.

In general, the behavior of O_EXCL is undefined if it is used
without O_CREAT. There is one exception: on Linux 2.6 and
later, O_EXCL can be used without O_CREAT if pathname refers
to a block device. If the block device is in use by the
system (e.g., mounted), open() fails with the error EBUSY.

... confirms the suspicion, that somebody is holding a handle to the block device. Running lsof(8) or fuser(1) yield nothing. It's not a file-handle, when kernel has your block device as hostage.

My only idea at this point was to do a wimpy Windows-style reboot. The thing is: Linux-men don't reboot on anything, but this time I was out of ideas. I'm sure somewhere there is an IOCTL-call to release the handle, but I couldn't find it easily. So, a reboot was in order.

After the reboot: yes results:

# pvcreate /dev/sda1 Physical volume "/dev/sda1" successfully created

Then I could proceed with my build sequence. Next, associate a Volume Group with the new Pysical Volume. The options would be to to add the drive into an existing VG, or create a new one. I chose the latter:

# vgcreate Box_vg1 /dev/sda1 Volume group "Box_vg1" successfully created

Then create a logical partition, or Logical Volume in LVM-lingo on the newly created VG:

# lvcreate -L 800G -n LogVol_wrk2 Box_vg1 Logical volume "LogVol_wrk2" created

As a physical partition also a LV needs to have a filesystem on it, to be usable for the operating system:

# mkfs.ext4 /dev/Box_vg1/LogVol_wrk2 mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 209715200 4k blocks and 52428800 inodes Filesystem UUID: 93be6c97-3ade-4a62-9403-789f64ef73d0 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 102400000 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done

Now the drive was ready to be mounted and I had plenty of completely empty space waiting for data to be stored on it.

I plugged in a SATA-USB -dock and started looking for my old data. I intentionally had created a VG with precisely the same name as the old drive had, so there was an obvious collision. My syslog had entries about the pvscan:

Nov 8 16:03:21 pvscan: device-mapper: create ioctl on Box_vg1-LogVol_wrk2 failed: Device or resource busy Nov 8 16:03:21 pvscan: 0 logical volume(s) in volume group "Box_vg1" now active Nov 8 16:03:21 pvscan: Box_vg1: autoactivation failed.

Yes, that one I had coming. No autoactivation, as VG names collided. A check:

# vgdisplay --- Volume group --- VG Name Box_vg0 ... --- Volume group --- VG Name Box_vg1 ... --- Volume group --- VG Name Box_vg1 ... VG UUID trx8sq-2Mtf-2tfa-2m1P-YPGq-cVzA-6fWflU

No surprises there, there were two Volume Groups with exactly same name. To address them, there are unique identifiers or UUIDs. With UUID, it is possible to rename the VG. Like this:

# vgrename trx8sq-2Mtf-2tfa-2m1P-YPGq-cVzA-6fWflU Box_vgold Volume group "Box_vg1" successfully renamed to "Box_vgold"

Now it would be possible to activate and it would appear on udev:

# vgchange -ay Box_vgold 1 logical volume(s) in volume group "Box_vgold" now active

Now the old data was available at /dev/Box_vgold/LogVol_wrk2 and ready to be mounted and files copied out of it.

Done and mission accomplished! Now I had much more space on a fast drive.

by Jari Turkia in Linux at 20:21 | Comments (0) | Share in LinkedIn

GNU Parted: Solving the dreaded "The resulting partition is not properly aligned for best performance"

Saturday, November 7. 2015

On the other day I was cleaning out junk from my shelfs and found a perfectly good WD Caviar Black hard drive. Obviously in the current SSD-era where your only computer is a laptop and most of your data is stashed into a cloud somewhere, no regular Joe User is using spinning platters.

Hey! I'm not a regular, nor joe. I have a Linux-server running with plenty of capacity in it for my various computing needs. So, the natrural thing to do is to pop out one of the old drives and hook this 1,5 TiB high performing storage monster to replace it. The actual hardware installation on an ATX-case isn't anything worth documenting, but what happens afterwards goes pretty much this sequence: 1) partition the drive, 2) copy all/some of the old data back to it and 3) continue living successfully ever after.

The typical scenario is that something always at least hiccups, if not fails. And as expected, I choked on the 1).

Here goes:

Preparation

The drive had been used previously, and I just wasted the beginning of the drive by writing 10k sectors of nothingness. This will remove all traces of possible partition tables, boot sectors and all the critical metadata of the drive you normally value highly:

# dd if=/dev/zero of=/dev/sda bs=512 count=10000

Pay attetion to the details. It would be advisable to target a correct drive. In my case a regular JBOD-drive really appears as /dev/sda on the Linux-side. On your case, I'm pretty sure your operating system runs on /dev/sda, so please don't wipe that.

Then with GNU Parted, create a GUID partition table (or GPT):

# parted /dev/sda GNU Parted 3.1 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mktable gpt

That's it for the preparation part.

Attempt 1: The stupid way

Regardless what's on the drive already (in my case, its completely empty), Parted syntax allows an approach, where you create a partition using the maximum allowed capacity from start 0, to end -1. Like this:

(parted) mkpart LVM ext4 0 -1 Warning: The resulting partition is not properly aligned for best performance. Ignore/Cancel? c

That obviously will emit an error about non-optimal partition alignment. But hey, that's what I asked for. I obviously cancelled that attempt.

Attempt 2: The smart way

A smart approach would be to see about the boundaries:

(parted) print free Model: ATA WDC WD1502FAEX-0 (scsi) Disk /dev/sda: 1500GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags
17.4kB 1500GB 1500GB Free Space

Now we have a range of 17.4 KiB to 1500 GiB which can be used for a new partition. Let's try that:

(parted) mkpart LVM ext4 Start? 17.4kB End? 1500GB Warning: You requested a partition from 16.9kB to 1500GB (sectors 33..2929687500). The closest location we can manage is 17.4kB to 1500GB (sectors 34..2930277134). Is this still acceptable to you? Yes/No? y Warning: The resulting partition is not properly aligned for best performance. Ignore/Cancel? c

I have bumped into this number of times earlier. Why in the f**k cannot the Parted tell me what values it wants to see there!! Come on!

This is the part where it hits me like a hammer: enough bullshit, let's solve this once and for all!

Attempt 3: Solution

This is the script I wrote: parted_mkpart_calc.sh.

It is based on the information found from following sources:

How to align partitions for best performance using parted, somebody else is having the same fight than I do
I/O Limits: block sizes, alignment and I/O hints, information about the Parted alignment calculation
https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-block, Linux kernel block-device ABI information

It is a Bash-script to do the math for you. Example usage:

$ ./parted_mkpart_calc.sh sda Using default 1 MiB default alignment in calc Calculated alignment for /dev/sda (gpt) is: 2048s If you would be root, you could create partition with: # parted /dev/sda mkpart [name] [type] 2048s 2930276351s Verify partition alignment with: # parted /dev/sda align-check optimal 1 Should return: 1 aligned

I just enter one argument to the script: sda. From that, the script deduces the alignment, that should be used when partitioning that block-device. In this case it is 2048 sector boundaries (what it doesn't say is, that a sector contains 512 bytes). But it outputs 2 commands which can be copy/pasted (as root):

parted /dev/sda mkpart [name] [type] 2048s 2930276351s

If you would replace [name] with a partition name and [type] with a partition type, it would create a correctly aligned partition to fill up most of the drive. It won't fill up exactly all of the drive, because of the alignment issues.

To help that issue, I added a feature to do the following:

$ ./parted_mkpart_calc.sh sda LVM ext4

Optionally, you can provide the partition name and type on the command line to get:

parted /dev/sda mkpart LVM ext4 2048s 2930276351s

as output. That's ready-to-go copy/paste material.

Finally, you can verify the correct alignment:

# parted /dev/sda align-check optimal 1 1 aligned

That's the proof, that calc worked ok.

Attempt 4: The simple way

It didn't take long, before I got my first comment on this article. It was simply: "Why didn't you use percentages?". What? What percentages.

Example:

(parted) unit s (parted) print Model: ATA WDC WD1502FAEX-0 (scsi) Disk /dev/sda: 2930277168s Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 2048s 2930276351s 2930274305s LVM (parted) rm 1 (parted) mkpart LVM ext4 0% 100% (parted) print Model: ATA WDC WD1502FAEX-0 (scsi) Disk /dev/sda: 2930277168s Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags
1 2048s 2930276351s 2930274305s LVM

Using range 0% 100% will produce exactly the same results. Amazing!

So, parted knows the alignment and can use it, but not if you don't first do a rain dance and knock three times on a surface sprinkled with holy water.

Final Words

Why does Parted complain about mis-alignment, but offers no help at all? That's just plain stupid!

Of course, I should add the feature to the source code and offer the patch to FSF, but on the other hand. Naah. I don't want to waste any more energy on this madness.

by Jari Turkia in Linux at 16:02 | Comments (31) | Share in LinkedIn

OS X El Capitan upgrade - Afterwork with Verisign certificates

Monday, October 26. 2015

I previously wrote about upgrading OS X El Capitan. After doing couple of boxes I ran into a SNAFU.

If you don't see anything in that page, that's correct! There is nothing there. It would be a safe assumption, that something had gone wrong.

Here is what web browser console says:

Error was: "Failed to load resource: The certificate for this server is invalid." As the errors were emitting from Amazon CloudFront, it didn't make any sense at all. Either Amazon had some sort of security fault happening, or I did. Unfortunately in such situations, the odds are always against me. I had upgraded couple of Macs already and had no problems with them, this box must have had something wrong with it.

My next move was to get a list of trusted root certificates shipping with an OS X. The list is available in Apple knowledgebase article HT205204. Here is what I got:

Another error: 'Safari can't verify the identity of the website "support.apple.com"'. Right. First Amazon was failing on me, then Apple. At this point I whipped up an already upgraded Mac and went for the page, this time it looked ok:

That was the proof, that something was badly off on that Mac.

For fact gathering I went trough the certificate chain of support.apple.com:

As the certificate wasn't trusted, the page looked horrible and there was no lock-icon on the address bar. The important fact here was, that the root certificate of VeriSign Class 3 Public Primary Certification Authority - G5 had version number 3 and serial number of 25 0C E8 E0 30 61 2E 9F 2B 89 F7 05 4D 7C F8 FD. On the working Mac same certificate:

A completely different serial number of 18 DA D1 98 26 7D E8 BB 4A 21 58 CD CC 6B 2B 4A.

Then the relevant question was: Why do they differ? The facts are at OS X certificate store. It so happens, that all certificates can be viewed and altered via Keychain Access -tool. I went to see the System Roots -keychain:

But that didn't solve my problem! VeriSign Class 3 Public Primary Certification Authority - G5 was there and had the proven correct serial number of 18 DA D1 98 26 7D E8 BB 4A 21 58 CD CC 6B 2B 4A. More poking around, and this is what I found:

A set of Verisign certificates on login-keychain. Weird. One of them was:

There was the 25 0C E8 E0 30 61 2E 9F 2B 89 F7 05 4D 7C F8 FD! The only appropriate action was:
And that solved it! Simply letting the weird ones go made all my websites work again.

But where did those certs come from? By googling I found Why is Symantec/Verisign CA appearing as an invalid authority? [closed] and Invalid certificate after Security Update 2015-004 in Mavericks. They both were pointing a finger to April 2015 security update. The release notes About the security content of OS X Yosemite v10.10.3 and Security Update 2015-004 say:

Certificate Trust Policy
Impact: Update to the certificate trust policy
Description: The certificate trust policy was updated. View the complete list of certificates.

I just happened to update the Mac too early and got a flawed upgrade. It is also possible, that on April, when I got bad certs, I may have gone to Verisign and manually loaded the proper root certificates in to fix my problem at the time. However, it just blew up on my face on OS X 11 upg.

If you never encountered any of this: good. I honestly don't think this issue is touching a wide audience. However, I disclosed this information for archive purposes. If something like this happens in the future, you have a clue what to look for.

by Jari Turkia in MacOS at 16:08 | Comments (0) | Share in LinkedIn

OS X El Capitan upgrade from USB-stick

Monday, October 12. 2015

Now that OS X El Capitan or version 11 is out, I'll do a refresh for the USB installation instructions.

The thing is ... the upgrade free, but it's big. The amount of downloading needed is easily 6+ GiB. I have 3 Macs to update and I don't want to download the huge package on all of them. So, let's figure out something smarter.

Step 0: Prerequisites

You'll need a bootable USB-stick with capacity of 8 GiB or more. All sticks should boot, but I have encountered some that didn't manage that.

USB-booting a Mac is trickier than a PC. The knowledge base article HT1948 states:
Intel-based Macs support starting from an external USB storage device's volume that:

Has been formatted with a GUID partition type
Contains an installation of Mac OS X 10.4.5 or later, or Mac OS X 10.5 or later, which is compatible with (or shipped with) the Mac that the USB device is connected to. Note: You should not use a version of Mac OS X that is earlier ("older") than the version your Mac shipped with.

So, if you just bought an USB-stick, the chances are, that it is MBR-partitioned FAT32. That's my experience of getting new ones. They are incompatible at their current state for USB-booting a Mac. Not to worry, that can be fixed!

If you happen to have a ready-made stick for any previous OS X version, that obviously can be loaded with new installer. You can even skip couple of things during the process as you don't have to reformat the stick.

Step 1: Go download

In your Apple menu (the top left apple-shaped thing at every program's menu), go for Software Update. An alternate is to click the App Store link:

Both options land you on the App Store main screen showing you something like this:

Click the image saying Free Upgrade (free as in beer):

Your Apple ID credentials are required for this free package. They'll keep track of who downloaded and what.

This is the part you'll wait for the download to complete:

When it's all on your machine, the installer will automatically kick in.

Step 2: Go USB

Now that you have the thing in your drive, don't proceed with the upgrade.

When you reach this screen:

do not proceed! You can quit the installer, if you want:

It won't delete the files from your drive. It is also possible to continue installing on that Mac, but don't do it yet. Take a copy of the files first.

Take at least a 8 GiB USB-storage. 4 won't do it, but any larger will. In my case, the USB-stick appeared as /dev/disk3. That may vary on your system. Also it is possible to use some GUI-tools on OS X to format your drives, but as a Linux-nerd I don't know about them.

It is very likely, that the disk is mounted and will display an icon on your desktop, and will appear on your Finder. The diskutil will unmount it automatically on partition, but I wanted to make sure and did:

# sudo diskutil umount /Volumes/MyUSBdrive

Next step is to make sure, the stick is in a Mac-format (this needs to be run as root, that's what the sudo is for). This will partition and format the entire stick into Mac-use:

# sudo /bin/bash root# diskutil partitionDisk /dev/disk3 1 GPT jhfs+ "OS X El Capitan" 0b

It will say something like this as a result:

Started partitioning on disk3 Unmounting disk Creating the partition map Waiting for the disks to reappear Formatting disk3s2 as Mac OS Extended (Journaled) with name OS X El Capitan Initialized /dev/rdisk3s2 as a 7 GB case-insensitive HFS Plus volume with a 8192k journal Mounting disk Finished partitioning on disk3 /dev/disk3 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *8.0 GB disk3 1: EFI EFI 209.7 MB disk3s1 2: Apple_HFS OS X El Capitan 7.7 GB disk3s2

Next thing is to confirm, that the volume with given name will be mounted:

root# ls -l /Volumes/ total 40 lrwxr-xr-x 1 root admin 1 Aug 29 12:24 Macintosh HD -> / drwxrwxr-x 7 root wheel 306 Oct 10 12:41 OS X El Capitan

If it does, you're ready to go. Copy the thing into it:

root# cd /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources/ root# ./createinstallmedia --volume /Volumes/OS\ X\ El\ Capitan/ \ --applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app/ \ --nointeraction

It will result in a lengthy process saying:

Erasing Disk: 0%... 10%... 20%... 30%...100%... Copying installer files to disk... Copy complete. Making disk bootable... Copying boot files... Copy complete. Done.

Step 3: Go update

Your stick is ready. This is the part you will be replicating to any of your Macs you want to upgrade.

Reboot the Mac and make sure to boot from the USB. This can be achieved by pressing down option-key during boot:

The official Apple instruction at knowlegebase article HT1948 states:
To start from a USB storage device that meets the above requirements:

connect the device
restart
immediately press and hold the Option key to access Startup Manager

If you successfully followed the steps, you will end up in Mac boot manager:

In that, you pretty much select the drive you want to boot from. In this particular case, making a choice for the recently prepared USB-stick will be a good one. Click the orange "Install OS X El Capitan". Both the gray HD and the network selection are there to confuse you. Ignore them and double click the USB-drive.

Most steps in this upgrade will include lot of waiting. Make sure that you have reserved couple of hours for the upgrade. Screens like this will become familiar to you:

Most time estimates are wild guesses. A 9 minute wait in reality is something like 45 minutes. Eventually the USB-stick finishes booting, and you will end up in a screen saying "To set up the installation of OS X, click Continue". Most screens will refer your upgrade as an install. It is nerve-wrecking thing, because you don't know if it is going to wipe your settings and data, or do a nice upgrade what you'd be expecting. My experience is, that it will upgrade nicely, but it won't say it properly.

Then there is a license screen which you must agree to continue. Then land on OS X Utilities. One thing you can do with a bootable stick is to upgrade or install an OS X:

When you select to go for upgrade or install, there is a welcome to OS X El Capitan screen. Click Continue. Yet another license screen appears. Click Agree and for the confirmation dialog: I have read and agree to the terms of the software license agreement, Agree. Select the install/upgrade destination as the one having a hard drive icon Macintosh HD (typical installations have that), then click Install. As the first thing, the installer/upgrader will prepare for the operation and then go for the real thing:

This is the most time-consuming part. After a minute or so, there will be an not-so-accurate estimate of time remaining for the preparation. When it has run its course, the actual installer will start. So if the screen says 1 minute left, do not believe it for a second. In my iMac, one cup of coffee doesn't do it. You can easily cook and eat a meal and then have the coffee while the installer/upgrader runs. Especially the final phase saying "About a second remaining" will take ages. My hardware isn't especially old or slow, but ... the process is.

Step 4: Finishing touches

After you've done your waiting. A reboot will result. At this point you'll need to login into your precious upgraded Mac with your local user account.

On successful login, you will end up in the OOBE (or out-of-box experience). Apple makes an effort to not allow stolen hardware to be used and they pretty much require you to login to Apple ID during install:

Since I have a 2-factor authentication enabled for my Apple ID, a 2FA-screen will appear during the process:

There are steps about sending your usage data to Apple. Then yet another license screen which you have to agree twice. First on the bottom screen and then on the pop-up that will appear. So if you want to complain later, they'll just say "but you did agree to our terms and conditions". It will look like this:

The last question you'll need to answer is about setting up iCloud Keychain. I choose not to share my passwords into any cloud services, but at this point you'll have the option to enable Keychain:

Little bit of setting up ... this one won't take long. And then you're pretty much done. Finally your upgrade is ready!

Step 5: Done!

Login for the first time:

One of the first things I did, was eject my USB-stick (there were couple other Macs to be updated, too):

At this point, you can continue using your precious Mac.

Aftermath

I updated anyway, as a nerd I like the latest stuff running on my computers. I should yield less problems and there needs to be some progress. I find myself stating the same thing in couple of my blog posts, "it wasn't worth it, but I did it anyway". With computers, it never will result any good to stand still and ignore future.

by Jari Turkia in MacOS at 05:37 | Comments (5) | Share in LinkedIn

HighPoint RocketRAID 620 Linux driver

Wednesday, May 27. 2015

I've had my run of bad things with Intel soft-RAID earlier. The constant RAID-verify -runs made me want stop using it. As its my Windows-box, I just wanted something that is hardware-based, reasonably fast and affordable. My choice is HighPoint RocketRAID 620.

For a switch-over -project I had a simple plan:

Clonezilla the existing RAID into an another drive which I could plug into a motherboard for the duration of the move
Un-configure the Intel soft-RAID at the motherboard
Plug in the RocketRAID-card
Change the hard-drive cables from motherboard RAID-connectors into RocketRAID-card
Configure a new RAID-1 mirror wit RocketRAID
Clonezilla the data back to the newly created RAID-1 volume
Be happy and continue computing

Guess what. Things fell trough at point #6. I was using an USB-bootable Clonezilla live on my first data move and obviously was planning to use it for the second one too. Whichever Linux-distro they use as the base for Clonezilla, they don't have the driver for the RAID-card. Darn!

The next best thing is a commercial distro for Clonezilla, Parted Magic. They used to be free (as in beer and speech), but they went commercial. The price is $9 USD for a single download, so I got it. And guess what again! They don't support Highpoint RocketRAID either.

I did ask about it in their support forums (closed to registered users only, sorry). And they replied:

We do not do "random" out-of-tree drivers because commonly these are supported by their vendors in a haphazard way. E.g. in the HighPoint case the latest driver is 3 versions behind our kernel version.

Luckily the vendor is providing the partial source code for the driver. There is a binary-part of in it and it is kind-of open-source. The biggest problem seems to be, that it doesn't build on any reasonably modern Linux.

By googling, I found that somebody else had the same process of thought and there was a Github project for the upgraded driver. Unfortunately that too was 3 years old and wouldn't build. Also it was for the vendor driver 1.1, and they already had 1.2 out.

In this imperfect world everything that you need to be done properly, you need to do by yourself. So, here it is: https://github.com/HQJaTu/rr62x
You can help yourselves with that one.

This is how it looks on my dmesg:

[ 85.518732] rr62x: module license 'Proprietary' taints kernel. [ 85.518737] Disabling lock debugging due to kernel taint [ 85.519709] rr62x:RocketRAID 62x SATA controller driver v1.2 (Jul 1 2012)
[ 85.735773] rr62x:adapter at PCI 3:0:0, IRQ 16 [ 85.950487] rr62x:[0 0 ] start port. [ 85.950488] rr62x:[0 0 ] start port hard reset (probe 1). [ 86.150712] rr62x:[0 1 ] start port. [ 86.150712] rr62x:[0 1 ] start port hard reset (probe 1). [ 89.093649] rr62x:[0 0 ] start port soft reset (probe 1). [ 89.841048] rr62x:[0 1 ] start port soft reset (probe 1). [ 90.501075] rr62x:[0 0 ] port started successfully. [ 90.501078] rr62x:[0 0 0] device probed successfully. [ 90.791364] rr62x:[0 1 ] port started successfully. [ 90.791369] rr62x:[0 1 0] device probed successfully. [ 90.806570] scsi host13: rr62x [ 90.806870] scsi 13:0:0:0: Direct-Access HPT DISK_13_0 4.00 PQ: 0 ANSI: 5 [ 90.809711] sd 13:0:0:0: [sdd] 2930114560 512-byte logical blocks: (1.50 TB/1.36 TiB) [ 90.809847] sd 13:0:0:0: [sdd] Write Protect is off [ 90.809852] sd 13:0:0:0: [sdd] Mode Sense: 2f 00 00 00 [ 90.809909] sd 13:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 90.832339] sdd: unknown partition table [ 90.832903] sd 13:0:0:0: [sdd] Attached SCSI disk

I've tested that with Linux 4.0.4 and 3.19.3. It builds and works on both. Any comments, Github forks, pull requests, etc. are welcome. I will get back to the actual disk cloning project later, the driver won't help unless it is used properly in an operating system.

by Jari Turkia in Linux at 19:44 | Comments (14) | Share in LinkedIn

Windows 10 preview - My experiences

Wednesday, November 26. 2014

Windows 10 Technical Preview has been out for two months now on Windows Insider Program. I installed it pretty much when it was released into a virtual machine and has been running a number of different software, that I'd use anyway on it.

The install process is pretty much the same than it has been since Windows 7. I created a virtual 20 GiB drive (which later turned out to be not enough) and chose to install on the non-initialized drive. No surprises there. This is what runs as out-of-box-experience:

It looks very much like Windows 8.1. Notice how the window border is quite thin. This is something Microsoft hasn't done in their operating systems ever before. Internet Explorer is still version 11 as in Windows 8.1 and the reported user-agent string is: Mozilla/5.0 (Windows NT 6.4; WOW64; Trident/7.0; rv:11.0) like Gecko.

It sees the virtual CPU incorrectly as a Xeon, however the physical CPU is of Sandy Bridge microarchitecture:

Finally the start menu is something, that I would like to use. Microsoft had a decent start menu on Windows 7, they dropped it for Windows 8 and restored a crappy version for Windows 8.1. This time it works and the stupid full-screen-apps -mode is gone. The start menu will contain both the classic and tile-based apps at the same time:

This is what computer properties and disc management look like. Pretty much the same than in Windows 7:

Also control panel is unchanged:

The version is displayed as Windows NT 6.4, but apparently it is going to change and the version number will be 10. Also IPv6 works as expected:

I don't know what the idea with forcing automatic updates on, but luckily there is a way to change that. The GUI won't let you touch it, but a direct registry hack to change it: How to disable Automatic Updates in windows 10 Tech Preview. I changed the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update value AUOptions into 3 (Automatically download and notify of installation). This is a change for previous Windowses, which are using a different registry key. See Article ID 328010 - How to configure automatic updates by using Group Policy or registry settings about it. Anyway, this is how it looks like:

One other thing I also changed, was to speed up the preview image process. As a default, new installations are on a slow track, but I wanted to get the new versions a bit faster. This is a bit funny thing, the only way to change it is to go for a new-style full-screen settings (luckily this runs in a window). There doesn't seem to be a control panel -setting for this. For details see: How to Stay Up to Date and Get the Latest Windows 10 Technical Preview Builds. This is how I changed the setting:

To upgrade into a newer build (9860 in this case) it goes something like this. I found the process quite amusing:

When the slow process completes, the bottom right corner will state the new version:

During the testing I haven't had many issues, but here is one:

For some reason system interrupts started hogging lot of CPU-resources. That is something I've never seen on any Windows. After a nice reboot, the problem disappeared. That's what you get from running beta operating system.

Also, my original 20 GiB of disc space ran out after couple of new build upgrades. This is what happened:

There wasn't much else to do, except to shut down the Windows 10 instance. Go to virtualization host (Linux) and add 10 GiB of space into the virtual drive:
qemu-img resize Windows\ 10.qcow2 +10G

As no Windows version is capable of actually expanding the partition and file system on a grown drive, I mounted an ISO-image of GParted and booted the virtual machine from that. It could expand the system partition in a couple of seconds (it is a really good partition manager software). After having 30 GiB of space, the latest build upgraded without problems.

My thanks goes to F-Secure. I've been running their new FS Protection (still on beta) on my Windows 10 and it works really well.

by Jari Turkia in Windows at 21:25 | Comments (0) | Share in LinkedIn

Microsoft Windows GPT support or "Excuse me! Which partition?"

Sunday, November 9. 2014

During a regular day I run a number of different computers. One day I wanted to take a copy of my Wall-E DVD and transfer it to iPad for later viewing. Since I sync that iPad to a particular Mac, I had to transfer the H.264 file of the movie there. As this was a one time thing and I happened to have a suitable USB-stick at my desk, I chose to go the old fashioned way. Copy the file to the stick and walk with the stick to the other machine. Except ... I couldn't copy the file to the USB stick. The stick was initialized with Mac OS X and contained a GUID partition table (or GPT). It completely refused to work with my Windows 7. What! Really?

The official word from Windows and GPT FAQ is:
Q: Can Windows 7, Windows Vista, and Windows Server 2008 read, write, and boot from GPT disks?
A: Yes, all versions can use GPT partitioned disks for data. Booting is only supported for 64-bit editions on UEFI-based systems.

Sorry, but I beg to differ. There seems to be some level of support, but ... not to the full extent. We're talking about Linux or OS X -level support here.

The stick looks like this during entire process:

# parted /dev/sde print
Model: JetFlash Transcend 8GB (scsi)
Disk /dev/sde: 8032MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start   End     Size    File system Name                  Flags
1      20.5kB 210MB   210MB   fat32        EFI System Partition boot
2      210MB   8031MB 7821MB fat32        Video

And this is how my Mac saw the situation:

$ mount
/dev/disk2s2 on /Volumes/VIDEO (msdos, local, nodev, nosuid, noowners)

This is what the Disk Manager of my Windows 7 said:

All looks good, except the OS failed to automatically assign a drive letter to the partition. No biggie, I can manually add one:

... except, I couldn't:

The error message of The system cannot find the file specified gave me an impression, that things aren't rolling my way.

At this point it wasn't about copying the file, but finding out what was going on. I had a Windows 8.1 right next to my Windows 7, so I plugged the USB into that one. This is what it saw:

But wait! There was something fishy about it. The drive capacity didn't match. To test it, I copied a file to the drive:

... and confirmed the Windows 8.1 error on a Linux:

# mount /dev/sde1 /mnt/usb/
# ls -l /mnt/usb/
total 1
-rwxr-xr-x. 1 root root 0 Oct 30 11:57 juttuhomma.txt
drwxr-xr-x. 2 root root 512 Oct 30 11:56 System Volume Information

Yep. The drive should be sde2, not 1! On a mac it was disk2s2, which would be equivalent. But a Windows 8.1 assigned the drive E: to 2nd partition! Nice. Apparently the support for GPT isn't that great.

And for those screaming "Piracy!" here's the pic:

According to Directive 2001/29/EC only "communication to the public" or "making available to the public" are limited. As I own a copy and am not publishing anything, ripping the encrypted MPEG-2 stream, converting it into H.264 and viewing it from my iPad is fully within my rights.

by Jari Turkia in Windows at 19:32 | Comments (0) | Share in LinkedIn

Fedora 20: Services failing to start on boot

Thursday, October 23. 2014

Running Services After the Network is up

Something funny happened to my Fedora 20 installation. Stuff that used to work stopped doing so. I understand, that I'm dealing with "bleeding edge" Linux-distro, but it kind of starts to annoy me, when things don't get back like they were.

Here is one:

# systemctl status iscsi.service --all
iscsi.service - Login and scanning of iSCSI devices
   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled)
   Active: inactive (dead)
           start condition failed at Thu 2014-07-17 12:23:18 EEST; 1min 24s ago
           none of the trigger conditions were met
     Docs: man:iscsid(8)
           man:iscsiadm(8)

Jul 17 12:23:17 box systemd[1]: Started Login and scanning of iSCSI devices.
Jul 17 12:23:18 box systemd[1]: Started Login and scanning of iSCSI devices.

Aow come on! "none of the trigger conditions were met"!! I don't know what changed with the precious trigger conditions. Basic checks didn't reveal anything noteworthy:

# systemctl is-active iscsi.service
active
# systemctl list-dependencies iscsi.service
iscsi.service

There are some other people suffering from the same issue. Fedora 20 ypbind won’t start at boot. NIS logins unavailable immediately after boot (ypbind service NOT started). and Bug 1027114 - nfs mount fails during boot because network is not ready point me to the direction of service trying to start too early. There is a document explaining how to run services only after network is up (Running Services After the Network is up), but the point is: somebody goofed! This used to work.

I tried a classic off/on -trick:

# systemctl disable iscsi.service
# systemctl enable iscsi.service

... no avail. Still after a boot I have a list of failing services:

# systemctl --failed
UNIT             LOAD   ACTIVE SUB    DESCRIPTION
arpwatch.service loaded failed failed Arpwatch daemon which keeps track of ether
dhcpd.service    loaded failed failed DHCPv4 Server Daemon
iscsi.service    loaded failed failed Login and scanning of iSCSI devices

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

3 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Finally I chose to add the After=network-online.target -line to my service description. First the service description needs to be copied from /usr/lib/systemd/system/ into /etc/systemd/system/. The rule is very simple: don't edit files under /usr/ and it applies also here. Systemd has a built in fall-back mechanism. If your own tailor made file doesn't exist in /etc/, it will take the default one. It is good to have that, but better if no such tinkering wasn't needed at all. What if there is an update to the service-description? I have to manually update my copied file.

So, I did a:

# cd /etc/systemd/system/
# cp /usr/lib/systemd/system/iscsi.service .

and edited the file. I added one line to Unit-description:

[Unit]
After=network-online.target

To be absolutely clear: I didn't touch the other lines.

Since systemd doesn't pick up the changes very easily, I did a:

# systemctl disable iscsi.service
# systemctl enable iscsi.service

after status informed, that my change was in place:

# systemctl status iscsi
iscsi.service - Login and scanning of iSCSI devices
Loaded: loaded (/etc/systemd/system/iscsi.service; enabled)

Notice how, the service description is now under /etc/.

Anyway, that did the trick for all of my failing services. Now they start nicely on bootup.

by Jari Turkia in Linux at 18:55 | Comments (0) | Share in LinkedIn

OS X Yosemite upgrade from USB-stick

Saturday, October 18. 2014

Latest OS X version Yosemite or 10 is out. Funny thing, the operating system 10 has a version 10 released.

The thing is ... it's free, but it's big. The amount of downloading needed is easily 5+ GiB. When I downloaded mine, it said 6 hours of load time. On my 250 Mbit/s fiber! Argh. It downloaded a hour or so and choked completely. Argh, argh! When I resumed, it picked a better server and I got rest of the file in 10 minutes or so. Anyway, I absolutely, positively don't want to do that on all of my Macs. So, let's figure out something smarter on that.

The sources of my information are: