GNU Parted: Solving the dreaded "The resulting partition is not properly aligned for best performance"
Saturday, November 7. 2015
On the other day I was cleaning out junk from my shelfs and found a perfectly good WD Caviar Black hard drive. Obviously in the current SSD-era where your only computer is a laptop and most of your data is stashed into a cloud somewhere, no regular Joe User is using spinning platters.
Hey! I'm not a regular, nor joe. I have a Linux-server running with plenty of capacity in it for my various computing needs. So, the natrural thing to do is to pop out one of the old drives and hook this 1,5 TiB high performing storage monster to replace it. The actual hardware installation on an ATX-case isn't anything worth documenting, but what happens afterwards goes pretty much this sequence: 1) partition the drive, 2) copy all/some of the old data back to it and 3) continue living successfully ever after.
The typical scenario is that something always at least hiccups, if not fails. And as expected, I choked on the 1).
Here goes:
Preparation
The drive had been used previously, and I just wasted the beginning of the drive by writing 10k sectors of nothingness. This will remove all traces of possible partition tables, boot sectors and all the critical metadata of the drive you normally value highly:
# dd if=/dev/zero of=/dev/sda bs=512 count=10000
Pay attetion to the details. It would be advisable to target a correct drive. In my case a regular JBOD-drive really appears as /dev/sda
on the Linux-side. On your case, I'm pretty sure your operating system runs on /dev/sda
, so please don't wipe that.
Then with GNU Parted, create a GUID partition table (or GPT):
# parted /dev/sda
GNU Parted 3.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable gpt
That's it for the preparation part.
Attempt 1: The stupid way
Regardless what's on the drive already (in my case, its completely empty), Parted syntax allows an approach, where you create a partition using the maximum allowed capacity from start 0, to end -1. Like this:
(parted) mkpart LVM ext4 0 -1
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? c
That obviously will emit an error about non-optimal partition alignment. But hey, that's what I asked for. I obviously cancelled that attempt.
Attempt 2: The smart way
A smart approach would be to see about the boundaries:
(parted) print free
Model: ATA WDC WD1502FAEX-0 (scsi)
Disk /dev/sda: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt Disk Flags:
Number Start End Size File system Name Flags
17.4kB 1500GB 1500GB Free Space
Now we have a range of 17.4 KiB to 1500 GiB which can be used for a new partition. Let's try that:
(parted) mkpart LVM ext4
Start? 17.4kB
End? 1500GB
Warning: You requested a partition from 16.9kB to 1500GB (sectors 33..2929687500).
The closest location we can manage is 17.4kB to 1500GB (sectors 34..2930277134).
Is this still acceptable to you? Yes/No? y
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? c
I have bumped into this number of times earlier. Why in the f**k cannot the Parted tell me what values it wants to see there!! Come on!
This is the part where it hits me like a hammer: enough bullshit, let's solve this once and for all!
Attempt 3: Solution
This is the script I wrote: parted_mkpart_calc.sh.
It is based on the information found from following sources:
- How to align partitions for best performance using parted, somebody else is having the same fight than I do
- I/O Limits: block sizes, alignment and I/O hints, information about the Parted alignment calculation
- https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-block, Linux kernel block-device ABI information
It is a Bash-script to do the math for you. Example usage:
$ ./parted_mkpart_calc.sh sda
Using default 1 MiB default alignment in calc
Calculated alignment for /dev/sda (gpt) is: 2048s
If you would be root, you could create partition with:
# parted /dev/sda mkpart [name] [type] 2048s 2930276351s
Verify partition alignment with:
# parted /dev/sda align-check optimal 1 Should return: 1 aligned
I just enter one argument to the script: sda
. From that, the script deduces the alignment, that should be used when partitioning that block-device. In this case it is 2048 sector boundaries (what it doesn't say is, that a sector contains 512 bytes). But it outputs 2 commands which can be copy/pasted (as root):
parted /dev/sda mkpart [name] [type] 2048s 2930276351s
If you would replace [name]
with a partition name and [type]
with a partition type, it would create a correctly aligned partition to fill up most of the drive. It won't fill up exactly all of the drive, because of the alignment issues.
To help that issue, I added a feature to do the following:
$ ./parted_mkpart_calc.sh sda LVM ext4
Optionally, you can provide the partition name and type on the command line to get:
parted /dev/sda mkpart LVM ext4 2048s 2930276351s
as output. That's ready-to-go copy/paste material.
Finally, you can verify the correct alignment:
# parted /dev/sda align-check optimal 1
1 aligned
That's the proof, that calc worked ok.
Attempt 4: The simple way
It didn't take long, before I got my first comment on this article. It was simply: "Why didn't you use percentages?". What? What percentages.
Example:
(parted) unit s
(parted) print
Model: ATA WDC WD1502FAEX-0 (scsi)
Disk /dev/sda: 2930277168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 2048s 2930276351s 2930274305s LVM
(parted) rm 1
(parted) mkpart LVM ext4 0% 100%
(parted) print
Model: ATA WDC WD1502FAEX-0 (scsi)
Disk /dev/sda: 2930277168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number Start End Size File system Name Flags
1 2048s 2930276351s 2930274305s LVM
Using range 0% 100% will produce exactly the same results. Amazing!
So, parted knows the alignment and can use it, but not if you don't first do a rain dance and knock three times on a surface sprinkled with holy water.
Final Words
Why does Parted complain about mis-alignment, but offers no help at all? That's just plain stupid!
Of course, I should add the feature to the source code and offer the patch to FSF, but on the other hand. Naah. I don't want to waste any more energy on this madness.
kozaki on :
Thank you for sharing.
Turo on :
Jari Turkia on :
Carmine on :
peter on :
Andre (Brazil) on :
If I already have data on my disks (production) and my partitions are misaligned, would this maneuver you taught above destroy my data? Was it a disruptive maneuver? I'm waiting and I'm grateful. Always success for you ...
Thank you
Jari Turkia on :
PcMedix on :
Thank You!
Matthijs Kooijman on :
When you specify a start (or end) position to the mkpart command, it internally generates a range of acceptable values. This range is centered on the value you specify, and extends equally on both sides by half the unit size you used (that's what I read from the code, the comments say one full unit size on both ends).
For example, when you specify "10M", it will try to use any position between 9.5M and 10.5M. The same goes for percentages, so if you specify 0% it will use any value between 0% and 0.5% (it does not go negative, obviously).
One exception is that when using power-of-two-units (such as KiB for 1024, as opposed to K for 1000) parted assumes you're trying to specify an exact position and only considers the exact value you specified.
Since optimal alignment typically seems to be 1MiB-aligned, the K and M units will often not have sufficient room to reach optimal alignment. Specifying positions in G should have plenty room, but % is usually also fine.
So, this is why 0% usually works, though even that has limits.
Alex on :
Jari Turkia on :
The obvious thing is, that many parted-recipes, including the one you're suggesting are not widely known. The obvious location would be the parted help, or man-page. No avail (yet).
Vincent on :
would use the whole disk and calculate optimal alignment. and yes, fsf ppl could use a patch for a more friendly warning
wouter on :
I tried this and it works for the first primary partition and also first logical partition.
I couldn't figure it out for the second logical partition inside the extended partition. the script won't hlpt you then an neither will the command with the percentages.
But I think I found the the best way to get the optimal alignment everytime by running parted with these options:
parted --align optimal
Tested it on GNU Parted 3.2
steve on :
melvyn on :
patrick on :
very interesting post, good script but with bug when optimal_io_size > 0
I think it will be better with:
sed -i '53s/\$2/\$3/' parted_mkpart_calc.sh
sed -i '54s/\$3/\$2/' parted_mkpart_calc.sh
sed -i '130s/default\(.*\)$/optimal\1 1048576/' parted_mkpart_calc.sh
sed -i '66,84d' parted_mkpart_calc.sh
Regards
Jari Turkia on :
The updated version is available at http://opensource.hqcodeshop.com/Parted%20calculator/parted_mkpart_calc.sh. Also the previous version is there.
Since patch/diff is more suited for such changes than sed, my patch is here:
--- ver-1/parted_mkpart_calc.sh 2015-11-07 19:51:19.284135585 +0200
+++ parted_mkpart_calc.sh 2019-06-10 20:15:22.384386454 +0300
@@ -2,7 +2,7 @@
# vim: softtabstop=4 shiftwidth=4 expandtab
-# Copyright (c) 2015 Jari Turkia (jatu@hqcodeshop.fi)
+# Copyright (c) 2015-2019 Jari Turkia (jatu@hqcodeshop.fi)
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -21,6 +21,7 @@
# for further details about using this script.
# Version history:
+# 0.2 10th Jan 2019 Changes suggested by Patrick
# 0.1 7th Nov 2015 Initial version of parted_mkpart_calc.sh
@@ -46,12 +47,12 @@
# Calculate the preferred sector alignment
# when a optimal_io_size is known
#
-function optimal_io_size_calc {
+function io_size_calc {
local __resultvar=$1
- local dev alignment_offset start_sector
+ local dev alignment_offset start_sector optimal_io_size
- optimal_io_size=$2
- dev=$3
+ dev=$2
+ optimal_io_size=$3
# alignment_offset: how many bytes the beginning of the device is offset
# from the disk's natural alignment [bytes]
@@ -63,25 +64,6 @@
eval $__resultvar="'$start_sector'"
}
-#
-# Calculate the preferred sector alignment
-# for default case of 1 MiB
-#
-function default_io_size_calc {
- local __resultvar=$1
- local dev alignment_offset start_sector
-
- dev=$2
-
- # alignment_offset: how many bytes the beginning of the device is offset
- # from the disk's natural alignment [bytes]
- alignment_offset=$(cat /sys/block/$dev/alignment_offset)
-
- start_sector=$(( (1048576 + $alignment_offset)/$physical_block_size ))
-
- eval $__resultvar="'$start_sector'"
-}
-
#
# Begin script
@@ -125,13 +107,13 @@
exit 1
fi
-if [ $optimal_io_size == 0 ]; then
- echo Using default 1 MiB default alignment in calc
- default_io_size_calc align_to_sector $BLOCK_DEVICE
+if [ $optimal_io_size -le 0 ]; then
+ echo Using default 1 MiB alignment in calc
+ optimal_io_size=1048576
else
- echo Using optimal_io_size: $optimal_io_size in calc
- optimal_io_size_calc align_to_sector $BLOCK_DEVICE $optimal_io_size
+ echo Using detected optimal_io_size alignment of $optimal_io_size bytes in calc
fi
+io_size_calc align_to_sector $BLOCK_DEVICE $optimal_io_size
if [ -z "$align_to_sector" ]; then
echo Internal error: Failed to calculate sector alignment for device /dev/$BLOCK_DEVICE
Ciantic on :
It doesn't seem to align automatically even with percentages either if you deal with small images, e.g.
dd if=/dev/null of=myimage.img bs=1 count=0 seek=128M
parted myimage.img --script -- unit s mklabel gpt mkpart primary fat32 0% 20% mkpart primary ext4 20% 100%
Throws a bunch of alignment problems :/
Nikolay on :
In the version available by the link you have:
aligned_start_sector=$(( ($free_space_start / $align_to_sector)+$align_to_sector ))
better written as
$(( ( ($free_space_start / $align_to_sector) + 1 )*$align_to_sector ))
Please notice, I removed '+' in favour of '*'
Jari Turkia on :
So, what's the gain here? Arithmetics can be written in infinite ways, but I'm not sure which one of those is better than other. There has to be some reasoning or logic for you to offer the alternative approach, but I'm missing it. Can you elaborate on your suggestion, thanks.
Nikolay on :
Look at my GPT:
root@drake:~# parted --script /dev/sdh unit s print free
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sdh: 976773168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
34s 2047s 2014s Free Space
1 2048s 4983072s 4981025s ext2
4983073s 4985119s 2047s Free Space
2 4985120s 11646168s 6661049s ext2
11646169s 11648215s 2047s Free Space
3 11648216s 13603944s 1955729s ext2
13603945s 13605991s 2047s Free Space
... and so on ...
14 385339352s 482130032s 96790681s ext2
482130033s 976773134s 494643102s Free Space
==== End of command output ====
I wanted to get alignment suggestion about the last Free space
Your version suggests parted command:
root@drake:~# ./thier.sh sdh
Using default 1 MiB alignment in calc
Calculated alignment for /dev/sdh (gpt) is: 2048s
Create partition with:
# parted /dev/sdh mkpart [name] [type] 237463s 976773119s
Verify partition alignment with:
# parted /dev/sdh align-check optimal 15
Should return: 15 aligned
==== End of command output ====
Partition start sector 237463s is obviously wrong, I expect value close to 482130033s but not one that small
Jari Turkia on :
Actually, I puzzles me why I wrote the INCORRECT arithmetic to begin with. Your suggestion is not equivalent to my original code. Given Bash integer-only arithmetic, for end sector I divide by sector alignment and multiply it back tp get the result. For start sector I only divide. Why would this work?!
Thank you for your time and effort. Also thank you for getting back to me to clarify the problem. A fix is uploaded to http://opensource.hqcodeshop.com/Parted%20calculator/
emrah on :
mkpart primary 0G -0G
Rafael on :
After a lot of problems trying to create a new table partition on it, nor GParted, KDE partitioner or Yast partitioner were able to create and make partitions to my USB pendrive. However, after trying your "manual" method, I have yet a full operative USB pendrive.
Thank you
Whitequill on :
I guess parted has a hard time with partitions that are 8MiB being partitioned as fat16
Artem on :
parted -s -a optimal /dev/xvdb mklabel gpt -- unit s mkpart /mnt/data xfs 0% 100%
Jari Turkia on :
H0lySh1t on :
It works as it supposed to, the only stupid people are those who never read the mostly dead engineer's manuals.
Still you spare many hours for casual humans.
Real question would be: "How am I this stupid and want it right now?"
Jari Turkia on :
H0lySh1t on :
It's almost like questioning the existence of the TCP but without it the blame wouldn't even exist at all.
So I'm just saying your article is nice except the blaming part, that I don't share, nor understand.
Jari Turkia on :