Humble Frozen Synapse Bundle

Another Humble ‘Donate-what-you-want’ Bundle is available offering another cool game (Frozen in fact!)……. I’ll give you a moment let that sink in…..

If you give more than the average ($4.80ish at time of writing) then you will get all the games in the previous Humble Bundle too! Steam, Desura and Direct2Drive keys can be provided if needed.

Go!     Fetch!      Good Gamer 😀

Zombie Partitions and the GPT Undead

I’ve just had a hard drive that I tried to join to a ZFS pool. The drive had been used before when testing ZFS on Linux. However, when trying to add the drive it would complain about not finding the first partition:

cannot resolve path '/dev/disk/by-id/ata-ST3160215ACE_9RX5F4K9-part1'

fdisk showed the drive as being empty but having a GPT table.

fdisk -l /dev/disk/by-id/ata-ST3160812A_5LSBGY5S

WARNING: GPT (GUID Partition Table) detected on ‘/dev/disk/by-id/ata-ST3160812A_5LSBGY5S’! The util fdisk doesn’t support GPT. Use GNU Parted.

Fdisk can’t do GPT so I fired up parted.

root@vhost:/home/administrator# parted /dev/disk/by-id/ata-ST3160812A_5LSBGY5S
GNU Parted 2.3
Using /dev/sdg
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Warning: /dev/sdg contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should. Perhaps
it was corrupted -- possibly by a program that doesn't understand GPT partition tables. Or perhaps you deleted the GPT table, and are now using an msdos partition
table. Is this a GPT partition table?
Yes/No? Yes
Model: ATA ST3160812A (scsi)
Disk /dev/sdg: 160GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 1049kB 160GB 160GB zfs
9 160GB 160GB 8389kB

It reported that the GPT signature was damaged and appeared to fix it for me as I could now see two partitions on the drive. I removed them and set the partition table to a simple msdos one:

(parted) rm 1
(parted) rm 9
(parted) mktable msdos
Warning: The existing disk label on /dev/sdg will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
Warning: /dev/sdg contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should. Perhaps
it was corrupted -- possibly by a program that doesn't understand GPT partition tables. Or perhaps you deleted the GPT table, and are now using an msdos partition
table. Is this a GPT partition table?
Yes/No? yes

Now let’s see if it’s all gone….

(parted) p
Model: ATA ST3160812A (scsi)
Disk /dev/sdg: 160GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags

(parted) q
Information: You may need to update /etc/fstab.

After reboot I tried to create the pool again but was met with the same error I had at the start!
I then tried the following:

  • zeroing the drive with dd.
  • Checking (and wiping) the drive with badblocks.
  • Using Seagate’s own HD utility software to check for errors (Pass!)
  • Sticking the drive into another computer and adding it to an existing RAID set in the hope that it would obliterate any remaining partition info
  • Using Seagate’s own HD partitioning utility to configure the drive
  • Using Seagate’s own HD utility software to low-level format the disk
  • Draping goat entrails across the platters in an attempt to appease any gods who might know about this sort of thing
  • Watching England Ladies lose to France in the Women’s World Cup (not really affecting the HD issue but I needed to do something whilst waiting for the various scans & wipes to happen!

After each of these the GPT partitions kept coming back!

Plugging the drive into Windows didn’t help as the Disk Management utility wouldn’t let me change the GPT table either. I then remembered the Windows command line tool diskpart but had forgotten which commands I needed:

C:\Documents and Settings\Me>diskpart

Microsoft DiskPart version 5.1.3565

Copyright (C) 1999-2003 Microsoft Corporation.
On computer: PC1

DISKPART> help

Microsoft DiskPart version 5.1.3565

ADD – Add a mirror to a simple volume.
ACTIVE – Marks the current basic partition as an active boot partition.
ASSIGN – Assign a drive letter or mount point to the selected volume.
BREAK – Break a mirror set.
CLEAN – Clear the configuration information, or all information, off th
disk.
CONVERT – Converts between different disk formats.
CREATE – Create a volume or partition.
DELETE – Delete an object.
DETAIL – Provide details about an object.
EXIT – Exit DiskPart
EXTEND – Extend a volume.
HELP – Prints a list of commands.
IMPORT – Imports a disk group.
LIST – Prints out a list of objects.
INACTIVE – Marks the current basic partition as an inactive partition.
ONLINE – Online a disk that is currently marked as offline.
REM – Does nothing. Used to comment scripts.
REMOVE – Remove a drive letter or mount point assignment.
REPAIR – Repair a RAID-5 volume.
RESCAN – Rescan the computer looking for disks and volumes.
RETAIN – Place a retainer partition under a simple volume.
SELECT – Move the focus to an object.

DISKPART> list disk

Disk ### Status Size Free Dyn Gpt
——– ———- ——- ——- — —
Disk 0 Online 233 GB 0 B
Disk 1 Online 149 GB 0 B
Disk 6 Online 149 GB 0 B

DISKPART> select disk 6

Disk 6 is now the selected disk.

DISKPART> clean

DiskPart succeeded in cleaning the disk.

DISKPART> exit

Leaving DiskPart…

I then removed the drive, put it back into my Linux computer and successfully added the drive to the ZFS pool. YAY!
I’m surprised that it took my Windows machine to do something that seems so simple. It’s odd that Seagate’s own utilities couldn’t wipe the disk either and the fairly comprehensive Linux utilities dd, parted and fdisk, which normally do so well, couldn’t wipe this disk.

As with everything to do with computers it shows that there are multiple tools for the same job and some work better in some situations that others.

High system load, low CPU and RAM usage. What’s going on?

I’ve just re-installed and re-configured my home fileserver as the old install was in need of a refresh. Several ‘quick hacks fixes’ meant I wasn’t happy with the setup and knew I could do it better. I’ve also learnt a lot since setting it up two years ago and wanted to do everything properly this time round.

I made the change from ZFS-FUSE to ZFS-on-Linux partly just to see how the different approach to ZFS compares and so I could use ZVOL volumes with iSCSI for virtual guests.

After the reinstall I noticed that my server had a high system load when I was copying data back to it. High as in 15+! The CPU and RAM usage was low and showed little activity. iostat showed that data reads and writes were quite low.

Some searching showed me that I needed to use sar to check my drives. If it’s not already on your system you need to install the sysstat package. I enabled and started sar and waited for some data to be collected (it’s not instant). After a few hours I used the ‘sar -d’ command to show me some disk stats:


root@vhost:/var/log/sysstat# sar -d
Linux 2.6.38-8-server (vhost) 28/06/11 _x86_64_ (2 CPU)

17:36:00 LINUX RESTART

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: dev8-16 0.65 0.06 6.63 10.30 0.01 13.34 12.56 0.82
Average: dev8-0 38.71 2.98 4812.72 124.41 0.35 8.94 3.62 13.99
Average: dev8-32 0.65 0.03 6.63 10.30 0.01 15.55 14.32 0.93
Average: dev8-48 40.45 2.96 4813.06 119.07 0.56 13.77 3.99 16.12
Average: dev8-64 13.56 27.49 1069.82 80.93 2.87 211.46 68.78 93.25
Average: dev9-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev9-1 0.85 0.09 6.24 7.48 0.00 0.00 0.00 0.00
Average: dev251-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev251-1 0.04 0.09 0.29 10.00 0.00 35.00 15.11 0.06
Average: dev251-2 0.07 0.00 0.52 8.00 0.00 21.99 10.00 0.07
Average: dev251-3 0.68 0.00 5.43 8.00 0.01 17.26 11.11 0.75

The bit I’m interested in was the averages section at the bottom, specifically the %util column. This shows that dev8-64 is, on average, in use 93% of the time. Because it’s nearly at full capacity it might explain why I had high system loads. This device is causing a queue to form.

I needed to find out which exact device dev8-64 was. All I had to do was look for the corresponding numbers in /proc/diskstats:


root@vhost:/# cat /proc/diskstats
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
8 16 sdb 3003 2538 182874 89990 18182 14195 213086 306750 0 267500 396730
8 17 sdb1 130 521 2720 6140 1980 4 78 54410 0 59560 60550
8 18 sdb2 2820 1851 178402 80630 16200 14191 213008 252280 0 215220 332900
8 0 sda 26815 31523 13645033 224160 562106 155798 80700699 5615990 0 2966510 5839980
8 1 sda1 26620 28832 13640509 223670 562100 155798 80700698 5615960 0 2966150 5839460
8 9 sda9 19 0 152 20 0 0 0 0 0 20 20
8 32 sdc 2981 3058 167774 89820 18040 14337 213086 341810 0 299090 431630
8 33 sdc1 122 443 2526 6200 1979 5 78 110070 0 115240 116270
8 34 sdc2 2807 2448 163496 80480 16059 14332 213008 231680 0 193050 312160
8 48 sdd 48443 9400 13680566 866870 584715 131571 80700698 8202480 0 3230750 9068950
8 49 sdd1 48308 7215 13676769 866280 584715 131571 80700698 8202480 0 3230490 9068360
8 57 sdd9 65 226 1768 410 0 0 0 0 0 410 410
8 64 sde 925 451 115592 1557230 144973 5169 16824587 16876100 1 6962060 18433460
8 65 sde1 48 93 1368 520 86392 466 2819424 7021170 1 5682130 7022050
8 66 sde2 753 155 111608 1556630 58580 4703 14005155 9854920 0 2981230 11411320
9 0 md0 797 0 2158 0 23 0 54 0 0 0 0
9 1 md1 9556 0 338854 0 27322 0 202968 0 0 0 0
251 0 dm-0 263 0 2104 6690 0 0 0 0 0 3680 6690
251 1 dm-1 6901 0 267002 290730 2081 0 16272 140080 0 55190 430810
251 2 dm-2 342 0 2706 9710 1732 0 13816 22690 0 16970 32400
251 3 dm-3 1815 0 65162 35500 21636 0 172880 434700 0 216130 470190

As you can see dev8-64 is actually /dev/sde. In my case this was a 2GB Compact Flash card I was using as a cache device for my ZFS filesystem. I’d used it to help improve reliability on my old heavily deduped filesystem without buying new RAM but since my new setup used deduplication less I could safely remove it now. Proper SSD drives would be better as would using more devices and mirroring them. They all involve extra spending though which is what I’m trying to avoid as it’s only a home server built for convenience.

Ubuntu, ZFS and missing devices

I’ve used ZFS in some form or other for a few years now, starting off with the OpenSolaris based Nexenta and moving over to Ubuntu Server as ZFS matured on Linux.

Recently I’d added a couple of new drives to the pool and all was good until after a reboot. After logging in to the server I could see the pool hadn’t mounted so I attempted it manually.

It was telling me the new drives I’d added were corrupt and the pool was offline. “No Problem” I though as I could just restore from backup and re-create the pool. The only problem with that is that my last backup is 6 months old*

I then noticed that the disk assignment was wrong. My ZFS array was looking for /dev/sdb and /dev/sdc when the new disks were now at /dev/sdf and /dev/sdg.

I then remembered using /dev/disk/by-id when I initially created the pool to avoid this issue but when adding the new drives I never gave it a thought.

To fix this without losing any data I did the following:

1) Rename your zpool.cache file
mv /var/lib/zfs/zpool.cache /var/lib/zfs/zpool.cache_old

2) Now import your pool again
zpool import -d /dev/disk/by-id/ poolname
It should now look through all of your disks and mount the pool as normal with each of the disks added by id.

Taadaa! Pool mounted and a full backup has now started.

*Yes, I know. How stupid of me… 😛