High system load, low CPU and RAM usage. What’s going on?

I’ve just re-installed and re-configured my home fileserver as the old install was in need of a refresh. Several ‘quick hacks fixes’ meant I wasn’t happy with the setup and knew I could do it better. I’ve also learnt a lot since setting it up two years ago and wanted to do everything properly this time round.

I made the change from ZFS-FUSE to ZFS-on-Linux partly just to see how the different approach to ZFS compares and so I could use ZVOL volumes with iSCSI for virtual guests.

After the reinstall I noticed that my server had a high system load when I was copying data back to it. High as in 15+! The CPU and RAM usage was low and showed little activity. iostat showed that data reads and writes were quite low.

Some searching showed me that I needed to use sar to check my drives. If it’s not already on your system you need to install the sysstat package. I enabled and started sar and waited for some data to be collected (it’s not instant). After a few hours I used the ‘sar -d’ command to show me some disk stats:


root@vhost:/var/log/sysstat# sar -d
Linux 2.6.38-8-server (vhost) 28/06/11 _x86_64_ (2 CPU)

17:36:00 LINUX RESTART

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
Average: dev8-16 0.65 0.06 6.63 10.30 0.01 13.34 12.56 0.82
Average: dev8-0 38.71 2.98 4812.72 124.41 0.35 8.94 3.62 13.99
Average: dev8-32 0.65 0.03 6.63 10.30 0.01 15.55 14.32 0.93
Average: dev8-48 40.45 2.96 4813.06 119.07 0.56 13.77 3.99 16.12
Average: dev8-64 13.56 27.49 1069.82 80.93 2.87 211.46 68.78 93.25
Average: dev9-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev9-1 0.85 0.09 6.24 7.48 0.00 0.00 0.00 0.00
Average: dev251-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: dev251-1 0.04 0.09 0.29 10.00 0.00 35.00 15.11 0.06
Average: dev251-2 0.07 0.00 0.52 8.00 0.00 21.99 10.00 0.07
Average: dev251-3 0.68 0.00 5.43 8.00 0.01 17.26 11.11 0.75

The bit I’m interested in was the averages section at the bottom, specifically the %util column. This shows that dev8-64 is, on average, in use 93% of the time. Because it’s nearly at full capacity it might explain why I had high system loads. This device is causing a queue to form.

I needed to find out which exact device dev8-64 was. All I had to do was look for the corresponding numbers in /proc/diskstats:


root@vhost:/# cat /proc/diskstats
1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
1 2 ram2 0 0 0 0 0 0 0 0 0 0 0
1 3 ram3 0 0 0 0 0 0 0 0 0 0 0
1 4 ram4 0 0 0 0 0 0 0 0 0 0 0
1 5 ram5 0 0 0 0 0 0 0 0 0 0 0
1 6 ram6 0 0 0 0 0 0 0 0 0 0 0
1 7 ram7 0 0 0 0 0 0 0 0 0 0 0
1 8 ram8 0 0 0 0 0 0 0 0 0 0 0
1 9 ram9 0 0 0 0 0 0 0 0 0 0 0
1 10 ram10 0 0 0 0 0 0 0 0 0 0 0
1 11 ram11 0 0 0 0 0 0 0 0 0 0 0
1 12 ram12 0 0 0 0 0 0 0 0 0 0 0
1 13 ram13 0 0 0 0 0 0 0 0 0 0 0
1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
7 2 loop2 0 0 0 0 0 0 0 0 0 0 0
7 3 loop3 0 0 0 0 0 0 0 0 0 0 0
7 4 loop4 0 0 0 0 0 0 0 0 0 0 0
7 5 loop5 0 0 0 0 0 0 0 0 0 0 0
7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
8 16 sdb 3003 2538 182874 89990 18182 14195 213086 306750 0 267500 396730
8 17 sdb1 130 521 2720 6140 1980 4 78 54410 0 59560 60550
8 18 sdb2 2820 1851 178402 80630 16200 14191 213008 252280 0 215220 332900
8 0 sda 26815 31523 13645033 224160 562106 155798 80700699 5615990 0 2966510 5839980
8 1 sda1 26620 28832 13640509 223670 562100 155798 80700698 5615960 0 2966150 5839460
8 9 sda9 19 0 152 20 0 0 0 0 0 20 20
8 32 sdc 2981 3058 167774 89820 18040 14337 213086 341810 0 299090 431630
8 33 sdc1 122 443 2526 6200 1979 5 78 110070 0 115240 116270
8 34 sdc2 2807 2448 163496 80480 16059 14332 213008 231680 0 193050 312160
8 48 sdd 48443 9400 13680566 866870 584715 131571 80700698 8202480 0 3230750 9068950
8 49 sdd1 48308 7215 13676769 866280 584715 131571 80700698 8202480 0 3230490 9068360
8 57 sdd9 65 226 1768 410 0 0 0 0 0 410 410
8 64 sde 925 451 115592 1557230 144973 5169 16824587 16876100 1 6962060 18433460
8 65 sde1 48 93 1368 520 86392 466 2819424 7021170 1 5682130 7022050
8 66 sde2 753 155 111608 1556630 58580 4703 14005155 9854920 0 2981230 11411320
9 0 md0 797 0 2158 0 23 0 54 0 0 0 0
9 1 md1 9556 0 338854 0 27322 0 202968 0 0 0 0
251 0 dm-0 263 0 2104 6690 0 0 0 0 0 3680 6690
251 1 dm-1 6901 0 267002 290730 2081 0 16272 140080 0 55190 430810
251 2 dm-2 342 0 2706 9710 1732 0 13816 22690 0 16970 32400
251 3 dm-3 1815 0 65162 35500 21636 0 172880 434700 0 216130 470190

As you can see dev8-64 is actually /dev/sde. In my case this was a 2GB Compact Flash card I was using as a cache device for my ZFS filesystem. I’d used it to help improve reliability on my old heavily deduped filesystem without buying new RAM but since my new setup used deduplication less I could safely remove it now. Proper SSD drives would be better as would using more devices and mirroring them. They all involve extra spending though which is what I’m trying to avoid as it’s only a home server built for convenience.

Ubuntu, ZFS and missing devices

I’ve used ZFS in some form or other for a few years now, starting off with the OpenSolaris based Nexenta and moving over to Ubuntu Server as ZFS matured on Linux.

Recently I’d added a couple of new drives to the pool and all was good until after a reboot. After logging in to the server I could see the pool hadn’t mounted so I attempted it manually.

It was telling me the new drives I’d added were corrupt and the pool was offline. “No Problem” I though as I could just restore from backup and re-create the pool. The only problem with that is that my last backup is 6 months old*

I then noticed that the disk assignment was wrong. My ZFS array was looking for /dev/sdb and /dev/sdc when the new disks were now at /dev/sdf and /dev/sdg.

I then remembered using /dev/disk/by-id when I initially created the pool to avoid this issue but when adding the new drives I never gave it a thought.

To fix this without losing any data I did the following:

1) Rename your zpool.cache file
mv /var/lib/zfs/zpool.cache /var/lib/zfs/zpool.cache_old

2) Now import your pool again
zpool import -d /dev/disk/by-id/ poolname
It should now look through all of your disks and mount the pool as normal with each of the disks added by id.

Taadaa! Pool mounted and a full backup has now started.

*Yes, I know. How stupid of me… ­čśŤ