Friday 27 June 2008

Munge columnar into one datafile (for processing in a spreadsheet, gnuplot or whatever)

Often we'll use awk or cut to spit out a single column of data from a bunch of different files, or maybe from the same file... anyhow once we have these files with one column of data, sometimes we'll want to splice those files together so we can look at each line (typically a stat of some sort) next to each other. Also it's very convenient if we want to import the data into a spreadsheet.

Anyhow, let's say I have three files iscsi_080626.txt greads.out.txt idle.out.txt. I can put them together into one file using the 'pr' command like so.

iscsi file
bash-3.00$ head iscsi_080626.txt 
32782319.86
32793022.93
32734064.23
32719233.74
32652552.75
32815570.70


gread file
bash-3.00$ head greads.out.txt

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00


idle file
bash-3.00$ head idle.out.txt 
248.722142
17.662915
17.195024
16.208969
15.856432
17.064519
16.874948
17.122956
17.191305
16.920703





bash-3.00$ pr -m -t iscsi_080626.txt greads.out.txt idle.out.txt > iscsi_greads_idle.txt


And the output file looke like this
bash-3.00$ head iscsi_greads_idle.txt 
32782319.86 0.00 248.722142
32793022.93 0.00 17.662915
32734064.23 0.00 17.195024
32719233.74 0.00 16.208969
32652552.75 0.00 15.856432
32815570.70 0.00 17.064519
32801494.63 0.00 16.874948
32696942.10 0.00 17.122956
32715608.64 0.00 17.191305
32835538.38 0.00 16.920703

Change disks used by an aggregate

Recently we needed to transfer an aggregate that was hosted on an external shelf to disks housed internally in our test FAS2020 and the same trick can be used to move an aggregate from one shelf to another.

The trick uses the disk-replace command. It works serially, and so it is a little time consuming, BUT it retains exactly the same layout on disk, since it's a bit-for-bit copy.

In short the command to use is 'disk replace start < new diskname> e.g.
 disk replace start 0b.19 0c.00.0

So here's what my disk layout looks like - I am using traditional volumes
Volume vol1 (online, raid_dp) (block checksums)
Plex /vol1/plex0 (online, normal, active)
RAID group /vol1/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0c.00.6 0c 0 6 SA:1 - SATA 7200 211377/432901760 211977/434130816
parity 0c.00.1 0c 0 1 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.19 0b 1 3 FC:B - ATA 7200 211377/432901760 211921/434014304
data 0c.00.3 0c 0 3 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.11 0c 0 11 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.10 0c 0 10 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.9 0c 0 9 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.8 0c 0 8 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.7 0c 0 7 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.4 0c 0 4 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.29 0b 1 13 FC:B - ATA 7200 211377/432901760 211921/434014304
data 0c.00.5 0c 0 5 SA:1 - SATA 7200 211377/432901760 211977/434130816


Spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
Spare disks for block or zoned checksum traditional volumes or aggregates
spare 0a.16 0a 1 0 FC:A - ATA 7200 211377/432901760 211921/434014304 (not zeroed)
spare 0a.20 0a 1 4 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.22 0a 1 6 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.24 0a 1 8 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.26 0a 1 10 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0b.17 0b 1 1 FC:B - ATA 7200 211377/432901760 211921/434014304 (not zeroed)
spare 0b.25 0b 1 9 FC:B - ATA 7200 211377/432901760 211921/434014304
spare 0b.27 0b 1 11 FC:B - ATA 7200 211377/432901760 211921/434014304
spare 0c.00.0 0c 0 0 SA:1 - SATA 7200 211377/432901760 211977/434130816 (not zeroed)

All the disks marked with a 'CHAN' type of FC:A or FC:B are actually SATA drives on the other side of a FC->SATA bridge, and are basically SATA disks in an external shelf. The disks marked SA:1 are SATA drive internal to the FAS2020. In the output above I am part-way through migrating the disks. Some of the disks in my volume/aggregate are internal and some are external. On of the disks that is still on the external shelf is disk 0b.19. Using the command below I will transfer the data on that individual disk to an internal SATA disk (0c.00.0). The command I use is.
 disk replace start 0b.19 0c.00.0

Then I receive a warning messge
*** You are about to copy and replace the following file system disk ***
Disk /vol1/plex0/rg0/0b.19

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
data 0b.19 0b 1 3 FC:B - ATA 7200 211377/432901760 211921/434014304
***
Really replace disk 0b.19 with 0c.00.0? y
disk replace: Disk 0b.19 was marked for replacing.

The output of sysconfig -r shows something like this
Volume vol1 (online, raid_dp) (block checksums)
Plex /vol1/plex0 (online, normal, active)
RAID group /vol1/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0c.00.6 0c 0 6 SA:1 - SATA 7200 211377/432901760 211977/434130816
parity 0c.00.1 0c 0 1 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.19 0b 1 3 FC:B - ATA 7200 211377/432901760 211921/434014304 (replacing)
data 0c.00.3 0c 0 3 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.11 0c 0 11 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.10 0c 0 10 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.9 0c 0 9 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.8 0c 0 8 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.7 0c 0 7 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.4 0c 0 4 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.29 0b 1 13 FC:B - ATA 7200 211377/432901760 211921/434014304
data 0c.00.5 0c 0 5 SA:1 - SATA 7200 211377/432901760 211977/434130816

And then once the replacement gets under way.. I see my copy under way.
Volume vol1 (online, raid_dp) (block checksums)
Plex /vol1/plex0 (online, normal, active)
RAID group /vol1/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0c.00.6 0c 0 6 SA:1 - SATA 7200 211377/432901760 211977/434130816
parity 0c.00.1 0c 0 1 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.19 0b 1 3 FC:B - ATA 7200 211377/432901760 211921/434014304 (replacing, copy in progress)
-> copy 0c.00.0 0c 0 0 SA:1 - SATA 7200 211377/432901760 211977/434130816 (copy 1% completed)
data 0c.00.3 0c 0 3 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.11 0c 0 11 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.10 0c 0 10 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.9 0c 0 9 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.8 0c 0 8 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.7 0c 0 7 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.4 0c 0 4 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0b.29 0b 1 13 FC:B - ATA 7200 211377/432901760 211921/434014304
data 0c.00.5 0c 0 5 SA:1 - SATA 7200 211377/432901760 211977/434130816

And finally when the copy is done, 0c.00.0 is part of the aggregate/volume and 0b.19 is spare.
Volume vol1 (online, raid_dp) (block checksums)
Plex /vol1/plex0 (online, normal, active)
RAID group /vol1/plex0/rg0 (normal)

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity 0c.00.6 0c 0 6 SA:1 - SATA 7200 211377/432901760 211977/434130816
parity 0c.00.1 0c 0 1 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.0 0c 0 0 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.3 0c 0 3 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.11 0c 0 11 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.10 0c 0 10 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.9 0c 0 9 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.8 0c 0 8 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.7 0c 0 7 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.4 0c 0 4 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.2 0c 0 2 SA:1 - SATA 7200 211377/432901760 211977/434130816
data 0c.00.5 0c 0 5 SA:1 - SATA 7200 211377/432901760 211977/434130816


Spare disks

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
Spare disks for block or zoned checksum traditional volumes or aggregates
spare 0a.16 0a 1 0 FC:A - ATA 7200 211377/432901760 211921/434014304 (not zeroed)
spare 0a.20 0a 1 4 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.22 0a 1 6 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.24 0a 1 8 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0a.26 0a 1 10 FC:A - ATA 7200 211377/432901760 211921/434014304
spare 0b.17 0b 1 1 FC:B - ATA 7200 211377/432901760 211921/434014304 (not zeroed)
spare 0b.19 0b 1 3 FC:B - ATA 7200 211377/432901760 211921/434014304 (not zeroed)
spare 0b.25 0b 1 9 FC:B - ATA 7200 211377/432901760 211921/434014304
spare 0b.27 0b 1 11 FC:B - ATA 7200 211377/432901760 211921/434014304
spare 0b.29 0b 1 13 FC:B - ATA 7200 211377/432901760 211921/434014304 (not zeroed)