Setup GEOM Mirror (gmirror)

Preface

First, this process is documented all over the internet and in some cases very poorly.  One of those poorly written cases just bit me so this write-up is really more for myself than public use.  But, it's placed here in case you find it useful or you want a second opinion.

The biggest assumption made here is both disks to be mirrored are identical.

Pre-Configuration

The first thing to determine is the names FreeBSD has assigned to each of the drives, we need to know the name of our boot drive and additional drive.

Figuring out the name of the boot drive is simple, just 'cat' the /etc/fstab file:

cat /etc/fstab

The following is example output from my system:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad4s1b             none            swap    sw              0       0
/dev/ad4s1a             /               ufs     rw              1       1
/dev/ad4s1e             /tmp            ufs     rw              2       2
/dev/ad4s1f             /usr            ufs     rw              2       2
/dev/ad4s1d             /var            ufs     rw              2       2

So I know the name of the boot drive is 'ad4' in this instance. 

Next I need to determine the name of the 2nd drive, so execute 'atacontrol':

atacontrol list

The following is example output from my system:

ATA channel 0:
    Master:      no device present
    Slave:       no device present
ATA channel 1:
    Master:      no device present
    Slave:       no device present
ATA channel 2:
    Master:  ad4 <WDC WD5001ABYS-01YNA0/59.01D01> SATA revision 2.x
    Slave:       no device present
ATA channel 3:
    Master:  ad6 <WDC WD5001ABYS-01YNA0/59.01D01> SATA revision 2.x
    Slave:       no device present
ATA channel 4:
    Master:      no device present
    Slave:       no device present
ATA channel 5:
    Master:      no device present
    Slave:       no device present 

So I now can see the name of the second device is 'ad6'.

Configuration

 I can now configure the gmirror device to use the two drives from above.

The first step is to allow geom to make changes to the boot drive while it's already mounted by executing the following:

sysctl kern.geom.debugflags=16

This kernel change will automatically return to it's original value upon reboot.

Now I need to create the gmirror device by adding the meta-data to the boot drive:

gmirror label -v -b load gm0 /dev/ad4

Choose which ever balance (-b) method you prefer, everyone has their reasons for what they select and the performance numbers.

The next step to is to update the /boot/loader.conf.  Using "vi" I'll add the following line if it doesn't already exist:

geom_mirror_load="YES"

The final step, before rebooting, requires changing the /etc/fstab file.  Personally I make a backup copy just in case something goes wrong:

cd /etc
mv fstab fstab.ad4
cp fstab.ad4 fstab

You can see the original fstab file above.  I now change all instances of "ad4" to "mirror/gm0" so the file looks like this:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/mirror/gm0s1b      none            swap    sw              0       0
/dev/mirror/gm0s1a      /               ufs     rw              1       1
/dev/mirror/gm0s1e      /tmp            ufs     rw              2       2
/dev/mirror/gm0s1f      /usr            ufs     rw              2       2
/dev/mirror/gm0s1d      /var            ufs     rw              2       2

I then perform a double check of the command line history and verify the /etc/fstab file is correct.  One mistake and the host will require manual intervention at the console to resolve any problems.

Once I'm confident everything has been performed correctly, reboot. 

Hopefully the host reboots properly and upon login I can execute "df" and see the new gmirror device being used:

[root@saturn ~]# df
Filesystem          1K-blocks    Used     Avail Capacity  Mounted on
/dev/mirror/gm0s1a     507630  423890     43130    91%    /
devfs                       1       1         0   100%    /dev
/dev/mirror/gm0s1e     507630  397752     69268    85%    /tmp
/dev/mirror/gm0s1f  465130280 5478370 422441488     1%    /usr
/dev/mirror/gm0s1d    2962062  124988   2600110     5%    /var

Adding The Mirror Drive

Now that I have a working gmirror device I can add the second device (ad6) discovered earlier.  The first step is actually to clear any existing partition or boot-code from the drive.  Since I swap drives around I never assume they are clean and this step only takes a second to execute:

dd if=/dev/zero of=/dev/ad6 bs=1k count=100

Now the drive is ready to be added, just execute the following command:

gmirror insert gm0 /dev/ad6

Monitor Mirroring

The mirror is created but now GEOM has to copy the data from the original good drive (ad4) to the newly added drive (ad6) which depending upon the size and speed can literally take hours.  I can monitor the progress by executing "gmirror status":

[root@saturn ~]# gmirror status
      Name    Status  Components
mirror/gm0  DEGRADED  ad4
                      ad6 (12%)

After almost 2 1/2 hours the mirroring to the ad6 drive was finally completed:

[root@saturn ~]# gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ad4
                      ad6

Conclusion

You should now have two drivers configured in a GEOM (gmirror) based RAID-1 configuration. 

There are fewer web pages regarding how to recover from a failure, but they do exist.  The first time I need to perform such an action I'll add that documentation to the site.  But, I've been lucky so far to have not lost a hard drive on the three systems running this configuration.