VxVM Recovering a Deleted Volume

Recovering a Deleted Volume

While debugging a C program on a Friday afternoon (around 6pm), one of my colleagues came to my desk and told me he had accidentally destroyed the production disk group instead of the test disk group he thought he was working on. A quick check of the Veritas configuration verified this:

$ vxprint -hft $

Since we used vxconfigbackup(1m) to take regular backups of the disk group configuration, we located the last known good working backup and passed this location to vxconfigrestore(1m)'s "-l" (location) option:

$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/oradg_0612205 oradg

Diskgroup oradg configuration restoration started ......

Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...

oradg's diskgroup configuration is restored (in precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.

Run:
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -d oradg ==> to abort the restoration.

We verified the configuration was consistent by running vxprint(1m), and committed the configuration to the disk headers and configuration database by invoking vxconfigrestore(1m) with the "-c" option:

$ /usr/lib/vxvm/bin/vxconfigrestore -c -l /var/tmp/back oradg

Committing configuration restoration for diskgroup oradg ....

oradg's diskgroup configuration restoration is committed.

Once we committed the configuration, the layout matched our last known good configuration backup:

$ vxprint -ht

Disk group: oradg

DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg oradg default default 10000 1127240283.19.winnie

v oravol01 - ENABLED ACTIVE 41943040 SELECT oravol01-03 fsgen
pl oravol01-03 oravol01 ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0 2/2 ENA
sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0 2/2 ENA
sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0 2/2 ENA

v oravol01-L01 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P01 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t1d0-02 oravol01-P01 c1t1d0 0 13981056 0 c1t1d0 ENA
pl oravol01-P02 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t4d0-02 oravol01-P02 c1t4d0 0 13981056 0 c1t4d0 ENA

v oravol01-L02 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P03 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t2d0-02 oravol01-P03 c1t2d0 0 13981056 0 c1t2d0 ENA
pl oravol01-P04 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t5d0-02 oravol01-P04 c1t5d0 0 13981056 0 c1t5d0 ENA

v oravol01-L03 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t3d0-02 oravol01-P05 c1t3d0 0 13981056 0 c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t6d0-02 oravol01-P06 c1t6d0 0 13981056 0 c1t6d0 ENA

To ensure that no damage had occurred to the file system, we used fsck(1m) to consistency check the file system:

$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01

and got our Oracle DBA to verify the integrity of each data file with Oracle's database verification utilities. Because accidents and disasters can strike at any time, it is important to perform regular backups of the Veritas configuration and store this data in a safe location (having a configuration database backup saved us from having to restore the database or re-create the Veritas objects by hand).