VxVM Veritas Unstartable Volume

Veritas Unstartable Volume
In this example of VXVM 4.0 on a Solaris 8 system, an array was temporarily unavailable, causing problems with a file system whose two plexes resided on the array.

bash-2.03# cd /files04
bash: cd: /files04: I/O error

The volume was in DISABLED ACTIVE state, and both plexes were in DISABLED RECOVER state.

v vol04 - DISABLED ACTIVE 29360128 SELECT - fsgen
pl vol04-01 vol04 DISABLED RECOVER 29367434 STRIPE 3/128 RW
sd appsdg01-04 vol04-01 cs_array07-f0 8392167 2797389 0/0 c1t0d0 ENA
sd appsdg07-01 vol04-01 cs_array03-f2 0 5594778 0/2797389 c4t2d0 ENA
sd appsdg07-04 vol04-01 cs_array03-f2 11189556 1396899 0/8392167 c4t2d0 ENA
sd appsdg02-04 vol04-01 cs_array07-f1 8392167 2797389 1/0 c1t1d0 ENA
sd appsdg10-02 vol04-01 cs_array06-f1 2797389 5594778 1/2797389 c5t1d0 ENA
sd appsdg10-05 vol04-01 cs_array06-f1 13986945 1396899 1/8392167 c5t1d0 ENA
sd appsdg03-04 vol04-01 cs_array07-f2 8392167 2797389 2/0 c1t2d0 ENA
sd appsdg11-02 vol04-01 cs_array06-f2 8392167 6991677 2/2797389 c5t2d0 ENA
pl vol04-02 vol04 DISABLED RECOVER 29367434 STRIPE 3/128 RW
sd appsdg04-02 vol04-02 cs_array07-f3 2797389 2797389 0/0 c1t3d0 ENA
sd appsdg04-05 vol04-02 cs_array07-f3 0 2797389 0/2797389 c1t3d0 ENA
sd appsdg04-06 vol04-02 cs_array07-f3 16784334 894159 0/5594778 c1t3d0 ENA
sd appsdg14-02 vol04-02 cs_array07-f6 12586455 3300129 0/6488937 c1t6d0 ENA
sd appsdg12-03 vol04-02 cs_array06-f3 5594778 2797389 1/0 c5t3d0 ENA
sd appsdg13-02 vol04-02 cs_array07-f4 12586455 5092038 1/2797389 c1t4d0 ENA
sd appsdg12-02 vol04-02 cs_array06-f3 16784334 894159 1/7889427 c5t3d0 ENA
sd appsdg05-02 vol04-02 cs_array03-f0 12586455 1005480 1/8783586 c4t0d0 ENA
sd appsdg09-02 vol04-02 cs_array06-f0 2797389 8392167 2/0 c5t0d0 ENA
sd appsdg09-06 vol04-02 cs_array06-f0 3591 1396899 2/8392167 c5t0d0 ENA

We confirmed that the storage array was available to the operating system.

# luxadm probe
Found Enclosure(s):
...
SENA Name:cs_array06 Node WWN:5080020000038ba8
Logical Path:/dev/es/ses6
Logical Path:/dev/es/ses7

# luxadm display cs_array06

SENA
DISK STATUS
SLOT FRONT DISKS (Node WWN) REAR DISKS (Node WWN)
0 On (O.K.) 2000002037094289 On (O.K.) 200000203709422e
1 On (O.K.) 2000002037093aaf On (O.K.) 2000002037094220
2 On (O.K.) 200000203709410b On (O.K.) 2000002037093ddd
3 On (O.K.) 2000002037094254 On (O.K.) 200000203709422b
4 On (O.K.) 20000020370940da On (O.K.) 2000002037094247
5 Not Installed Not Installed
6 On (O.K.) 2000002037093df0 On (O.K.) 200000203709383f

Next, we reattached the disks to the disk group they were in. You may want to run vxreattach -c diskname to check if a reattach is possible before attempting to reattach the disks.

# vxdisk list
...

- - cs_array06-f0 appsdg failed was:c5t0d0s2
- - cs_array06-f1 appsdg failed was:c5t1d0s2
- - cs_array06-f2 appsdg failed was:c5t2d0s2
- - cs_array06-f3 appsdg failed was:c5t3d0s2
- - cs_array06-r4 appsdg failed spare was:c5t20d0s2
- - cs_array06-f4 appsdg failed was:c5t4d0s2

# cd /usr/lib/vxvm/bin
# ./vxreattach c5t0d0s2
# ./vxreattach c5t1d0s2
# ./vxreattach c5t2d0s2
# ./vxreattach c5t3d0s2
# ./vxreattach c5t20d0s2
# ./vxreattach c5t4d0s2

We then followed the "Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State" procedure in the Volume Manager Troubleshooting Guide.

1. Force plex vol04-01 into the OFFLINE state.
# vxmend -g appsdg -o force off vol04-01

2. Place plex vol04-01 into the STALE state.
# vxmend -g appsdg on vol04-01

3. There are no other clean plexes in the volume, so make plex vol04-01 DISABLED and CLEAN.
# vxmend -g appsdg fix clean vol04-01

4. Start the volume, and perform resynchronization of the plexes in the background.
# vxvol -g appsdg -o bg start vol04

At this point, the file system is unmounted, checked for file system consistency, and remounted.

# umount /files04

# mount /files04
UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/appsdg/vol04 is corrupted. needs checking

# fsck -F vxfs /dev/vx/rdsk/appsdg/vol04
log replay in progress
replay complete - marking super-block as CLEAN

# mount /files04