OK so Friday the 13 was just that .. I got to work today and was greeted by are you working on that database volume problem. Ok not a good way to start since there was no prior knowledge of this. Turns out that all the iscsi volumes attached to 4 of our db servers were in a bad state as far as the filesystem .. this required each to be cleaned with fsck.ext3. The volumes all lost a bit of data but did come back after the check. The root issue turned out to be one of the Equallogic boxes not failing a disk when it should have. ((Isilon anyone??)) Any way it was all fixed and back together again by days end however, the notifications need to be streamlined with regards to this.
Posted by Kevin Foote in sys-admin on July 13, 2007