During the server upgrades a few days ago I switched to kernel 2.6.9. This has been rolled back to 2.6.5...
I woke up to find the drive that stores all of the primary data gone... nothing could read any of the data (all 2.6.9 machines). After a bunch of hopeless testing, I tried booting back into 2.6.5 and the drive worked fine.
I have concluded that there is some incompatibility in 2.6.9. Oh well. Just glad it works now.