Today, I hit one of the weirdest problems on a server I've seen so far. I'm putting a replicated Gluster storage setup into production.
After some initial problems and an upgrade to Gluster 3.2.2 (which you should really do if you are running Gluster), everything seemed to be running smoothly.
Until I noticed that the contents of one directory started showing up in another directory. Clearing the second one seemed to help only to get the files back minutes later. It was a mistery.
So, I het the #gluster chatroom on Freenode and one of the guys in there asked me to check the extended attributes of the directories on the servers. The command ' getfattr -d -m . dev", "sudo getfattr -d -m . <dir>' showed me that trusted.gfid values where identical! Removing one directory from the filesystem below gluster on both nodes and recreating it fixed the problem. Apparently the cause was a cp -a I did on Gluster 3.1 which copied the gfid (a bug that is fixed in 3.2)...
Fun, those clustered filesystems :-)