[Gluster-devel] Can't delete or add files when a node fails.

[Brian Hirt]

I am trying to track down a problem I reported on the list last week and 
discovered a new problem during my testing.

If you have a four node setup with replicate/distribute and one of the nodes 
has a filesystem failure, the operating system will typically remount the 
filesystem read only.   When this happens, the glusterfsd is still running on 
the failed machine, but i doesn't seem to recognize that there is a problem.   
If you try to create new files from a client and do an ls you will see that 
some of the files don't appear.  Conversely if you remove files from the client 
they will still be there along with their content. 

This is trivial to reproduce by remounting the filesystem readonly on one of 
the bricks.   If you are on a typical linux install and the gluster export 
directory is part of the root filesystem, you would only need to 'mount -o 
remount,abort /'

Considering that this is a very typical path for failure, I would expect 
gluster to handle this properly.

Regards,

Brian Hirt