Re: [Gluster-devel] mainline--2.5--patch-267: cp -ax runaway

[Krishna Srinivas]

Hi Gerry,

We were trying to see how glusterfs behaves when "cp -ax"
is started and while cp is happening, kill all the servers.
The log messages which were produced were same as the ones you
have given, So are you sure that glusterfsd processes were not dead
when cp was happening? ( Connection refused would indicate
that glusterfsd is not listening at that port)

We are checking how the log messages can be avoided
when this particular situation is encountered.

Thanks
Krishna

On 7/4/07, Gerry Reno <address@hidden> wrote:
> Actions:
>
> address@hidden mnt]# cd glusterfs2
> address@hidden glusterfs2]# ls
> dir1  file2  file3  file333  local
> address@hidden glusterfs2]# rm -rf local
> address@hidden glusterfs2]#
> address@hidden glusterfs2]#
> address@hidden glusterfs2]#
> address@hidden glusterfs2]# find /mnt/glusterfs*
> /mnt/glusterfs0
> /mnt/glusterfs0/dir1
> /mnt/glusterfs0/dir1/file0
> /mnt/glusterfs0/file3
> /mnt/glusterfs0/file333
> /mnt/glusterfs0/file2
> /mnt/glusterfs1
> /mnt/glusterfs1/dir1
> /mnt/glusterfs1/dir1/file0
> /mnt/glusterfs1/file3
> /mnt/glusterfs1/file333
> /mnt/glusterfs1/file2
> /mnt/glusterfs2
> /mnt/glusterfs2/dir1
> /mnt/glusterfs2/dir1/file0
> /mnt/glusterfs2/file3
> /mnt/glusterfs2/file333
> /mnt/glusterfs2/file2
> /mnt/glusterfs3
> /mnt/glusterfs3/dir1
> /mnt/glusterfs3/dir1/file0
> /mnt/glusterfs3/file3
> /mnt/glusterfs3/file333
> /mnt/glusterfs3/file2
> address@hidden glusterfs2]# cd ..
>
> address@hidden mnt]# cp -ax /usr /mnt/glusterfs0
> cp: cannot create directory `/mnt/glusterfs0/usr': Device or resource busy
> address@hidden mnt]#
>
> In the log I see this:
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
>
> repeated tens of thousands of times
>
> The log is growing rapidly.  It is now 3G.
> Log attached.
>
>
> About ten minutes later I retried the same command without doing
> anything to the servers or clients and it is running:
> address@hidden mnt]# cp -ax /usr /mnt/glusterfs0
>
>
> This behavior is very strange.
>
>
>
> Gerry