Re: 5th anniversary of buggy system/popepn/popen2

[David Bateman]

Hi,

Yeah I'd love to get this problem fixed as well. The fact is an easier
way to demonstrate it is with "ls", but doesn't use exactly the same
mechanism as popen. ls uses iprocstrem and popen uses the
octave_iprocstream. The fact that the two functions fail in the same
way, using similar but slightly different functionality lead me to
believe that the problem was in the underlying gnu libraries. It would
be interesting to see what happens under the intel compilers or the
very very latest gnu compilers and see if the problem is solved...

Regards
David


According to address@hidden <address@hidden> (on 11/16/04):
> Hi,
> 
> 1. Maybe I'm missing some older issues
> 2. Sorry for the not polite subject, but nobody found a solution.
>    This problem has long history and so many other sotware has flawles
>    popen :(
> 
> I've prepared a test script that exploits race condition with higher
> probability, yes it's random. Latest test in this message gives some
> suggestions.
> 
> I'll suprised if one can run (less then 10 minutes) my popen_test (attached
> at the end) on recent x86 machine with final result:
> 
> report =
> 
>   21   1   2   3   4
>    1   0   0   0   0
>    2   0   0   0   0
> 
> 
> Short description (rest in code):
> 
> - these zeros means no failure (max number can be equal left-upper value)
> 
> - two rows number 1 and 2 means two variants of commands "cmd_t" (pipelines)
>   executed:
>   1) dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d
>   2) dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d;sleep 1
> 
> - four colums (1-4) for methods of reading (opening) pipeline
>   1) f=popen(cmd_t,'r');
>   2) f=popen(cmd_t,'r'); sleep(1)
>   3) [in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); fclose(in)
>   4) [in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); sleep(1);fclose(in)
> 
> ------------------------------------
> My sample results (debian sarge, athlon 1600xp - machine is very, very
> stable).
> 
> Octave 2.1.61/57/50 & 2.0.17:
>   21   1    2    3   4
>    1   0-16 2-9  21  21
>    2   0    0    21  21
> 
> Octave 2.1.61/57/50 & 2.0.17 with renice -19:
>   21   1    2   3   4
>    1  19-21 21  21  21
>    2   0    0   21  21
> 
> Comment: I played  with renice because my I observed in my application it
> hangs when reniced - it has only popen('gzip -dc file'). 
> 
> Seldom I observe lower error counts (but I not logged them) - but I can't
> find any correlation. Machine has without octave 99.8% iddle state. All in
> this test run directly from memory cache/buffers (I have 1.5 GiB RAM).
> 
> ------------------------------------
> The same software configuration on Pentium I no-MMX 200Mhz
> 
> Octave 2.0.17/2.1.61 (renice doesn't influence on results)
>   21   1    2   3   4
>    1   0-1  0  21  21
>    2   0    0  21   1-21
> 
> Some difference but also horrible results.
> 
> ------------------------------------
> And finaly my best result achived with 2.1.61 on Athlon:
> 
>   21   1   2   3   4
>    1   0   0   X   0
>    2   0   0   X   0
> 
> X - means no test because octave sleeps in waiting state.
> 
> This is done by two modifications:
> - my popen2 (column 4) that uses fifos not pipes
> - lobotomed octave - disabled SIGCHLD handler (diff below script)
> 
> That means buggy handler - I have no idea what to do.
> Maybe is worth place popen2 in cc code.
> 
> Complete test logs (to long to present here) says two types of failure:
> - popen truncate semi-random number of bytes at the end less 4KB
> - popen2 reads 0 or 4096 bytes.
> 
> Ah, I observe another issue: sleep and pause don't work if popen2 is started
> with original sighandlers.cc and work with lobotomy.
> 
> Sorry for so long message - I've planned for today deal with performance
> degradation with recent octave versions in my aplication (2.1.50: 70s,
> 2.1.57: 170s, 2.1.61: 270s) but it's late night here 3.30 am ...
> 
> Bye
> 
> Mirek
> 
> ------------------------------------
> File: popen_test.m
> ========================================
> page_screen_output=0;
> 
> describe=1;
> 
> cmds=1:2;
> opns=1:4;
> 
> r=zeros(length(cmds),length(opns));
> 
> step=100;
> block_sizes=3000:step:5000;
> file_size0=1e5;
> 
> cmdT=[
>   'dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d';
>   'dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d;sleep 1'
>   ];
> 
> opT=[
>   "f=popen(cmd_t,'r');"
>   "f=popen(cmd_t,'r'); sleep(1)";
>   "[in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); fclose(in);"
>   "[in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); sleep(1);fclose(in);"
>   ];
>   
> for o=opns; run_open=opT(o,:); for c=cmds; cmd=cmdT(c,:);
> 
>   if describe
>     printf('\n');
>     printf('c%d: %s\n',c,cmd);
>     printf('o%d: %s\n',o,run_open);
>     disp('O C   BS     MISSING     READ FEOF FCERR');
>   end  
> 
>   for bs = block_sizes
> 
>     blocks = fix(file_size0/bs);
>     file_size = blocks*bs;
>     cmd_t=sprintf( cmd, bs, blocks);
>     eval(run_open)
>     
>     [x count]=fread(f,inf,'int8');
>  
>     missing = file_size-count;
> 
>     fe=feof(f);
>     ferr=ferror(f);
>     fc=fclose(f);
> 
>     printf('%d %d %6d %9d %9d %2d  %2d\n', [o c bs missing count fe fc ]);
>     if ~isempty(ferr);ferr=ferr,end
> 
>     r(c,o) = r(c,o)+(missing ~= 0);
> 
>   endfor
>   
> endfor;endfor
> 
> printf('\n');
> report=[ [length(block_sizes);cmds(:)] [ opns(:)'; r] ]
> ========================================
> 
> $ diff -u sighandlers.cc.ORG sighandlers.cc.TEST-C
> --- sighandlers.cc.ORG  2004-11-16 00:47:13.000000000 +0100
> +++ sighandlers.cc.TEST-C       2004-11-15 23:08:07.000000000 +0100
> @@ -51,6 +51,10 @@
>  #include "toplev.h"
>  #include "utils.h"
> 
> +// LD_PROFILE patch
> +#undef SIGPROF
> +#undef SIGCHLD
> +
>  // Nonzero means we have already printed a message for this series of
>  // SIGPIPES.  We assume that the writer will eventually give up.
>  int pipe_handler_error_count = 0;
> @@ -571,13 +575,13 @@
>    m.assign ("BUS", SIGBUS);
>  #endif
> 
> -#ifdef SIGCHLD
> -  m.assign ("CHLD", SIGCHLD);
> -#endif
> -
> -#ifdef SIGCLD
> -  m.assign ("CLD", SIGCLD);
> -#endif
> +//#ifdef SIGCHLD
> +//  m.assign ("CHLD", SIGCHLD);
> +//#endif
> +
> +//#ifdef SIGCLD
> +//  m.assign ("CLD", SIGCLD);
> +//#endif
> 
>  #ifdef SIGCONT
>    m.assign ("CONT", SIGCONT);
> ========================================
> 
> 
> 
> -------------------------------------------------------------
> Octave is freely available under the terms of the GNU GPL.
> 
> Octave's home on the web:  http://www.octave.org
> How to fund new projects:  http://www.octave.org/funding.html
> Subscription information:  http://www.octave.org/archive.html
> -------------------------------------------------------------

-- 
David Bateman                                address@hidden
Motorola CRM                                 +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 1 69 35 77 01 (Fax) 
91193 Gif-Sur-Yvette FRANCE

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------