5th anniversary of buggy system/popepn/popen2

[mirek]

Hi,

1. Maybe I'm missing some older issues
2. Sorry for the not polite subject, but nobody found a solution.
   This problem has long history and so many other sotware has flawles
   popen :(

I've prepared a test script that exploits race condition with higher
probability, yes it's random. Latest test in this message gives some
suggestions.

I'll suprised if one can run (less then 10 minutes) my popen_test (attached
at the end) on recent x86 machine with final result:

report =

  21   1   2   3   4
   1   0   0   0   0
   2   0   0   0   0


Short description (rest in code):

- these zeros means no failure (max number can be equal left-upper value)

- two rows number 1 and 2 means two variants of commands "cmd_t" (pipelines)
  executed:
  1) dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d
  2) dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d;sleep 1

- four colums (1-4) for methods of reading (opening) pipeline
  1) f=popen(cmd_t,'r');
  2) f=popen(cmd_t,'r'); sleep(1)
  3) [in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); fclose(in)
  4) [in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); sleep(1);fclose(in)

------------------------------------
My sample results (debian sarge, athlon 1600xp - machine is very, very
stable).

Octave 2.1.61/57/50 & 2.0.17:
  21   1    2    3   4
   1   0-16 2-9  21  21
   2   0    0    21  21

Octave 2.1.61/57/50 & 2.0.17 with renice -19:
  21   1    2   3   4
   1  19-21 21  21  21
   2   0    0   21  21

Comment: I played  with renice because my I observed in my application it
hangs when reniced - it has only popen('gzip -dc file'). 

Seldom I observe lower error counts (but I not logged them) - but I can't
find any correlation. Machine has without octave 99.8% iddle state. All in
this test run directly from memory cache/buffers (I have 1.5 GiB RAM).

------------------------------------
The same software configuration on Pentium I no-MMX 200Mhz

Octave 2.0.17/2.1.61 (renice doesn't influence on results)
  21   1    2   3   4
   1   0-1  0  21  21
   2   0    0  21   1-21

Some difference but also horrible results.

------------------------------------
And finaly my best result achived with 2.1.61 on Athlon:

  21   1   2   3   4
   1   0   0   X   0
   2   0   0   X   0

X - means no test because octave sleeps in waiting state.

This is done by two modifications:
- my popen2 (column 4) that uses fifos not pipes
- lobotomed octave - disabled SIGCHLD handler (diff below script)

That means buggy handler - I have no idea what to do.
Maybe is worth place popen2 in cc code.

Complete test logs (to long to present here) says two types of failure:
- popen truncate semi-random number of bytes at the end less 4KB
- popen2 reads 0 or 4096 bytes.

Ah, I observe another issue: sleep and pause don't work if popen2 is started
with original sighandlers.cc and work with lobotomy.

Sorry for so long message - I've planned for today deal with performance
degradation with recent octave versions in my aplication (2.1.50: 70s,
2.1.57: 170s, 2.1.61: 270s) but it's late night here 3.30 am ...

Bye

Mirek

------------------------------------
File: popen_test.m
========================================
page_screen_output=0;

describe=1;

cmds=1:2;
opns=1:4;

r=zeros(length(cmds),length(opns));

step=100;
block_sizes=3000:step:5000;
file_size0=1e5;

cmdT=[
  'dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d';
  'dd if=/dev/zero bs=%d count=%d 2>/dev/null|gzip|gzip -d;sleep 1'
  ];

opT=[
  "f=popen(cmd_t,'r');"
  "f=popen(cmd_t,'r'); sleep(1)";
  "[in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); fclose(in);"
  "[in, f, pid] = popen2 ('/bin/bash',['-c'; cmd_t]); sleep(1);fclose(in);"
  ];
  
for o=opns; run_open=opT(o,:); for c=cmds; cmd=cmdT(c,:);

  if describe
    printf('\n');
    printf('c%d: %s\n',c,cmd);
    printf('o%d: %s\n',o,run_open);
    disp('O C   BS     MISSING     READ FEOF FCERR');
  end  

  for bs = block_sizes

    blocks = fix(file_size0/bs);
    file_size = blocks*bs;
    cmd_t=sprintf( cmd, bs, blocks);
    eval(run_open)
    
    [x count]=fread(f,inf,'int8');
 
    missing = file_size-count;

    fe=feof(f);
    ferr=ferror(f);
    fc=fclose(f);

    printf('%d %d %6d %9d %9d %2d  %2d\n', [o c bs missing count fe fc ]);
    if ~isempty(ferr);ferr=ferr,end

    r(c,o) = r(c,o)+(missing ~= 0);

  endfor
  
endfor;endfor

printf('\n');
report=[ [length(block_sizes);cmds(:)] [ opns(:)'; r] ]
========================================

$ diff -u sighandlers.cc.ORG sighandlers.cc.TEST-C
--- sighandlers.cc.ORG  2004-11-16 00:47:13.000000000 +0100
+++ sighandlers.cc.TEST-C       2004-11-15 23:08:07.000000000 +0100
@@ -51,6 +51,10 @@
 #include "toplev.h"
 #include "utils.h"

+// LD_PROFILE patch
+#undef SIGPROF
+#undef SIGCHLD
+
 // Nonzero means we have already printed a message for this series of
 // SIGPIPES.  We assume that the writer will eventually give up.
 int pipe_handler_error_count = 0;
@@ -571,13 +575,13 @@
   m.assign ("BUS", SIGBUS);
 #endif

-#ifdef SIGCHLD
-  m.assign ("CHLD", SIGCHLD);
-#endif
-
-#ifdef SIGCLD
-  m.assign ("CLD", SIGCLD);
-#endif
+//#ifdef SIGCHLD
+//  m.assign ("CHLD", SIGCHLD);
+//#endif
+
+//#ifdef SIGCLD
+//  m.assign ("CLD", SIGCLD);
+//#endif

 #ifdef SIGCONT
   m.assign ("CONT", SIGCONT);
========================================



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------