Re: Aw: Re: memory leak in regexprep ? (windows/3.7.2+/VS)

[Tatsuro MATSUOKA]

--- On Wed, 2013/3/27, Christoph Ellenberger wrote:
> > --- On Wed, 2013/3/27, Tatsuro MATSUOKA wrote:
> > > --- On Wed, 2013/3/27, Tatsuro MATSUOKA wrote:
> > > > --- On Tue, 2013/3/26, Christoph Ellenberger  wrote:
> > > > > I am experiencing some strange behavior and wanted to check if it is  
> > > > > a bug or me....
> > > > > It happens when I try to read in and manipulating some ascii data 
> > > > > files. It only happens with the VS version of octave. (tested with 
> > > > > several gcc versions on windows and didn't showed up).
> > > > > Ok here is some example code to reproduce:
> > > > > First generating an ascii file:
> > > > > s="0123456789";
> > > > > for i=1:21, s=cstrcat(s,s);,endfor
> > > > > save TestData.dat s
> > > > >  
> > > > > Then readin and manipulate:
> > > > > fid = fopen("TestData.dat","r");
> > > > > s = fscanf(fid,'%c');
> > > > > fclose(fid);
> > > > > s = regexprep(s,'0','2');
> > > > >  
> > > > > Clear all and repeat:
> > > > > clear all
> > > > > 
> > > > > fid = fopen("TestData.dat","r");
> > > > > s = fscanf(fid,'%c');
> > > > > fclose(fid);
> > > > > s = regexprep(s,'0','2');
> > > > >  
> > > > > Depending on datasize and how many replacement were performed it 
> > > > > either fails on the new readin or regexprep with a: "error: out of 
> > > > > memory or dimension too large for Octave's index type"
> > > > > Any help is appreciated
> > > > > Christoph
> > > > 
> > > > I have tested on octave-3.6.2 VS, 3.6.2 MinGW, 3.7.2+ VS and 3.6.2 
> > > > cygwin
> > > > (win 7 64 bit Home premium, 4GB memory)
> > > > 
> > > > 
> > > > 1. octave-3.6.2 VS and 3.6.2 MinGW*****************************
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > warning: range error for conversion to character value
> > > > >> fclose(fid);
> > > > >> s = regexprep(s,'0','2');
> > > > >> clear all
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > error: memory exhausted or requested size too large for range of 
> > > > Octave's index type -- trying to return to prompt
> > > > ********************************************************************
> > > > 
> > > > 
> > > > 2. octave-3.7.2+ VS*****************************
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > warning: range error for conversion to character value
> > > > >> fclose(fid);
> > > > >> s = regexprep(s,'0','2');
> > > > >> clear all
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > error: out of memory or dimension too large for Octave's index type
> > > > ********************************************************************
> > > > 
> > > > 3.Octave-3.6.2 Cygwin**********************************************
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > >> fclose(fid);
> > > > >> s = regexprep(s,'0','2');
> > > > >> clear all
> > > > >> fid = fopen("TestData.dat","r");
> > > > >> s = fscanf(fid,'%c');
> > > > >> fclose(fid);
> > > > >> s = regexprep(s,'0','2');
> > > > ********************************************************************
> > > > 
> > > > For me, both VS and MinGW version caused problem. Cygwin version seemed 
> > > > to work correctly.
> > > > 
> > > > Regards
> > > > 
> > > > Tatsuro
> > > 
> > > 
> > > I have also tested on 
> > > octave-3.2.4 and 3.4.3 MinGW version.
> > > 
> > > 4. octave-3.2.4 and 3.4.3 MinGW *****************************
> > > >> fid = fopen("TestData.dat","r");
> > > >> s = fscanf(fid,'%c');
> > > warning: range error for conversion to character value
> > > >> fclose(fid);
> > > >> s = regexprep(s,'0','2');
> > > >> clear all
> > > >> fid = fopen("TestData.dat","r");
> > > >> s = fscanf(fid,'%c');
> > > warning: range error for conversion to character value
> > > >> fclose(fid);
> > > >> s = regexprep(s,'0','2');
> > > **************************************************
> > > 
> > > The problem seemed to be happened on octave-3.6 or later on native 
> > > windows.
> > > 
> > > Regards
> > > 
> > > Tatsuro 
> > 
> > Sorry for successive post.
> > 
> > I have tested the following  :
> > 
> > %*********************************
> > s="0123456789";
> > for i=1:21, s=cstrcat(s,s);,endfor
> > save TestData.dat s
> > load TestData.dat
> > s = regexprep(s,'0','2');
> > clear all
> > load TestData.dat
> > s = regexprep(s,'0','2');
> > %*********************************
> > 
> > The above worked correctly on octave-3.2.4 (MinGW), 3.4.3 (MinGW), 3.6.2 
> > (MinGW,VS, Cygwin) and 3.7.2+ (VS).
> > 
> > I think that the problem does not rely on regexprep but on fscanf for large 
> > data.
> > 
> > Regards
> > 
> > Tatsuro
> > 
> Thank you very much for the fast confirmation of the same problem. Actually 
> it must be a strange combination of both I tested:
> 
> octave-3.7.2+ VS*****************************
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> clear all
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> clear all
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> clear all
> ********************************************************************
> 
> So it can't be fscanf only. On the other hand I tested as well (because your 
> last example the second regexprep does nothing, as all 0 were already 
> exchanged)
> 
> octave-3.7.2+ VS*****************************
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> s = regexprep(s,'0','2');
> s = regexprep(s,'1','2');
> s = regexprep(s,'2','3');
> error: out of memory or dimension too large for Octave's index type
> ********************************************************************
> 
> So it clearly also depends on how many replacements are performed within the 
> regexprep. (same fails as well in 3.6.0_gcc4.6.2 and 3.2.4_gcc4.4.0)
> As something like this always works:
> 
> octave-3.7.2+ VS*****************************
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> s = ctsrcat(s,"aaaaaaaaa", s);
> s = regexprep(s,'a','2');
> clear all
> fid = fopen("TestData.dat","r");
> s = fscanf(fid,'%c');
> warning: range error for conversion to character value
> fclose(fid);
> clear all
> ********************************************************************
> 
> Kind regards
> Christoph

You are right.
#**************************************
s="0123456789";
for i=1:21, s=cstrcat(s,s);,endfor

s = regexprep(s,'0','2'); 
s = regexprep(s,'1','2'); 
s = regexprep(s,'2','3'); 
#**************************************

The above caused 'out of memory ...' or 'memory exhausted ...' on octave 
3.7.2+(VS),
3.6.2(VS, MinGW), 3.4.3 (MinGW) and 3.2.4 (MinGW).

But for cygwin version octave(3.6.2), the above test did not cause memory error.

I think that it is better that you resist this strange behavior of regexprep as 
a bug to the bug tracker.

Regards

Tatsuro