Re: File size limits in textread()?

[Keith Goodman]

On 6/9/05, address@hidden <address@hidden> wrote:
> I'm using textread() to read in mixed string and integer data from a space 
> delimited text file. I found it worked okay with one data file, but then 
> hangs when reading another,  larger data file. Hitting ctrl-c brings back the 
> octave prompt.
> 
> Any ideas as to why textread() doesn't like the larger data file? This file 
> has about 6000 rows and 10 columns, 4 of which are strings, the others are 
> integers.
> 
> I'm now using a loop with fgetl() and split(), but it's very slow.
> 
> The data file is generated from a database query. I might use 2 queries, one 
> for
> integer data and one for string data. That way at least I can use load or 
> dlmread()
> to quickly pull in the integer data, and then deal with the string data using
> textread().

It works, but you have to be patient. Try using the function waitbar
to watch the progress.

In textread.m change

while (!feof(fid))

to

ccc= 0;
while (!feof(fid))
ccc = ccc + 1;
waitbar(ccc/6000);

The time is long because there is a for loop with 10 iterations nested
inside a while loop with 6000 iterations. And the output vectors and
cells are not initialized to a length of 6000 before the loop. Is it
possible to initialize varargout{i} before the loop?

varargout{1} = repmat(NaN,6000,1);
...
varargout{10} = cell(6000,1);

The best solution is probably to write textread, and all other Octave
string functions, in C++.



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------