File size limit?

[John W. Eaton]

On 16-Apr-1999, Andre Bonfrer <address@hidden> wrote:

| Does Octave have a limit imposed on reading in files from ascii
| data?

As far as I know, you are limited by the amount of memory you have, or
the largest value that can be represented in a signed int, which
Octave uses internally for storing matrix dimensions (I know, it
should probably be size_t, but it is not, at least yet).

| I was trying to use fscanf to read in a file with about 4million rows, 7
| columns (fixed length):
|                294     2 298        11    2.6200         1            1
|                294     2 299          9    2.5100         1            1
| 
|                294     2 300          6    2.4900         1            1
| 
|                294     2 301          5    2.9900         1            1
| 
|                294     2 302          1    2.4900         1            1
| 
| ...
| 
| Using fscanf:
| infid = fopen("data.dat","r","native");
| [wb,count]=fscanf(infid,"%g",[inf,7]);

I think you probably want to write

  [wb,count]=fscanf(infid,"%g",[7, inf]);

here.

| Each time the size of wb is limited s.t. count=33795
| Can anyone help me?  Is there some parameter I have to set in
| configuration to allow Octave to recognise beyound 33795?

Do you see any error messages?  What version of Octave are you using?
I had no trouble reading a file like yours containing several hundred
thousand lines.  I ran out of memory trying to read a file with 4
million lines (That's not surprising, since a 7x4000000 matrix
requires more than 210 MB.  If you don't tell Octave how large the
matrix is in the fscanf call, it has to reallocate a buffer many
times, starting with a result matrix containing just nr*32 elements,
then doubling the number of columns each time it runs out of room.

If you want to optimize this for larger matrices, you can.  Just take
a look at the code the code that handles scanf conversions in
src/oct-srtream.cc

jwe