Re: Reading data from large ascii-file

[Peter Norlindh]

Thank you Marco!  I have now tried to read the data in bigger chunks, and that seems to have sped things up a bit:)

 

Currently, I'm waiting for dlmread(FILE) to read the entire file.  It's been running for about one hour now and I expect it to continue for quite some time.  Are there ways to decrease the reading time even further?

 

/hpon 

On Tue, Apr 19, 2011 at 10:09 AM, marco atzeri <address@hidden> wrote:

On Tue, Apr 19, 2011 at 9:41 AM, Peter Norlindh  wrote:
> Hi,
>
> I am analyzing data from a 400MB ascii-file.  There are about 30 million
> data points in the file and the execution takes forever.
>
> Currently the program reads and processes the file element-by-element ( val
> = dlmread(fileName, "emptyvalue", [i, col, i, col]) ) and I suspect that
> this is a very unafficient way to do it.  Could you suggest a better
> approach?

surely not reading one element at time as you are doing with [i, col, i, col]

The RANGE parameter may be a 4-element vector containing the upper
    left and lower right corner `[R0,C0,R1,C1]' where the lowest index
    value is zero.

if the data is already in matrix form, you should read all the matrix
in one pass
with just

DATA = "" (FILE)



>
> Regards,
> hpon

Marco