help-octave
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Reading data from large ascii-file


From: Rudolf Widmer-Schnidrig
Subject: Re: Reading data from large ascii-file
Date: Tue, 19 Apr 2011 12:22:18 +0200
User-agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9

On 19.04.11 12:05, Peter Norlindh wrote:
Thank you Marco!  I have now tried to read the data in bigger chunks, and that seems to have sped things up a bit:)
 
Currently, I'm waiting for dlmread(FILE) to read the entire file.  It's been running for about one hour now and I expect it to continue for quite some time.  Are there ways to decrease the reading time even further?
 
/hpon 

On Tue, Apr 19, 2011 at 10:09 AM, marco atzeri <address@hidden> wrote:
On Tue, Apr 19, 2011 at 9:41 AM, Peter Norlindh  wrote:
> Hi,
>
> I am analyzing data from a 400MB ascii-file.  There are about 30 million
> data points in the file and the execution takes forever.
>
> Currently the program reads and processes the file element-by-element ( val
> = dlmread(fileName, "emptyvalue", [i, col, i, col]) ) and I suspect that
> this is a very unafficient way to do it.  Could you suggest a better
> approach?

surely not reading one element at time as you are doing with [i, col, i, col]

The RANGE parameter may be a 4-element vector containing the upper
    left and lower right corner `[R0,C0,R1,C1]' where the lowest index
    value is zero.

if the data is already in matrix form, you should read all the matrix
in one pass
with just

DATA = "" (FILE)



>
> Regards,
> hpon

Marco

_______________________________________________ Help-octave mailing list address@hidden https://mailman.cae.wisc.edu/listinfo/help-octave
Dear Peter,

if you have to read the file more than once it might be worth to save it in binary format after
having read it for the first time. All subsequent reads should be very much faster.

                    -Ruedi
-- 


reply via email to

[Prev in Thread] Current Thread [Next in Thread]