Re: reading data from ascii files.

[Jaroslav Hajek]

On Sat, Jul 17, 2010 at 12:32 AM, Veloci <address@hidden> wrote:
>
> Ok, after some trial and error, i've got a working version.
>
> As i wrote the dlmread function is realy slow if you try to read big
> packages. So i changed the whole thing into a read piece by piece:
>
> function
> [val_max,cnt_gr_as,val_min,cnt_lw_as,val_mean,val_std,val_count,data] =
> read_cal_data_oct(filename,gr_as,lw_as)
>
> nr_of_header_rows = 9;
> nr_of_rows2read = 1000;
>
> fid = fopen(filename);
>
> rnum = 0;
> while (feof(fid)~= 1)
>  rnum = rnum + 1;
>  l = fgetl (fid);
> end;
>
> %allocating memory
> data =zeros((rnum-1)-(nr_of_header_rows-1),4);  %rnum-1 because last row in
> my files is empty
> disp(size(data));
>
> %delimited read of the data
>
> %calculate number of fitting range vectors
> %range vector = [first_row,first_column,last_row,last_column]
>
> nr_rv_cnt = floor((rnum-1-nr_of_header_rows)/nr_of_rows2read);
> disp(nr_rv_cnt );
>
> for j = nr_of_header_rows:nr_of_rows2read:((nr_rv_cnt-1)*nr_of_rows2read+9)
>                r = [j 1 (j+nr_of_rows2read-1) 4];
>                disp(r);
>                disp(j);
>                be= j- (nr_of_header_rows-1);
>                en= be+ nr_of_rows2read-1;
>                disp(be);
>                disp(en);
>                data(be:en,:) = dlmread(filename,'\t',r);
> end;
>
> %calculate last range vectors
>
> j= j+nr_of_rows2read;
> r = [j 1 (rnum-1) 4];
> disp(r);
>
> be= en + 1;
> en= be +  (rnum-1-j);
> disp(be);
> disp(en);
> data(be:en,:) = dlmread(filename,'\t',r);
>
> fclose(fid);
>
> %data processing
> ...
>
>
> With these changes i can process my files in a matter of some
> seconds/minutes (the bigger "nr_of_rows2read", the slower it gets).
> Maybe with some adjustment it can be helpful to some of you.
>

This shows that dlmread probably has some serious performance
problems. It would be nice if you submitted a bug report.

-- 
RNDr. Jaroslav Hajek, PhD
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz