Re: Fast generation of text file

[Andy Buckle]

On Wed, Jun 1, 2011 at 10:26 AM, Andy Buckle <address@hidden> wrote:
> On Wed, Jun 1, 2011 at 9:11 AM, wild.manul <address@hidden> wrote:
>> Hello
>>
>> I need to write task for linear programming solver in text file from Octave.
>> The structure of text file is following:
>>
>> a11 x1+ a12 x2+ a13 x3+ ... +a1n xn
>> ....
>> an1 x1+ an2 x2+ an3 x3+ ... +ann xn
>>
>> where a11, ..., ann are given coefficients. x1,...,xn are symbols x with
>> indexes. Coefficients and indexes are stored in matrix a:
>> a11 1 a12 2 a13 3 ... a1n n
>> ....
>> an1 1 an2 2 an3 3 ... ann n
>> Problem is that matrix is rather huge - 2000x2000 elements and writing it to
>> text file is slow.
>>
>> I wrote the following code to do that:
>>
>> for i:row do
>>  fprintf(fid,'%7.5f x%d+',a(i,:));
>> end
>>
>> May be there is faster way how to do it?
>
> I could not figure out a way to make it faster. Some notes, and thoughts:
> - You do not have linebreaks in your fprintf.
> - The looping through fprintf calls can be avoided, but this does not
> help much. I don't know how to have linebreaks if the looping is
> avoided.
> - I tried writing the whole file with one fwrite call, instead of
> fprintf. i did not include this in the example below. This actually
> made it slightly slower.
> - Use a local drive, rather than a network drive.
> - An oct file may make things faster.
>
> my output (for 500x500) on win xp mingw octave 3.2.4 from sf.net:
>                                 no loop: 3.0s
>                                    loop: 3.2s
>                             make string: 10.5s
>     make string (less memory thrashing): 3.2s
>
> my script:
>
> % generate some dummy data
> n=500;
> inds=repmat(1:n,n,1);
> coeffs=rand(n);
> a=zeros(n,2*n);
> a(:,2:2:end)=inds;
> a(:,1:2:end)=coeffs;
>
> % write in one call to fprintf
> tic
> fid=fopen("example1.txt",'w');
> fprintf(fid,'%7.5f x%d+',a');
> fclose(fid);
> printf("%40s: %.1fs\n","no loop",toc);
>
> % loop and call fprintf
> tic
> fid=fopen("example2.txt",'w');
> for i=1:rows(a)
>  fprintf(fid,'%7.5f x%d+',a(i,:));
> end
> fclose(fid);
> printf("%40s: %.1fs\n","loop",toc);
>
> % construct string in memory first
> tic
> s="";
> for i=1:rows(a)
>  s=[s sprintf('%7.5f x%d+',a(i,:))];
> end
> fid=fopen("example3.txt",'w');
> fprintf(fid,"%s",s);
> fclose(fid);
> printf("%40s: %.1fs\n","make string",toc);
>
> % construct string in memory first: less memory thrashing
> chunksize=2^22;
> tic
> s=char(zeros(1,chunksize));
> usedlength=0;
> for i=1:rows(a)
>  append=sprintf('%7.5f x%d+',a(i,:));
>  len=length(append);
>  if length(s)<(usedlength+len)
>        s=[s char(zeros(1,chunksize))]
>  end
>  s(usedlength+1:usedlength+len)=append;
>  usedlength+=len;
> end
> fid=fopen("example4.txt",'w');
> fprintf(fid,"%s",s(1:usedlength));
> fclose(fid);
> printf("%40s: %.1fs\n","make string (less memory thrashing)",toc);

I was curious, so I used an oct file. warning: I am a great c/c++
coder. It may be possible to get this faster still.

For a 700x700 matrix (n=700, in my last script):
                                 no loop: 6.0s
                                    loop: 6.1s
                                oct file: 4.1s

writematrix.cpp:

#include "octave/oct.h"
#include <stdio.h>

DEFUN_DLD (writematrix, args, nargout,
        "example writing formatted output: two arguments, the matrix to
write, then the filename") {
        
        if (args.length()!=2) {
                error("writematrix: two arguments, the matrix to write, then 
the filename");
                return octave_value ();
        }
        
        Matrix a(args(0).matrix_value().transpose());
        double *a_p=a.fortran_vec();
        const double *a_end_p=a_p+a.length();
        std::string filename = args(1).char_matrix_value ().row_as_string(0);
        FILE * pFile;

        pFile = fopen (filename.c_str(),"w");
        
        double *p=a_p;
        while(p < a_end_p) {
                double f=*p++;
                int d=(int)(*p++);
                fprintf(pFile,"%7.5f x%d+",f,d);
        }
        
        fclose (pFile);
        
        return octave_value ();
}



-- 
/* andy buckle */