Re: Import large field-delimited file with strings and numbers

[João Rodrigues]

> > I need to import a large CSV file with multiple columns with mixed
> > string and number entries, such as:
> >
> > field1, field2, field3, field4
> > A,        a,        1,       1.0,
> > B,        b,        2,        2.0,
> > C,        c,        3,        3.0,
> >
> > and I want to pass this on to something like
> >
> > cell1 ={[1,1] = A; [2,1] = B; [3,1] = C};
> > cell2 ={[1,1] = a; [2,1] = b; [3,1] = c};
> > arr3 =[1 2 3]';
> > arr4 =[1.0 2.0 3.0]';
> >
> > furthermore, some columns can be ignored, the total number of entries is
> > known and there is a header.

In response to Thomas Dean, Ben Abbot and Philip Nienhuis: Thanks for 
the tips. (I did not know either the fileread or csv2cell functions).

Nevertheless, the problem persists. Either with fscanf, cellfun + 
strsplit or csv2cell (indeed the fastest) the memory requirements blow 
up (4 GB RAM + a few more GB of swap) when I want to import the big csv 
(~3.5 million lines, 15 columns, 200 MB).

On the contrary, if I use a loop with strsplit it takes forever (~ 30 
min) but memory use is just a few hundred MB.

> Another useful trick that I sometimes use myself would be to read the file
> with textscan but then in chunks. You can specify the number of lines to
> read. textscan should remember the file position (see "help textscan").
> After having read chunk# N, you can simply restart textscan (w/o headerlines
> param!) to read chunk# N+1, and repeat until EOF.
I guess this is the right direction. I had never noticed the N option in 
textscan. Using textscan to read chunk by chunk is much faster (~ 8 min).

Yet, csv2cell is orders of magnitude faster. I will break the big file 
into chunks (using fileread, strfind to determine newlines and fprintf) 
and then apply csv2cell chunk-wise.

Thank you all
Joao