My intention is to loop over unknown size text files, each having rows of 16 coloumns, by traversing each row , and selectively appending to 12 different matrices(1 col wide) and let all the matrices grow. At the end of parsing all files, i wish to do some operation over each of the matrices.
Based on the feedback i received on this forum, i have below
1. create cell arrays , each of size defined by file size( wc -l filename) 2. read file line by line, do sscanf into array of 16 values, do conditional checks and append to one of the matrices. (see below)
Questions.
1. How do i optimize sscanf to directly read to an array(onerow), instead of the ugly workaround which i did ? 2. the below code has a problem, file size is 330,000 rows, and i pre-allocate 12 matrices, each is zeros(330,000,1), i cannot APPEND to this matrix, since the zeros take space, i should be really tracking the row index, and over writing the zero starting from index 1, right ? in that case i will need row iterators , for each matrix, (see row_index in code ), so i end up with 12 iterators. The problem with this approach though is that since i have 5 days to track, and hence 5 files to go over, and then append all the matrices, i.e matrix 5 of day 1 with matrix 5 of day2 , and so on.... i would end up having 60 different matrices, and 60 different iterators. Is there a better approach, without affecting speed ?
3. There is another unknown. at the end ofparsing a file, not all matrices will be of the same size, i.e it is not guaranteed, that every row in the file, will be useful and hence each of matrices maynot necessarily be 330k, it could be lesser. I do not mind skipping the 5 day loop ( thereby avoid nested loops ), and actually serially creating sets of 12 matrices , one set of 12 for each day, and finally appending all correspodning matrices, if we have a cleaner approach for a single day.
4. the 12 matrices, are only 1 col wide, and i need to do some plots over the values, so maybe we dont necessarily need to use matrices? arrays ? cells ?
Any suggestions, are deeply appreciated. I am hoping the gurus will have code review comments.
Thanks Manoj
42 function result = createArrays(nArrays, rows, cols) 43 result = cell(1, nArrays); 44 for i = 1 : nArrays 45 result{i} = zeros(rows,cols); 46 endfor
47 endfunction
fid = fopen(file_name); 212 213 row_index = -1; 214 215 %create 12 matrixes, big enough for the whole file size, memory allocation to speedup 216 command = cstrcat("wc -l ",file_name)
217 [success file_rows] = system(command); 218 219 %as we do this for many days, we will append all the matrices 220 alarm_combination_day1 = createArrays(14,str2num(file_rows),1); 221 222 row_index = -1
223 %loop over entire file 224 while((row = fgets(fid)) > 0) 225 row_index = row_index + 1 226 [v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16] = sscanf(row,"%f,%f,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d,%d","C");
227 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16]; 228 229 % interval check 230 interval=floor((mod(v1,86400)/interval_size) ); 231 if (interval != interval_num)
232 continue; 233 endif 234 235 mac_address = onerow(2); if ( mac_address && some condition ) // do something, and determine "value"
% appending to a matrix . alarm_combination_day1{5} = [ alarm_combination_day1{5} ; value ]
% or over writing a zero? alarm_combination_day1{5}(row_index) = value;