Re: Problem with large array

[George]

Hi,

Thanks for all the help.

The print statements are not an issue, because I am only printing one every few seconds.

I found the approved answer to this Stack Overflow question very helpful:

http://stackoverflow.com/questions/4420235/efficient-multiplication-of-very-large-matrices-in-matlab

Here's how I fixed it:

1. whos after each main loop update - lets me see how much memory is being used

2. Initialized the matrices to sparse matrices using   

      X_big = sparse([]);

      Y_big = sparse([]);

This worked particularly well for this case because over 99% of my matrix was 0. I guess it would not help if most of the matrix was non-zero, but the Stack Overflow Answer above has other possibilities that might help.

George

On Tue, Mar 20, 2012 at 8:28 AM, Francesco Potortì <address@hidden> wrote:

>error: memory exhausted or requested size too large for range of Octave's
>index type -- trying to return to prompt.
>
>The size of the matrix when the error occurs is 23,289 x 2,387 or about
>55,000,000 elements.

That is around half a GB.

If Octave manages to return to prompt, try 'whos', which will tell you
the size of allocated variables.  If not, try to put whos calls in your
loop, so you can monitor the memory consumption growth.

>Working in WIndows XP, on Octave 3.2.4. I have a 64 bit CPU, but I think
>running Windows 32 bit version.
>
>I am combining the elements of two matrices. Here is the code:
>
>  X_big = [];
>  Y_big = [];
>  nm = size(R, 1);
>  np = size(R, 2);
>  for m_counter = 1 : nm
>    printf("%i...", m_counter);
>    for p_counter = 1 : np
>      if R(m_counter, p_counter) == 1
>        this_row = [features(m_counter, :), Y'(p_counter,:) ];
>        X_big = [X_big; this_row];
>        Y_big = [ Y_big; Y(m_counter, p_counter) ];
>      endif
>    endfor
>  endfor

The printf statements may slow things down: try to remove it and see if
that helps.

The if statement could be removed like this, which may be faster:
for p_counter = find (R(m_counter,:) == 1)

Preallocating X_big and Y_Big may bring some speed improvement.

How much RAM memory do you have?

--
Francesco Potortì (ricercatore)        Voice:  +39.050.315.3058 (op.2111)
ISTI - Area della ricerca CNR          Mobile: +39.348.8283.107
via G. Moruzzi 1, I-56124 Pisa         Fax:    +39.050.315.2040
(entrance 20, 1st floor, room C71)     Web:    http://fly.isti.cnr.it