Re: Loading a 318 MB matrix into Octave 32 bit

[Elliot Gorokhovsky]

Thank you for your help. I am going to run the script on a computer that has matlab when I get the chance. 

Elliot Gorokhovsky

On Mon, Jul 29, 2013 at 7:02 AM, Mike Miller <address@hidden> wrote:

On Mon, Jul 29, 2013 at 7:57 AM, Nicholas Jankowski wrote:
> On Sun, Jul 28, 2013 at 8:21 PM, Elliot Gorokhovsky <address@hidden>
> wrote:
>>
>> I am running 32 bit octave on a 64 bit system (I would really like to keep
>> it that way if possible) and I have a .mat with a double matrix size 5492743
>> by 3. Octave says it takes 131,825,832 bytes. When I run the following
>> script (with num=140000) I get, immediately[, error: memory exhausted or
>> requested size too large for range of Octave's index type -- trying to
>> return to prompt.
>>
>> data = ""> >> for i = 1:length(data)-num
>> data(i,:)=[data(i,1:3),vec(data((i+1):(i+num),1:3)')'];
>> disp(i)
>> end
>> data = ""> >>
>> Can I fix this without recompiling? If I have to recompile, how do I do
>> it?
>>
>> Thanks,
>> Elliot Gorokhovsky
>>
>
> just trying to interpret what you did... you're saying you are loading a
> "5492743 by 3" double precision array into the variable 'data', then running
> the above script with num=140000? I get the same error message doing the
> following:
>
> ----------------------------------
> octave:18> num=140000;data=""> > octave:19> whos
>
> Variables in the current scope:
>
>    Attr Name         Size                     Bytes  Class
>    ==== ====         ====                     =====  =====
>         data   5492743x3                  131825832  double
>         num          1x1                          8  double
>
> Total is 16478230 elements using 131825840 bytes
>
> octave:20> data = ""> >
> error: memory exhausted or requested size too large for range of Octave's
> index type -- trying to return to prompt
>
> ----------------------------------
>
> so, i guess we need to look at what's going on in that first line of the
> script. I played around with it, and on Win764-bit running the Windows
> Octave 3.6.4 32-bit (MinGW version) from the precompiled archive, your
> script fails for n > 4.
>
> -----------------------------------
> octave:33> clear all;num=4;data="" =
> [data,zeros(length(data),num*3)];
> octave:34> whos
>
> Variables in the current scope:
>
>    Attr Name         Size                     Bytes  Class
>    ==== ====         ====                     =====  =====
>         data   5492743x15                 659129160  double
>         num          1x1                          8  double
>
> Total is 82391146 elements using 659129168 bytes
> -----------------------------------
>
> n=5 throws the error. If I get a chance I'll look into why, but if you play
> with a much smaller 'data', you'll see that you're multiplying the size of
> data by some variable related to n, which is going to make data very large
> very quickly using that script.

Exactly, because the zeros matrix being constructed in the first
statement is too large for Octave's 32-bit index type, whether Octave
is 32- or 64-bit. Assuming num = 140000 as in the original:

  octave:1> tmp = zeros (5492743, 3*140000);
  error: out of memory or dimension too large for Octave's index type
  octave:2> 5492743 * 3 * 140000
  ans =    2.3070e+12

--
mike