Re: memory problem

[Daniel Heiserer]

address@hidden wrote:
> 
> On  1-Feb-2000, Daniel Heiserer <address@hidden> wrote:
> 
> | The machine has sufficient memory to process my request.
> 
> Apparently not. :-(
> 
> How much virtual memory do you have?
> 
> | octave:2> tic;eval(sprintf("L_%08d=eye(1000);",[1:1000]));toc
> | error: memory exhausted -- trying to return to prompt
> |
> | .........
> |  wd   matrix             1000   1000  L_00000232
> |  wd   matrix             1000   1000  L_00000233
> |
> | ==> octave could create only 233 matrizes.
> 
> That's about 233 * 1000 * 1000 * 8 ==> 1.8GB of data (unfortunately,
> even though eye produces a diagonal matrix, Octave currently uses a
> full matrix to store it).

I know. I have 16Gb of memory. :-)

> 
> | In matlab it works fine. See below.
> |
> | I had an experience with fortran before,
> | where I had a memory limitation inside
> | a subroutine, but defining everything as
> | a common block it worked. Maybe that is
> | a hint ..........
> |
> | matlab:
> | >> tic;eval(sprintf('L_%08d=eye(1000);',[1:1000]));toc
> |
> | elapsed_time =
> |
> |    72.8044
> | ...................
> |
> |   L_00000999    1000x1000     8000000  double array
> |   L_00001000    1000x1000     8000000  double array
> |
> | Grand total is 1000000000 elements using 8000000000 bytes
> 
> Thats about 8GB of data.  Do you really have that much virtual memory

I know that these are 8Gb. I have 16GB.

> available?  If not, I'd guess that Matlab is actually using a smarter
> storage scheme (either storing only the diagonal, or even just
> recognizing an identity matrix as special, or recognizing that it
> really only needs data for one 1000x1000 identity matrix that can be
> shared among all your identifiers).
> 

No. matlab stores all matrices as full unless i tell matlab
that the matrix is sparse. And even then he stores null elements
if they haven't been excluded. Sparse stuff in matlab is not 
perfect.

The limits are:
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       524288 kbytes
coredumpsize    unlimited
memoryuse       524288 kbytes
vmemoryuse      unlimited
descriptors     200 

Main memory size: 16384 Mbytes

> What happens if you replace eye() with rand()?  Do you still have
> enough memory to do this operation?
----------------------------------------------------------------------
matlab:
>> tic;eval(sprintf('L_%08d=rand(1000);',[1:1000]));toc

elapsed_time =

  229.8259
>> whos
  Name             Size         Bytes  Class

  L_00000001    1000x1000     8000000  double array
  L_00000002    1000x1000     8000000  double array
  L_00000003    1000x1000     8000000  double array
...................
  L_00000999    1000x1000     8000000  double array
  L_00001000    1000x1000     8000000  double array

Grand total is 1000000000 elements using 8000000000 bytes
>> tic;eval(sprintf('clear L_%08d;',[100:1000]));toc          

elapsed_time =

    0.5240
----------------------------------------------------------------------
GNU Octave, version 2.0.14 (mips-sgi-irix6.4).
Copyright (C) 1996, 1997, 1998, 1999 John W. Eaton.
This is free software with ABSOLUTELY NO WARRANTY.
For details, type `warranty'.

octave:3> tic;eval(sprintf('L_%08d=eye(1000);',[1:1000]));toc
error: memory exhausted -- trying to return to prompt
octave:3> whos

*** currently compiled functions:

prot  type               rows   cols  name
====  ====               ====   ====  ====
 wd   user function         -      -  clock
 wd   user function         -      -  tic

*** local user variables:

prot  type               rows   cols  name
====  ====               ====   ====  ====
 wd   matrix             1000   1000  L_00000001
 wd   matrix             1000   1000  L_00000002
..........................
 wd   matrix             1000   1000  L_00000229
 wd   matrix             1000   1000  L_00000230
 wd   matrix             1000   1000  L_00000231

octave:4> 
----------------------------------------------------------------------
Trying the rand function is HOPELESS in octave.
Some benchmarks before showed me, that rand is
one of the operations where matlab is factors faster.
I stopped the 
"tic;eval(sprintf('L_%08d=rand(1000);',[1:1000]));toc"
after a few minutes and he had only done 38 matrices!

Regardless of the inefficieny of "rand" i just
have no idea why octave cannot allocate the 8GB.

Daniel



-----------------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.che.wisc.edu/octave/octave.html
How to fund new projects:  http://www.che.wisc.edu/octave/funding.html
Subscription information:  http://www.che.wisc.edu/octave/archive.html
-----------------------------------------------------------------------