RE: error: memory exhausted or requested size too large for range of Octave's index type --

[William Krekeler]

-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Martin Helm
Sent: Tuesday, March 15, 2011 5:12 PM
To: c.
Cc: address@hidden
Subject: Re: error: memory exhausted or requested size too large for range of 
Octave's index type --

Am Dienstag, den 15.03.2011, 22:42 +0100 schrieb c.:
> On 15 Mar 2011, at 22:04, Przemek Klosowski wrote:
> 
> > On 03/15/2011 02:43 PM, Martin Helm wrote:
> > 
> >> Not a problem at all with 32 bit octave if you use version 3.4 since only 
> >> the
> >> index is limited to 2e9 not the bytesize of the array. An array can have 
> >> up to
> >> 16 GB (if you have enough RAM + Swap).
> > 
> > I don't understand that---in a 32-bit environment, the virtual address of 
> > each process is limited to 32 bits i.e. 4GB. It's true that you can have 
> > multiple processes, each with 32-bit address space, that span 36-bit (64GB) 
> > physical memory, but that doesn't help octave.
> 
> I think what Martin meant is that you can compile a 64bit version of octave 
> whithout enabling 64bit integer indices
> so the process will be able to allocate more than 4GB even though the maximum 
>  size of any single array will still be limited.
> 
> > p.klosowski
> c.

Sorry that I was too unclear. I just noticed myself that my comment was
very short and I omitted the essential details.
I simply checked quickly my build of 3.4 which is done without 64 bit
enabled (but on a 64 bit machine and 64 bit operating system - opensuse
11.3) so it is built without 64 bit indexing. The limititation is then
that you can not create a matrix with more than 2e9 elements (but it can
have much more than 2 GB memory consumption per matrix). What I cannot
test is if that also applies to 32 bit pae systems and esp. I cannot
check if this works at all for windows.
Creating a 5GB matrix (double 25000x25000) is in my case no problem at
all (of course it swaps because I have only 4 GB physical RAM).
I should have explained better. 
The question is still on what type of system (OS/version and
architecture) the octave runs on where the memory exhaustion happens to
see if an update to 3.4 is possible and will solve the problem.

Hope that makes it clearer.

_______________________________________________
Help-octave mailing list
address@hidden
https://mailman.cae.wisc.edu/listinfo/help-octave


I have 12 GB memory in a 64 bit windows 7 OS, Octave binary build 3.2.4. I can 
confirm that you can not build a matrix that large in Windows.

x = zeros( 25000,25000,'double');
        error: memory exhausted or requested size too large for range of 
Octave's index type -- trying to return to prompt

Largest successful array is interestingly only 13000x13000

Attr Name        Size                     Bytes  Class
 ==== ====        ====                     =====  =====
      x       13000x13000             1352000000  double

This has something to do with how Octave (or programs in general) interfaces 
with Windows memory management. I know that newer versions, I'm using win7, 
pre-allocate memory and 'lock' it out even though it technically isn't in use. 
The 'lock' preloads commonly used programs to allow for faster access. The 
memory is not truly locked because (while running similar background programs) 
I can create at least a 48000x45000 double variable in that other program (ML) 
on the same machine.

Bill Krekeler

Amended response, to be clear I'm using 32-bit Octave binary, and 64-bit ML.

Bill