Re: out of memory or dimension too large for Octave's index type

[Markus Bergholz]

On Sun, Sep 7, 2014 at 4:06 PM, Tatsuro MATSUOKA <address@hidden> wrote:

----- Original Message -----

> From: Philip Nienhuis 
> To: help-octave
> Cc:
> Date: 2014/9/7, Sun 20:50
> Subject: Re: out of memory or dimension too large for Octave's index type

>
> mtalberg wrote
>>  Hello
>>
>>  I am having an issue with the amount of memory Octave can use compared to
>>  the available memory on my system. I am running on Window's 7 64-bit,
> 16GB
>>  of RAM, and have GNU Octave version 3.8.2, configured for
>>  "i686-w64-mingw32".
>>
>>  I saw this error after trying to write a 9329664x2 matrix to a .wav file
>>  using the wavwrite command.
>>
>>  I monitor the memory usage of my computer as a whole, and have determined
>>  that Octave is only using roughly 500MB of RAM. Is there any way to boost
>>  the amount of RAM accessible by Octave? Any help would be greatly
>>  appreciated, thank you!
>
> A few months ago I replied to a similar question here in more detail, you
> can search the mailing list archive for that.
> In short:
> You're hit by memory fragmentation, and that is very "normal" on
> any OS.
> Octave isn't even so bad in this respect. I tried both Octave and Matlab on
> the same PC (32bit XP w. 4 GB RAM) to find out how much memory was
> available. Matlab has a call for that, in Octave one needs trial and error.
> Turned out the largest contiguous piece of RAM ML could assign amounted to
> only (IIRC) ~600 MB. Octave was marginally better, but that number had to be
> inferred (indirectly).
>
> You'd have to wait for a 64-bit Octave. I heard rumours that that
> shouldn't
> take too long now.
> Cross-building it on Linux is fairly easy these days. I have one here,
> Markus has one, but there needs to be some polishing done before it can be
> -unofficially-  "released".
>
> Philip
>  

Unofficial Large Indexing was build with enable-64 in 
mxeoctave.osuv.de.


http://mxeoctave.osuv.de/


Note that the binary are *experimental*!!!

yeah, very very experumental!

Another question is, do you realy need *all* data at once in your workspace? or just 2-3 columns at once?

I've put an interpreter (lua) in the octave interpreter (simple mex function), so you can use lua
* to prevent bottlenecks

* database connections

* and much more

e.g. I've got here a 1.8GB csv file (1mio lines, 100 columns). If I just want to access column number 18, it takes ~20 seconds to read it into octave.


octave:14> tic, m=lua('csvaccess','big.csv','col',18,','); toc
all done, lua stack top 0
Elapsed time is 20.6918 seconds.
octave:15> system('du -csh big.csv')
1,8G    big.csv
1,8G    insgesamt
ans =                    0
octave:16> whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        ans         1x1                          8  double
        m           1x1000000              8000000  double

Total is 1000001 elements using 8000008 bytes

octave:17> memory

 Memory used by Octave:   300.188 MB
 Physical Memory (RAM): 7929.52 MB

and of course that means, you have to write a lua function for your need... but you can still write a C/C++ mex/oct file for that too. But for me, lua is easier to write.

I can put my lua function later online, but it's still experimental too :D

 

Tatsuro

_______________________________________________
Help-octave mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/help-octave

--
icq: 167498924
XMPP|Jabber: address@hidden