[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Importing large amounts of data
From: |
Markus Bergholz |
Subject: |
Re: Importing large amounts of data |
Date: |
Mon, 18 Jun 2012 14:56:59 +0200 |
On Mon, Jun 18, 2012 at 11:33 AM, Francesco Potortì <address@hidden> wrote:
>>Right now I am trying to do this with a 150x150x1000 int array. This
>>array has a small memory footprint in C++ and the file being pushed from
>>the C++ program to the octave script is around 65MB.
>
> Those are 22.5e6 elements. If you are using a binary representation
> with 4-byte integers, you should have a 90-MB file. If you use 16-bit
> integers, half that measure. 65 MB, if there are no errors, indicates
> that you are using a text representation, which is good and easy to
> debug for your case, but may become slow if you are planning to use much
> bigger arrays.
>
>> When reading this
>>into Octave it already consumes 8GB of RAM, which is quite a surprise,
>
> Octave uses 8-byte floats by default, but it can read and write 1, 2, 4
> and 8-byte integers. Even when using the default, your array should
> consume around 180 MB. If you see 8 GB, something is going wrong.
I can not help to the topic, but i've written a memory function for
linux to display the memory usage of octave.
http://git.osuv.de/git.osuv.de/tree/octave/memory.m
PS, I have not made any claims for correctness of the memory function.
>
>>but not the main problem (I have memory to spare right now). However the
>>reshaping is already going of for two days now on a multi-cpu Xeon Server.
>
> This too is strange. Should be in the order of a few seconds at most.
>
>>Whats going wrong? How should I approach this to get it done?
>
> Tell us exactly what format you are using for writing the file (an
> example with a small array will suffice) and what commands exactly you
> use for reading it in. For example, try with a 2x2x3 array first.
>
> --
> Francesco Potortì (ricercatore) Voice: +39.050.315.3058 (op.2111)
> ISTI - Area della ricerca CNR Mobile: +39.348.8283.107
> via G. Moruzzi 1, I-56124 Pisa Fax: +39.050.315.2040
> (entrance 20, 1st floor, room C71) Web: http://fly.isti.cnr.it
> _______________________________________________
> Help-octave mailing list
> address@hidden
> https://mailman.cae.wisc.edu/listinfo/help-octave
--
icq: 167498924
XMPP|Jabber: address@hidden
- Importing large amounts of data, Hannes, 2012/06/18
- Importing large amounts of data, Hannes, 2012/06/18
- Re: Importing large amounts of data, Francesco Potortì, 2012/06/18
- Re: Importing large amounts of data, Francesco Potortì, 2012/06/19
- Re: Importing large amounts of data, Hannes, 2012/06/19
- Re: Importing large amounts of data, indium, 2012/06/19
- Re: Importing large amounts of data, Francesco Potortì, 2012/06/19