Re: [lmi] Limits on pastable census size

[Vadim Zeitlin]

On Fri, 9 Feb 2018 14:06:14 +0000 Greg Chicares <address@hidden> wrote:

GC> Yesterday we had to work with a census with over ten thousand cells.
GC> (We've seen one that large only once before.) It seems to work now,
GC> but it was difficult to create with "Census | Paste census", which
GC> ran out of memory (it threw bad_alloc).

 This is surprising. In the current 32 bit build, sizeof(Input) is "just"
8920 (BTW, with MSVC it's 8248, so it looks like some space could be saved
just by using tighter packing), so allocating enough space for 10000
objects of this type requires about 85MiB of RAM, which shouldn't create
any trouble for a system capable of running any contemporary software
(looking at Firefox and 2.5GiB in its working set column on my system right
now...). So either the machine this was tested on was really underpowered
or its RAM was heavily fragmented for some reason. BTW, what version of MSW
and of which bitness did this happen under?

 If you'd like me to debug this further, please send me the data you
attempted to paste so that I could at least see if I can reproduce this
here.

GC> I imagine this problem wouldn't arise with a 64-bit build (though I
GC> haven't tested that). Maybe this long-deferred PR:
GC>   https://github.com/vadz/lmi/pull/18
GC> would help marginally

 AFAIR it makes things faster, but I don't think it reduces memory
consumption. I could be wrong though, I did this a long time ago.

 Regards,
VZ