Re: [lmi] testing actuarial_table performance

[Greg Chicares]

On 2012-05-29 13:23Z, Vaclav Slavik wrote:
> On 2012-05-27 19:40, Greg Chicares wrote:
>> Here's a very direct way: do File | New | Illustration | OK and look
>> at the statusbar. The "Calculate:" time is about thirty percent 
>> longer with xml instead of binary.
> 
> Surprisingly, it's only ~8 % for me (using debug build with Visual C++;
> average of 2nd and 3rd run because the 1st loads the database;
> LMI_USE_XML_TABLES set and actuarial_table_rates*() functions modified
> to load just one version,
> i.e. use the "SOA !! Ultimately, there will be only one class" portion).

The timings I previously reported probably loaded both and compared them;
but that shouldn't make much difference because the "soa" implementation
is so fast. Anyway, I built it without that run-both-and-compare overhead,
and the results reported below are about the same: for me, that overhead
is negligible.

> With the (testing, not production) patch below, I get the slowdown to ~2
> %. What is its effect on your system?

It works well.

Without patch, "xml" implementation only (don't test "soa" vs. "xml"):

make --jobs=1 system_test > ../log 2>&1  78.09s user 59.74s system 42% cpu 
5:20.88 total
make --jobs=1 system_test > ../log 2>&1  78.39s user 59.17s system 42% cpu 
5:23.35 total
make --jobs=1 system_test > ../log 2>&1  78.43s user 60.54s system 42% cpu 
5:23.67 total

Without patch, "soa" implementation only:

make --jobs=1 system_test > ../log 2>&1  76.07s user 58.91s system 64% cpu 
3:30.65 total
make --jobs=1 system_test > ../log 2>&1  74.91s user 57.85s system 63% cpu 
3:29.58 total
make --jobs=1 system_test > ../log 2>&1  75.54s user 59.76s system 64% cpu 
3:30.05 total

With patch ("xml" implementation only, with caching):

make --jobs=1 system_test > ../log 2>&1  77.93s user 62.30s system 64% cpu 
3:37.29 total
make --jobs=1 system_test > ../log 2>&1  79.17s user 61.24s system 64% cpu 
3:36.10 total
make --jobs=1 system_test > ../log 2>&1  77.38s user 59.49s system 63% cpu 
3:36.13 total

I think this 'system_test' target models actual usage pretty well. Along
with a few individual ('.ill') tests, it runs 54 censuses ranging from one
to one hundred eight-five cells. For each input file, the program is
reloaded, wiping out the cache. Each input file uses only a fairly small
subset of tables. Both large and small tables are covered. Each census
uses at most a handful of different products (type 'ce_product_name').
Most cells in the larger censuses use the same 'IssueAge', which wouldn't
happen in the real world; but, as explained below, I don't think that
affects the likelihood of a cache miss.

For the 'File | New | Illustration | OK' test, discarding the first run,
I observe timings of 75, 75, and 97 msec in the three scenarios above.
Thus, caching removes the 30% "xml" penalty.

It's safe to assume that using xml tables will make the guideline-premium
server much slower. It runs only a single scenario at a time, and performs
only a small part of the illustration calculations. But that's not a big
enough concern to obstruct the goal of migrating to xml: that server is
not currently used in production; and it uses only a few small tables,
which could be hard-coded if we want (e.g., table_42() in the unit test
for tables is already hard coded, and adding a few dozen like it wouldn't
take much space).

>> And if we retain many MiB of xml trees, then we might gain speed but
>> risk taking too much space.
> 
> xml_actuarial_table doesn't keep XML tree in memory (that would indeed
> be wasteful w.r.t. memory, as well as terribly inefficient). It's memory
> requirements are roughly same as soa_actuarial_table's -- just two
> vectors of doubles with the data.

Physically, 'data_' is a std::vector. Correct me if I'm wrong, but I think
that *notionally* it's a matrix in the 'e_table_select_and_ultimate' case.
This is important because, if it's true, then we're caching the entire
contents of the table: all rows and columns of the notional matrix. That's
what I think we want to do, because, for a census with fifty cells that
use the same tables but have different issue ages, we'll have forty-nine
cache hits.