Re: [lmi] testing actuarial_table performance

[Greg Chicares]

On 2012-05-26 14:16Z, Václav Slavík wrote:
> 
> On 23 May 2012, at 17:35, Vaclav Slavik wrote:
>> Should you encounter performance problems with the XML implementation,
>> this is a likely cause. I expect loading from XML to be considerably
>> slower than the old binary format.
> 
> What would be a good way to test actuarial_table_rates*() functions' 
> performance
> realistically? Is there some particular part of LMI that makes heavy use of 
> them
> and that would be most affected by worse performance of the XML version?

Soon I'll commit some changes to make it easier to switch between xml
and binary files, but I'll go ahead and report some results now.

One way to measure performance is to use the unit test:

$make unit_tests unit_test_targets=actuarial_table_test.exe
  Speed test: 5.278e-04 s =     527837 ns, mean of 100 iterations
  Speed test: 8.842e-03 s =    8841504 ns, mean of 100 iterations

That just says that parsing xml is much slower than reading binary
floating point, but we already knew it would be.

A more useful way is to run a complete regression test, which we
hope exercises most of the code (including some proprietary things,
so you don't have the data to run this yourself):

$make --jobs=1 system_test > ../log 2>&1 [...] 3:31.42 total
$make --jobs=1 system_test > ../log 2>&1 [...] 5:20.64 total

  320.64 / 210.42 = 1.52...

That tells us that the effect on calculation speed is serious enough
to disappoint end users, so we'll have to do some sort of caching.

> I realized that I have some API design choices to make that depend on this, so
> I'd like to benchmark it myself.

Here's a very direct way: do
  File | New | Illustration | OK
and look at the statusbar. The "Calculate:" time is about thirty percent
longer with xml instead of binary.

> If you're wondering, I'm considering two approaches to multi-dimensional 
> tables: 
> 
> a) If acturarial table loading time is so small to be insignificant, then we 
> can
> continue to load the tables for just one use in actuarial_table_rates() and 
> then
> discard it. In that case, it would make most sense to load only a subset of 
> the
> XML file corresponding to given database_index vector.

I think the effect on performance is too large for that to be our best option.

> b) If, on the other hand, I worsened the performance in a perceivable way, 
> we'll
> need to make changes to keep loaded and parsed actuarial_table instances 
> around
> for as long as they are needed. In that case, it would be necessary to load 
> the
> whole table into memory and use database_index when querying it for values.
> 
> Or would you prefer to do b) regardless, even if it's a more significant 
> change
> to more code?

I guess (a) doesn't work, so we'll either have to do (b), or invent a (c) or (d)
that mitigates the slowdown. I haven't looked into it, so I don't know where the
penalty arises--e.g., do numeric conversions like this:
        data.push_back(value_cast<double>(xml_lmi::get_content(*i)));
take so much time that the libxml2 overhead is unimportant? And if we retain 
many
MiB of xml trees, then we might gain speed but risk taking too much space.