Re: [lmi] Terse list of valuable projects

[Greg Chicares]

On 2010-03-16 20:10Z, Greg Chicares wrote:
> 
> Hmmm...I just looked at an "XTbML" table, and...the more I think about
> this, the more I suspect your vector format for our '.db4' and '.tir'
> files is better than what they've done....

Picking this up again, I think this "XTbML" format was once published
on the soa.org website deliberately with no restrictive copyright,
but that appears no longer to be the case. IMO, it's technically
inferior anyway, and is now not usable in the free world, so we want
to design our own format.

Both this "XTbML" thing and the earlier, free-as-in-freedom binary
format that we actually use, provide a lot of metadata that we aren't
really interested in, along with the values that we actually use. But
both formats write only a slice of data in each "table". For instance,
lmi's nonproprietary 'sample' product, as seen in this excerpt from
'dbdict.cpp' (second row, third column), uses table forty-one for
male, unismoke:

    double TgCOI[9] =
        {
         39,  37,  35, // female: sm ns us
         45,  57,  41, // male:   sm ns us
        111, 109, 107, // unisex: sm ns us
        };

Such tables typically vary by
  {female, male, unisex}
  {smoker, nonsmoker, unismoke}
as well as attained age, and a good format wouldn't encapsulate only
age in a "table" and use distinct files for the other axes, because
this is really a 3 X 3 X 100 array of floating-point numbers, not
nine 100-element arrays.

The crucial knowledge in the problem domain is that we need to store
such multidimensional tables of floating-point numbers, which at present
are read by 'actuarial_table*.?pp' (from non-xml tables--and those C++
files are based only on public information, and hence are untainted by
any copyright on the "XTbML" format). The enumerated
  e_reenter_at_inforce_duration
  e_reenter_upon_rate_reset
  e_reenter_never
methods for table lookups are important; they're explained in the
header, and probably they can and should be decoupled from the format
in which the data are stored. The "records" of interest in the
publicly-documented, free-as-in-freedom interface are:
///   2    4-byte integer:  Table number
///   3    [unsigned] char: Table type: {A, D, S} --> {age, duration, select}
///   12   2-byte integer:  Minimum age
///   13   2-byte integer:  Maximum age
///   14   2-byte integer:  Select period
///   15   2-byte integer:  Maximum select age (if zero, then it's max age)
///   17   8-byte doubles:  Table values
of which type "2" is unimportant, and type "15" may be useless. Type
"3" just means that a "table" can vary by either one or two axes.

Today, we're using only those two axes. Going forward, we want to add
other axes; additional axes are limited to those already used in
'dbindex.hpp'. Problem-domain analysis may reveal that the two axes now
used in tables can be mapped to
    ,e_axis_issue_age = 3
    ,e_axis_duration  = 6
so that a "table" can be stored in the same way as an element in a
'.database' file; but I'm not sure. (I'm trying to favor speed over
rigor in this email.)

The present tables are stored in a binary format, which is ugly but
presumably is fast at run time. Run-time performance is important, so
we don't want to make it slower. Probably that concern can be addressed
by caching the data read from xml. The real purpose of using xml here
is to make it viewable and editable in (an enhanced version of) the
product editor.

Many of these tables are shared among different "products". Some are quite
large, so we'd want to avoid duplicating identical tables. Thus, in the
"TgCOI" example above, 39, 37, etc. are pointers to tables; instead, we'd
want a single pointer to a single table that has more dimensions.

We would certainly want one collection of proprietary tables, and another
that's public information.

The (clearly free) binary tables we're using now are manipulated by an
obsolete 16-bit program that apparently is problematic with msw 7. Our
own 'actuarial_table*.?pp' code performs some of that program's most
important functions, but not all. To clear the path for moving to the
latest msw version, we want to sweep away the ancient stuff, eschew the
non-free stuff, and do it right, once and for all.