[lmi] Design decisions behind '.policy' annotations

[Greg Chicares]

Revision 4842 of 20100418T2119Z combines two approaches that have until now
evolved separately despite certain similarities:

(a) 'xml_serializable.?pp': flat model--everything's stringizable
    a collection of heterogeneous types (e.g., mc_enum, tn_range)
    every type has a natural mapping to and from a string
    heterogeneity accommodated through type erasure
    supports iteration over the collection of data members
    consistency checks; provision for backward compatibility

(b) 'xml_serialize.hpp': recursive type descent--arbitrary complexity
    a single compound datatype per file--per-class specialization
    '.strata' file, for example:
        <tiered_ak_premium_tax>
          <values> <item>0.027</item><item>0.001</item> </values>
          <limits> <item>100000</item><item>INF</item>  </limits>
        </tiered_ak_premium_tax>
    serialization walks the types, e.g. through the vectors
    no built-in support for iteration over data members

Well, those are the differences; the similarity is here:
 // (a)
    if(residuary_names.end() != current_member)
        {
        read_element(x, node_tag, t, file_version);
        residuary_names.erase(current_member);
        }
 // (b)
    #define GET_ELEMENT(name) xml_serialize::get_element(root, #name, 
round_##name##_)
    GET_ELEMENT(specamt         );
    // And each other element...
    #undef GET_ELEMENT
I.e., both iterate over a set of named member data, for both serialization
to and deserialization from xml.

They're used in disjoint sets of files:
  (a) input.hpp, mec_input.hpp, mec_state.hpp
  (b) ihs_funddata.cpp, ihs_rnddata.cpp, stratified_charges.cpp
but now 'product_data.hpp' uses parts of both.

The data-member lists in the older (a) classes map not only between members
and xml elements, but also between members and controls (through the mvc
framework). We might have made each 'product_data.?pp' annotation a
distinct member, e.g.:
  std::string DatabaseFilename;           // what used to be there
  std::string DatabaseFilenameAnnotation; // what would be added
which would have made it possible to write an mvc-based GUI like this:
   textcontrol  static-text        textcontrol
  [sample.db4] Database filename  [some comment]
  [sample.xyz] Another filename   [another comment]
[Conforming to the existing '.strata' GUI is *not* a design imperative.]
Whatever charm that might appear to possess doesn't extend to the '.strata'
example above: we certainly don't want members like
  tiered_ak_premium_tax_values
  tiered_ak_premium_tax_limits
(which wouldn't be scalar anyway--and, unlike in the GUI for class Input,
we don't want to "flatten" these vectors into 'input sequences) or, worse,
  tiered_ak_premium_tax_value0
  tiered_ak_premium_tax_value1...
which would be scalar but dreadful. OTOH, we can hope eventually to unify
the product-editor GUIs along the lines already used for '.strata' and
'.db4' files: a tree control on the left, which selects a single (complex)
datum to be shown on the right. For '.policy' files, that's a better fit
anyway, because many data representing footnotes might be hundreds of
characters long, and therefore don't fit in a default-sized textcontrol:
each one wants a panel to itself.

Therefore, the stringization stuff in any_member<> is undesirable for
'.policy' files, and for the time being I've coarsely suppressed it, but
eventually any_entity ought to be better separated from any_member<>.
I'd like to rework the other product-data classes first, deriving them all
from class xml_serializable (and using ns xml_serialize for the datatypes
they contain, as at present). Before doing anything else, though, I want
to clean up the xml_serializable API; for example, I had originally planned
to pass any_member<T> arguments to read_element() and write_element(), but
instead passed a T& and the member name; and the T& is unneeded, because
it's just (T)(*this).

We always knew xml would be slower than the old product-file code. Here are
unit-test results for 'product_file_test':

before xml:
  Read 'policy'     : 1.539e-04 s =     153918 ns, mean of 100 iterations
original xml:
  Read 'policy'     : 1.158e-03 s =    1157625 ns, mean of 100 iterations
today (with annotations):
  Read 'policy'     : 3.253e-03 s =    3253169 ns, mean of 100 iterations

And here are timings for complete system tests:

revision 4841:
  make system_test > ../log 2>&1  83.08s user 66.67s system 44% cpu 5:37.89 
total
revision 4842 [I happen to have three runs]:
  make system_test > ../log 2>&1  82.88s user 67.82s system 42% cpu 5:54.44 
total
  make system_test > ../log 2>&1  81.80s user 67.58s system 42% cpu 5:48.10 
total
  make system_test > ../log 2>&1  82.05s user 68.31s system 42% cpu 5:52.62 
total

An extra ten or fifteen seconds doesn't sound like much, but reading a
'.policy' file only takes "3.253e-03 s" according to the unit tests, so it
seems likely (and experimentation confirms) that these files are loaded
several times for each illustration produced; perhaps that matters enough
now to justify either reworking file access, or caching the files, or both.
These pre-xml timings:
  Read 'database'   : 1.484e-02 s =   14843822 ns, mean of  68 iterations
  Read 'fund'       : 7.479e-05 s =      74795 ns, mean of 134 iterations
  Read 'policy'     : 1.539e-04 s =     153918 ns, mean of 100 iterations
  Read 'rounding'   : 7.639e-05 s =      76392 ns, mean of 131 iterations
  Read 'stratified' : 3.558e-04 s =     355843 ns, mean of 100 iterations
indicate why I'm holding back the '.database' conversion to xml until I
find time to consider caching carefully.

(BTW, class Ledger has stuff like
  <data>
   <newcolumn>
     <column name="AVRelOnDeath_Guaranteed">
       <duration number="0" column_value="0"/>
       <duration number="1" column_value="0"/>
which should be reworked similarly someday. One thing at a time.)