Re: [lmi] change file formats to XML

[Greg Chicares]

On 2010-03-06 08:35Z, Vaclav Slavik wrote:
> 
> On Sat, 2010-03-06 at 00:16 +0000, Greg Chicares wrote:
>> Explanation: for now, we still have some non-published code that uses
>> the published product-file code; it writes files in the new format,
>> but with the old extensions. 
> 
> We could also detect XML or plain file formats based on files' content
> and reuse the "old" extensions, if that's preferable.

No, actually I think it's preferable to change the extensions, although
perhaps we arrive at that same conclusion for different reasons.

My reason is simply that lmi's msw-3.1 roots are now distant enough that
we needn't retain confusing three-letter extensions. The '.db4' extension
was just the fourth in a numbered '.db?' series, and it's so old that it
was sometimes mistaken for a "dBase IV" file; but now I guess lmi has
already overtaken Ashton-Tate on our inexorable march toward total world
domination. Anyway, I was thinking:
  .db4 --> .database
  .fnd --> .fund
  .pol --> .policy
  .rnd --> .round
  .tir --> .tier [or .stratified]
(but we don't need to make that change yet).

OTOH, I suppose your goal was backward compatibility. Generally I treat
that goal as crucial, especially since commercial alternatives to lmi
usually get it wrong for the usual input files (like our '.ill').
However, product files have never been backward compatible: they become
invalid whenever we alter any of the formats (even if we just add a new
entity to the database), and switching to xml has the same consequence
(except that henceforward the xml files will be backward compatible).
Very few users have created customized product files, and most have
been thrown away after a single use.

I thought of providing a separate, standalone program to convert files
from the old formats, but the benefit seems too small to justify the
cost. Furthermore, such a program would probably require maintenance
(even though we might like to believe otherwise), whereas I'm eager to
get rid of the legacy code permanently.

>> [Some of those files contain literal
>> ampersands, which must be replaced with xml '&' entities; after
>> replacement in the non-published source, we see:
>>   Income & Growth
>>   M&E
>> rather than:
>>   Income & Growth
>>   M&E
>> .]
> 
> I'm not sure I understand. Are you saying that the code produces invalid
> XML files? Or that XML files with entities such as & are read
> incorrectly? If either of these cases isn't handled by xmlwrapp
> transparently, then it should be fixed.

Syntactically valid xml files were written with incorrect contents.
I don't know whether that's an xmlwrapp issue or not. For example:

    static std::string const asset_charge_footnote =
        "... a mortality and expense risk charge (\"M&E\") ...";  // original
//      "... a mortality and expense risk charge (\"M&E\") ..."; // fixed
    TProductData* x = new TProductData();
    x->AssetChargeFootnote = asset_charge_footnote;
    // (That code is so ancient, it prefers the heap in order to
    // conserve a 64K stack.)

With the 'original' code, the file contained
  ... a mortality and expense risk charge ("M
and everything after the ampersand was dropped.

If that indicates an xmlwrapp issue, then I support fixing it; but
either way it's not important for the task at hand. The workarounds
I gave are trivial and few. Moreover, we intend to retire the code
that requires these workarounds. Instead of maintaining a program
to generate product files in the various legacy formats, we'll just
maintain the xml files, most likely by editing the xml as text.

>>          static void to_xml(xml::node& out, T const& in)
>>          {
>> +            out.set_content(value_cast<std::string>(in).c_str());
> 
> I avoided value_cast<> because of precision. If using numeric_io_cast<>
> (which I managed to overlook somehow) is acceptable, then I'll update
> the patch to use it. 

No, you needn't do that. I'm already working on it.