[lmi] Discussion of using external templates for PDF generation

[Vadim Zeitlin]

 Hello again,

 This is the second part of the PDF generation discussion. In the first one
I've shown the current approach and the first natural question is: what's
not to like about it? My main problem with it is that it's repetitive and
while I probably should have defined an array of terms names/definitions
in the particular example I chose to make it less so, using arrays doesn't
work for the other pages and even in this example I'm not totally sure it
would help things that much because of the conditions surrounding some of
the paragraphs.

 The other problem is that this is still C++ code and I think it's too
intimidating for anybody but C++ programmers to modify it. Again, I don't
know if it's actually a problem or an advantage from your point of view,
but from purely technical perspective, I think it would make sense to pull
all the contents into an external file, which could then be edited
independently. Such file for the same example would look roughly like this:

---------------------------------- >8 --------------------------------------
<p align="center">
Column Headings and Key Terms Used in This Illustration
</p>
<font size="-1">
<p>
<b>{{AvName}} Value:</b>
The accumulation at interest of the net premiums paid,
{{#SinglePremium}}
less any withdrawals,
{{/SinglePremium}}
less any monthly charges deducted.
</p>
<p>
<b>{{CsvName}} Value:</b>
{{AvName}} Value less policy debt.
{{#Has1035ExchCharge}}
{{CashSurrValueFootnote}}
{{/Has1035ExchCharge}}
</p>
{{^IsInforce}}
<p>
<b>Current Illustrated Crediting Rate:</b>
{{CreditingRateFootnote}}
</p>
{{/IsInforce}}

... all the rest of HTML corresponding to the code above ...

</font>
---------------------------------- >8 --------------------------------------

and the code itself would just read this file into a string and pass it to
interpolate_html, before calling output_html() with it.

 There are several problems/disadvantages with doing this, however, here is
a list of the most important ones I'm currently aware about and haven't
solved yet:

- Some conditions above can't be expressed in Mustache: e.g.
  "ModifiedSinglePremium || ModifiedSinglePremium0" one, so it would mean
  either duplicating the contents guarded by this conditions or predefining
  "ModifiedSinglePremiumOrModifiedSinglePremium0" variable in the code,
  which is arguably better but is still not great. Alternatively, we could
  extend Mustache syntax -- but this contradicts the point just above. Or
  we could switch to Handlebars (these web 2.0 guys are so funny), which is
  (almost exact) superset of Mustache, but supports "{{#if condition}}"
  blocks. However for Handlebars there is no decent C++ library, so I'd
  have to write one myself, which not only contradicts the point above, but
  would take some non-trivial amount of time, too.

- Some things that can be done by directly drawing on wxPdfDC can't be done
  with wxHTML, the most trivial example is the blue (sorry, #002F6C) border
  on the cover page. So I'd need to either extend wxHTML to support borders
  like this or have some hack for drawing just this part of the page in
  code. Of course, a more difficult problem is not this simple rectangle,
  but drawing the lines in different tables which are also not directly
  supported by wxHTML. And modifying wxHTML is not exactly simple, Vaclav
  did do a good job with it, but this was 20 years ago and, well, quite a
  few things have changed since then...

- The tables are, actually, the problem that I haven't yet managed to solve
  satisfactorily with HTML templates. They're clearly easier to handle in
  the code, there are multiple problems with generating them from HTML. The
  first and obvious one is that basic Mustache just has no way of doing
  this at all. If we used Handlebars, we could use its "helpers", which are
  blocks like {{#my_table some args}} that invoke a custom function
  "my_table", defined at the code level, with the given arguments. But
  using Handlebars is problematic, as mentioned above. So finally I decided
  to make this work with just {{my_table_some_args}} by defining the
  appropriate variable but so far I didn't yet manage to do even this
  (mostly due to lack of time though, not any fundamental problems).

- The second problem with tables is even worse as I just don't know how
  to solve it currently: it's about page breaks. In C++ code it's not
  really nice neither, but is doable because I know the height of the page
  and the height of each line of text, so it's just a question of tracking
  the number of lines and I do this already in group_quote_pdf_generator_wx
  code. But with external templates I am really not sure how to do it, I
  think it could involve modifying wxHTML again.

- While the code is repetitive, it does allow easily defining helpers
  making it less so, e.g. add_term_paragraph() function in the example
  above. In HTML, everything would need to be written out by hand, which
  could well end up being quite annoying too. Again, Handlebars helpers
  could help with this, but for now I am still not convinced switching to
  Handlebars is worth it.

- This is not a technical problem, but I'd still like to mention it: it's
  easy to imagine that you can write arbitrary HTML when editing the
  template files because you could open them in the browser (which, on its
  own, is an important _advantage_ of this approach, of course) and see
  that it renders it correctly, but this is not at all the case as wxHTML
  is limited to HTML3, i.e. 1990s state of things. Currently, with the
  (very few) predefined HTML tags and attributes that we have, all HTML
  generated by C++ code using these helpers is almost guaranteed to be
  displayed correctly ("almost" because you could always use unsupported
  attribute value, they're untyped).


 As for the advantages of this approach are probably quite clear, but let
me list them here just to be explicit:

 + Much better (although still not perfect because complex conditions still
   require modifying C++ code in order to introduce ad hoc variables for
   them) separation between the code and the contents.

 + Ability to update the PDF without recompiling the program. This may not
   seem like much, but it's very appreciable during development, being able
   to tweak HTML slightly and just return the illustration generation is
   orders of magnitude faster than updating the code, rebuilding,
   relaunching the program, reloading the illustration and finally
   generating it again.

 + Possibility to preview HTML in any browser. This is probably not that
   useful with the already finished template files as Mustache stuff gets
   in the way (although by using a (subset of the) standard template
   syntax, we get the ability to run any of Mustache processor programs to
   get rid of it if necessary), but I believe it could be quite handy when
   drafting a new page for example: you'd start from just HTML, make sure
   it looks like you intend in the browser and then just replace some parts
   of it with variables and/or put section begin/end markers around other
   parts.

 There are other possible/potential advantages too, but I think even just
these ones are convincing enough to agree that it would be better to
generate illustrations using external templates. The only question, but
unfortunately not the least one, if I can manage to produce tables in this
way. All the rest translates to external templates without any real
problems (except for minor details like the blue frame on the cover page),
but I'm still not sure what to do about the tables.

 But to be honest, while writing these emails and thinking about all this
again, I became personally convinced that we should use external templates
in any case: for everything if possible, but if not, then I'd still like to
use them for everything except the tables and handle the tables in some
other way. This does mean that a lot of work already done (e.g. all the
HTML generation helpers...) will have been wasted, but even nicely
type-safe HTML generation code in C++ is still not as nice as just writing
this HTML directly in a file.

 Would you agree with switching to this approach?
VZ