[lmi] Calculation summary and costly IRR calculations

[Greg Chicares]

The quick answer is that we should suppress the IRR calculations
when (and only when) generating the calculation summary. They are
slow, and I'm not sure it's possible to make them really fast.
Ledger::write(xmlpp::Element&) shouldn't calculate those IRRs
every time it's called: that's a known problem that we haven't
addressed yet because (until now) they've been used only in
conjunction with printing, where the time they've taken has been
overwhelmed by the xsl-fo overhead.

[quoted by permission]
On 2006-10-17 16:04 UTC, Evgeniy Tarassov wrote:
> 
> There is a speed problem with calculation summary. If the binary
> lmi_wx.exe is compiled non-optimized then it takes from 10-15 seconds
> to generate calculation summary, due to the speed of
> Ledger::write(Element&).

I would imagine that optimization would make little difference.
If you already happen to know whether it does or does not, then
I'd be (academically) interested in knowing that.

> This function (Ledger::write(xmlpp::Element&) const) calls
> | ledger_invariant_->CalculateIrrs(*this);
> and this function call takes ages. Other parts of calculation summary
> generation are really fast (dosens of a second).

Suppose you suppress CalculateIrrs(). Then can you tell me about
how many milliseconds it takes for the calculation summary to
appear? It's sufficient to know the first decimal digit of the
mantissa (and the whole exponent, of course).

To calibrate that measurement (because our computers aren't
likely to run at exactly the same speed), please also say what
you see on the status bar when running an illustration this way:

 - File | New | Illustration
 - OK [ignore what the status bar says this first time]
 - Illustration | Edit
 - OK [say what the status bar says now]

The reason for running it twice is that the first time often
involves considerable overhead for reading data files into the
cache. Here, for instance, I see "Calculate: 154 milliseconds"
the second time.

I'd also be interested in knowing what platform you're using for
these measurements: GNU/Linux, msw, or whatever.

Perhaps we should expand the status-bar report, e.g.
  Calculate: XXX milliseconds; Format: YYY milliseconds
That would make it easy to quantify any perceived difference in
speed between the old and new calculation summaries.

> To generate html and tsv data we were:
> 1. generating xml data for ledger (Ledger::write(Element&)
> 2. applying the corresponding stylesheet
> 
> Nor HTML neither TSV need a call to ledger_invariant_->CalculateIrrs
> which is the bottleneck in the new code for calculation summary ATM.
> 
> As a workaround we could generate different versions of ledger xml
> data for different purposes:
> 1. HTML and TSV -- a light version without CalculateIrrs
> 2. xsl-fo -- a heavy version with CalculateIrrs
> 
> Do you think we should do that?

Yes. That's exactly what we should do.

> I have modified my local code to do that (to pass a boolean parameter
> to Ledger::write and to generate two versions of xml depending on the
> output format). And it works well. The calculation summary is
> generated almost instantly and apache-fop still gets the full set of
> data for ledger.
> 
> Of course it brings some complications such as not being able to use
> variables "IrrCsv" and "IrrDb" in HTML and TSV. We could also check
> that these columns are present (or absent) in the supplemental_report
> section and switch to the light or heavy version of xml.

Maybe we should just decide that no IRR will be shown on the
calculation summary--or, if an IRR is requested, it'll be
shown as zero there, which I would guess is what your local
tree does right now anyway.

That brings up another question: how should we express which
columns are to be displayed on the calculation summary? I had
originally thought of using a separate xml file--today's
'configurable_settings.xml':
  <calculation_summary_columns>
    some_column_name
    some_other_column_name
    ...
  </calculation_summary_columns>
That's very flexible, but requires editing the xml, which may be
more than we want to ask end users to do.

However, it sounds like you've come up with the idea of using the
supplemental-report column selections for the calculation
summary. That makes it easy for end users to change the
selections, but is somewhat less flexible in that the same
selections would be used for both purposes; yet that might well
be exactly what's wanted in most cases.

If that's what you've done already, then let's leave it alone for
now: it might be good enough. We can play with it and see how
suitable it seems.

> Could you please confirm that this is a way to go it, so that i could
> submit the new patch to lmi savannah page.

Confirmed. But I don't think you need to spend time redoing the
patch right now for this single, small, isolated change. Why not
wait until I answer your email of 2006-10-16 22:11 UTC? I'm
trying to understand libxml++ better first.