Re: [lmi] PDF generation performance

[Greg Chicares]

On 2018-02-02 17:17, Vadim Zeitlin wrote:
> 
>  While testing the pagination anomaly bug, I couldn't help noticing how
> awfully long PDF generation took for the census used to reproduce the
> problem. With the (optimized) MSVS build, it took ~40 seconds and while I
> didn't measure it exactly with MinGW, it doesn't seem to be any faster.

Running the following commands, each followed by loading that file and
pressing Ctrl-Shift-I, the timings noted on the statusbar are as follows
on my machine:

wine ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data --pyx=only_old_pdf
15403 milliseconds

wine ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data --pyx=only_new_pdf
14875 milliseconds

However, because the 'java' msw installer fails here, the "old" runs
don't actually invoke 'fop'; if they did, the "new" code would surely
be much faster than the "old".

I could probably boot an old version of GNU/Linux where that software
did install, and measure it there; but it's easier to wait until Monday
and ask Kim to run both ways in the same msw environment that end users
have. (That'll also be more accurate, because the "anti-malware" suite
and the hardware will be more representative of typical users.)

>  I will, of course, profile the code to see if I can speed it up in its
> current form, but it seems clear that to achieve any really important
> gains, as we'd need to make this more bearable, we have to parallelize the
> output files generation.
> 
>  Would you agree that doing this is important?

Parallelization is a separate task, which is important but not urgent.
Reviewing and testing the new PDF code in great depth is the task we're
focusing on now, and I think the weather will be much warmer by the
time we can give proper attention to any other major task. Worthwhile
tasks for the future certainly include adapting to current hardware
realities: everyone now has a 64-bit machine with multiple cores, but
lmi is a single-threaded program and we distribute only 32-bit builds.