Re: [lmi] PDF unit tests [Was: Integrate wxPdfDocument into lmi build system]

[Vadim Zeitlin]

On Thu, 27 Aug 2015 12:40:12 +0000 Greg Chicares <address@hidden> wrote:

GC> I'm in favor of regression-testing the group premium quote PDF.
GC> 
GC> I'm opposed to adding such tests to lmi's $(unit_test_targets). It doesn't
GC> fit there: those tests don't require big libraries or real product files,
GC> and I want to preserve that minimalist nature.

 This sounds perfectly reasonable.

GC> It may make the most sense to do this within lmi's 'system_test' target,
GC> which runs the production system and has access to product data files.

 I really don't have any opinion about this, I'd just like the tests to be
ran either automatically (preferred) or (at least) manually for any new
release.

GC> PDFs are not machine-comparable, so we'd want some sort of text output:

 My idea was to extract text from the generated PDF and compare it with the
expected. I planned to do it myself as I thought it wouldn't be too
difficult for our simple case but, actually, it looks like I wouldn't even
have to do it as there are several existing tools capable of doing it. I
didn't spend time on examining all of them but at least one, pdf2html, has
an online demo page at http://pdf2html.tabesugi.net:8080/ and uploading a
report to it results in reasonably sensible text output, so I think chances
are good that it will work for us.

GC> probably TSV, because it's simple, flexible, and already used for other
GC> tests.

 Yes, but it's not really generated by the same code, so it wouldn't be
testing it at all...

GC> but it would be much nicer if we could find a good way to separate
GC> report creation from PDF generation so that the CLI binary could
GC> create a TSV group premium quote, and CLI and GUI would both derive
GC> from a common base class that contains everything we want to test.

 I agree that it would be nicer if the goal was to test TSV output. I don't
think this should be the goal though, or at least not the main one. For me
the main danger is that something breaks in the future in the PDF-specific
code and testing TSV wouldn't help with this at all.

GC> The ideas above would ensure that the number of participants is correct.
GC> I don't think any automated test can easily tell us whether a PDF file
GC> has been truncated.

 Well, we could at least check that it contains all the expected text.
Granted, this still doesn't ensure that the text is actually readable (it
could be rendered in 0-sized font or in white-on-white colour...), but IMHO
it's a useful test to have.

GC> Exhaustive comparison of TSV output that contains all the data, combined
GC> with occasional manual inspection of PDF files--that's how we test PDF
GC> illustration output (created by XSL-FO) today, and it works pretty well.

 If it works, why not. But I think it's easy to fail to notice something
missing in a PDF by visual inspection.

GC> >  It would also be useful, IMHO, to test reports with different numbers of
GC> > lines to check that all possible pagination cases, i.e. a single page
GC> > report, a multi-page report with the footer on the same page and on its 
own
GC> > page, work as expected as this is probably one of the most fragile parts.
GC> 
GC> Perhaps a new module added to 'wx_test' could conceivably test that. I can
GC> see how it could run several different pagination scenarios, but I don't
GC> know how it would count actual PDF pages under program control.

 This is actually trivial, PDF contains information about the pages and we
can get it easily.

GC> I'm not opposed to automated testing. I'm only opposed to performing it
GC> in $(unit_test_targets).

 I'm absolutely fine with this. The main disagreement is that I'd like to
really test the PDF itself and not something else that is also used for PDF
generation.

 Regards,
VZ