Re: [lmi] Automated GUI testing, revisited

[Greg Chicares]

On 2014-11-12 01:55Z, Greg Chicares wrote:
[...]
> Vadim--I believe I copied 'wx_test.conf' from an old email that
> you had sent. Was "time_disk=11000" an actual measurement? For
> single tasks, your machine should be faster than mine; so I'm
> wondering whether we're running the same test.
> 
> My "456ms" above looks a lot like "time_run" in 'wx_test.conf':
>   time_run=434
>   time_disk=11000
>   time_spreadsheet=710
> (I mention that without further investigation on my part).

Here's more information, from another run:

11:49:25: Run case for census_1: 416ms elapsed (expected -4.15%)
11:49:25:     benchmark_census: ERROR (Assertion 'std::fabs(diff_in_percents) < 
10' failed
(expected 11000ms, but actually took 416ms, i.e. -96.22%).
[file /lmi/src/lmi/wx_test_benchmark_census.cpp, line 101] )

Doesn't this suggest that the "ERROR" comparison is incorrect?
I can match "expected -4.15%" this way:
  (416-434)/434 = -0.04147...
so that seems right. Assuming that "416ms" is my "time_run",
I can match "i.e. -96.22%" this way:
  (416-11000)/11000 = -0.96218...
but it seems incorrect to compare observed "time_run" to
expected "time_disk", if that's what's really being done.

But I still think it's best to remove these relative-error
calculations, as suggested in the message I'm replying to.