Re: [lmi] Robust timings in unit tests

[Greg Chicares]

On 2017-05-11 16:45, Vadim Zeitlin wrote:
> On Thu, 11 May 2017 14:57:18 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2017-05-07 22:14, Vadim Zeitlin wrote:
> GC> > On Sun, 7 May 2017 17:49:29 +0000 Greg Chicares <address@hidden> wrote:
> GC> [...]
> GC> > GC> An argument might be made for reporting the lowest measurement 
> rather
> GC> > GC> than the mean.
> GC> > 
> GC> >  This seems a good argument to me and this is exactly what I do when
> GC> > measuring CPU-bound code.
> GC> 
> GC> Commit 1a629bf changed AliquotTimer so that it reports the minimum.
> 
>  Thanks! Is it normal that I (reproducibly) get the warning below in this
> test output now:
> 
>   5.268e-08 s mean;          0 us least of 189833 runs
>   7.143e-06 s mean;          7 us least of 140 runs
>   7.088e-05 s mean;         70 us least of 100 runs
>   1.000e-02 s: first trial took longer than 1.000e-02 s desired limit
>   1.000e-02 s mean;      10001 us least of  10 runs
>   1.000e-02 s mean;      10001 us least of  11 runs
>   1.000e+00 s mean;    1000065 us least of   2 runs
> 
> ?

Yes. The middle test is contrived to exercise that warning:

    std::string takes_too_long = TimeAnAliquot(WaitTenMsec, 0.0099999).str();
    BOOST_TEST(contains(takes_too_long, "took longer"));

I.e.: time a function that takes 10 msec, but spend less than 10 msec
timing it.

> [...] I'm still going to mention that now that tests can be
> run under Linux it is also possible to use "perf" for measuring much more
> than just the time taken and it works even for code executing for very
> short time as perf uses hardware CPU counters, which allows to see, for
> example, the number of instructions retired per cycle, which is arguably
> the single most important number for estimating code performance.

Maybe someday. Right now, I'm working on a new premium calculation,
and I also want to upgrade gcc and try 64-bit builds. After that, I
would like to start using clang myself, routinely. But I'll keep
'perf' in mind for the future.