[PATCH 2/4] added autotest support

[Andrei Kholodnyi]

On Wed, Sep 29, 2010 at 8:27 PM, Christopher Brannon
<cmbrannon79 at gmail.com> wrote:
> Andrei Kholodnyi <andrei.kholodnyi at gmail.com> writes:
>
>> you mean , w/o make check, make installcheck fails?
>
> Exactly.

would it be possible you fix it, or should I drop a look?

>> yes, every make *check, e.g. make distcheck will run tests automatically.
>
> It would be nice if we could disable them for make distcheck.

yes, exactly this question I have asked on the autotool mail list,
and they told me it is an intentional behavior. :D

"Well, distcheck (which comes from Automake) really aims to be a "come
on, let me ensure this package is good in all kinds of ways" target,
so it also tries out installcheck. "

they have proposed to hook it /autotest/ to another rule than check.

"If your 'installcheck' is not generally usable, then I suggest just not
hooking the autotest testsuite execution to it (i.e., omit the
installcheck-local rule).  If it is generally usable, then I wonder why
it shouldn't work in the distcheck setting for you."

> The unfortunate thing is that the tests produce a large quantity of
> spoken output. ?There's no way to verify it automatically.
> I suppose we could write a "null" module, for testing purposes.
> The null module would simply log everything that it receives, without
> speaking anything. ?We could then compare the log against a file that
> describes expected behavior.
> Obviously, this would not test the synthesizers, but it would be perfect
> for testing libraries, client-server communication, and server-module
> communication.

why not? what we can do is following:
record expected output using synth directly.
then compare it with produced output by spd using a plugin with the same synth,
but redirecting e.g. pulse output to file.
then compare both files.

> For now, I suspect that the best way to test synthesizers is by
> listening. ?Yes, we could come up with a scheme for recording audio from
> an output module, and then compare the output against a sample file.

exactly!

> However, seemingly insignificant changes to the underlying TTS engine could
> cause a bit-by-bit comparison to fail. ?E.G., inserting a pause, or improving
> the pronunciation of a word. ?I think we'd have to use some sort of
> heuristic. ?Are there any good methods of comparing audio clips for
> similarity?

I'm pretty sure there is something available. Let us drop a look...
the most preferable way is to push both to ASR and make text out of it
for strcmp :D

Andrei.