Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user

[Eric Schulte]

Achim Gratz <address@hidden> writes:

> Eric Schulte <address@hidden> writes:
>> Moving forward on this point, many of the existing tests explicitly
>> `require' new Org-mode functionality (mainly language support for
>> testing code blocks execution).  I do not think that tests should ever
>> be activating new packages changing a users global environment.
>
> Thanks.
>
> I've had to deactivate the inclusion of ./testing/lisp/contrib into the
> test flow due to a recent removal of said directory by Nicolas (he's
> probably doing that patch himself) and now get this:
>

Yes,

I talked with Nicolas this morning, and at my urging he moved everything
from ./testing/lisp/contrib into ./testing/lisp and removed the contrib
directory.  I think moving forward we are best served by a single test
directory, especially given that we already have good methods of
conditionally loading only those tests appropriate for the users
environment.

>
> lisp/org-mode> make test-dirty |& grep failed
>    failed    5/109  ob-exp/export-from-a-temp-buffer
>    failed   17/109  org-missing-dependency/test-ob-C
>    failed   18/109  org-missing-dependency/test-ob-R
>    failed   19/109  org-missing-dependency/test-ob-awk
>    failed   20/109  org-missing-dependency/test-ob-fortran
>    failed   21/109  org-missing-dependency/test-ob-lilypond
>    failed   22/109  org-missing-dependency/test-ob-maxima
>    failed   23/109  org-missing-dependency/test-ob-octave
>    failed   24/109  org-missing-dependency/test-ob-python
>    failed   25/109  org-missing-dependency/test-org-element
>    failed   26/109  org-missing-dependency/test-org-export
>
> This is expected, as my local configuration doesn't have any of these
> packages activated.
>

Yes, I assume that the lower case "failed" means expected failure.

>
> lisp/org-mode> make test-dirty | & grep FAILED
>    FAILED   27/109  test-ob-exp/org-babel-exp-src-blocks/w-no-file
>    FAILED   62/109  test-org-babel/get-src-block-info-body
>    FAILED   63/109  test-org-babel/get-src-block-info-language
>    FAILED   64/109  test-org-babel/get-src-block-info-tangle
>
> This is unexpected and I don't really see where they are coming from.
> The backtrace always starts at (regexp-quote org-test-file-ob-anchor).
>

My guess is that when I removed require statements for org-html and
org-ascii from from test-org-html and test-org-exp, they removed some
functionality not loaded by default in the test scripts.  These should
probably be added into the Makefile.

>
>> For this reason I have just pushed up a commit which changes all
>>
>>   (require 'org-foo)
>>
>> to
>>
>>   (unless (featurep 'org-foo)
>>     (signal 'missing-test-dependency "Org support for doing foo."))
>>
>> so that those tests simple aren't run on the users system.  Please let
>> me know if anyone thinks this is a mistake and we can discuss.  The only
>> drawback I see is that batch-mode scripts will have to explicitly
>> activate the features which they would like to test, by evaluating forms
>> like (require 'org-foo) before the call to the test suite -- and I would
>> argue that being explicit about such things is a benefit.
>
> Well, currently all testing runs under my user settings, which is not
> ideal for testing anyway.  I suppose it would not be difficult to just
> inject a different (minimal) startup file specifically for testing to
> separate these issues.
>

Yea, I also run most tests from my normal Emacs session, which is part
of the motivation for removing these embedded require statements.

Cheers,

>
>
> Regards,
> Achim.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/