Re: [Orgmode] Re: Testing --- again...

[Sebastian Rose]

"Eric Schulte" <address@hidden> writes:
> Hi Sebastian,
>
> Sebastian Rose <address@hidden> writes:
>
>> "Eric Schulte" <address@hidden> writes:
>>> Hi,
>>>
>>> This is exciting.
>>>
>>> Rather than impose a complete directory/layout schema before-hand I'd
>>> lean towards starting with a little more chaos and then letting the
>>> structure of the test directory develop /naturally/.  From the
>>> discussion below it sounds like an initial structure of
>>>
>>> testing/lisp/
>>> testing/contrib/lisp/
>>
>>
>> I believe it makes sense enforcing rules.  Many developers plus power
>> users will want to be able to use the test system.  No system is what we
>> had in the past.
>>
>> The idea is, to have a system to automate the laoding of tests.  How
>> should a function like `org-test-test-current-defun' find the tests
>> otherwise?
>>
>>   sh$ cd org-mode
>>   sh$ find . -name '*.el' | wc -l
>>   146
>>
>> Also, we could provide services like setup temporary directories,
>> buffers and files for tests.  This cannot be automated in a save way, if
>> there is no structure.  The tests are written in elisp.  Hence one could
>> do whatever he likes ;)
>>
>> It's like Perl.  You don't need to follow the conventions, but it will
>> make your live easier (hopefully).
>>
>> Just what I think.
>>
>
> Point well taken, I suppose my point is more a matter of personal taste,
> and I fully understand if we disagree and would yield to your judgment
> as you've put more thought into this.  But having said that here's my
> thoughts and opinions :)
>
> Could we just load every test in tests/lisp by default, and then use the
> existing `ert' selection method to select and run tests.  For example if
> we enforce our conventions on the level of test function name rather
> than on file name, we could (I believe) do something like the following.
> Say every test for a particular function (say org-export) includes that
> name of that function in the test name (say test-org-export), then we
> could use something simple like the following to run the tests for the
> current function (i.e. function around the point)
>
>
> (defun org-test-current-function ()
>   "Test the current function."
>   (ert (format "%S" (which-function))))


I have no function called `which-function' !?

But yes, those prefixes are (and have to be) part of our "standard".
The entire Emacs elisp structure is build on that way of "namespacing".

It's basically working like that (but I used more lines to get to the
function name...).

All this is _not_ meant to get in the way of ERT.  I just hope a) to add
some support to get people started and b) provide some help and
maintainance of that testing stuff.


E.g. `org-test-run-tests (&optional selector)'  runs tests enclosed in 

  (let ((deactivate-mark nil))
    (save-excursion
      (save-match-data
      ;; ... run tests here
      )))

But yes, I guess the art is to do not too much.


> This way we could maintain a much simpler directory structure inside of
> tests/ (or testing/) in which we don't need a separate file name for
> every function, but rather maybe one test file per elisp file
> (e.g. test-ob.el for ob.el), and possibly other files for tests
> organized around concepts that span multiple files (e.g. test-blocks.el
> or somesuch).


Hmmmm - I see.  But you'd still test single units of code, and a single
unit of code lives in one source file, not in many source files.  Even
though many source files are involved, which is always the case when you
execute code.


> I may well be misunderstanding the framework you are proposing, so maybe
> the best thing to do is to just get started and then see how things
> develop.

OK. Same here :)

I just pushed my little starter to a new branch "ert-testing".  Pooha,
git always a little challenging in such rare situations :)


> BTW: back when I worked on ruby-on-rails projects I developed jump.el
> [1] for jumping between functions and their tests, I could probably
> fairly easily apply this to the org-mode repo if that's desirable.


That's such a helpful little tool I think of!  I'll check that, too :)


I just created the branch ert-testing and pushed my little starter.


>>> may make sense, reserving the top level for "meta" testing stuff, like
>>> functions for running tests, common fixtures, example files, etc...
>>>
>>> I have two questions.
>>>
>>> 1) while waiting for ert to be included into Emacs, should we include an
>>>    ert distribution as part of the Org-mode repository (maybe using git
>>>    sub-modules) or should we just agree that users should have a certain
>>>    version of ert installed locally?  I'm honestly not sure which of
>>>    these options sounds preferable.
>>
>>
>> I thought about this, too.  I guess not.  Developers and users that want
>> to test will be able to follow the current ERT git repo.
>>
>> But ERT is just 7 *.el files plus 1 texinfo file.
>>
>> An what I don't know is:
>>
>>    How would git submodules work?
>>
>
> using git submodules we could specify a location (e.g. tests/ert) and a
> version (some particular git commit) for the org-mode repository.  Then
> running
>
>   git submodule init
>   git submodule update
>
> would checkout the appropriate version of ert to that location.  Users
> who don't want to run tests could just never run the two above commands
> and shouldn't notice any difference aside from a .gitmodules file in the
> base of their org-mode repo.


Wow.  That sounds _very_ good.  From what I read on emacs-devel,
Christian Ohler is going to adjust the code of ERT to get it into Emacs'
core.  Using this technique, we would ensure that the tests work, even
though ERT has changed.  We'd just need to "update" the sub project and
all developers will pull the right thing then?

That sounds absolute fantastic!  It's the best way to do it I came
across so far :)





   Sebastian



>>> 2) should the initial population of the testing/ directory take place in
>>>    a separate branch of the repository or in the master branch?  Again I
>>>    don't know which I would prefer, branches add complication but could
>>>    result in cleaner commit histories.
>>
>>
>> I'll start on a branch first and constantly rebase as long as the
>> structure evolves.  The first simple commit will be what you can see on
>> github, with some doc strings adjusted.
>>
>>
>
> Great, when I get a chance I'll check it out and try to write a couple
> of simple Babel tests.
>
> Cheers -- Eric





>>
>>
>>    Sebastian
>>
>>   
>>
>>> Best -- Eric
>>>
>>> Carsten Dominik <address@hidden> writes:
>>>
>>>> Hi Sebastian,
>>>>
>>>> the lack of a testing suite for Org-mode is really frustrating,
>>>> and even more frustrating is that we have had like seven attempts
>>>> to start one, and each of these lead to nothing.  So I would
>>>> be perfectly happy to give a free hand, write access to the repo
>>>> and a full directory in the distribution to implement one.
>>>> Once there is a framework, I am sure many people would be
>>>> willing to contribute tests.
>>>>
>>>> More comments below.
>>>>
>>>> On Oct 2, 2010, at 5:51 AM, Sebastian Rose wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>> I thought about testing again recently.  This is something, that never
>>>>> really got started.  For a reason:  there's no framework for testing.
>>>>>
>>>>> I therefore wrote a very rough proposal,  found  on
>>>>> http://github.com/SebastianRose/org-test
>>>>>
>>>>> The idea is, to provide two simple commands:
>>>>>
>>>>>
>>>>>  *  org-test-test-current-defun
>>>>>     will search for tests for the defun point is in or behind
>>>>>     (`beginning-of-defun') and execute them surrounded by
>>>>>
>>>>>  (let ((select (or selector "^org"))
>>>>>           (deactivate-mark nil))
>>>>>    (save-excursion
>>>>>      (save-match-data
>>>>>
>>>>>
>>>>>  *  org-test-test-buffer-file
>>>>>     will search for tests for the entire file and execute them the
>>>>> same
>>>>>     way.
>>>>
>>>> FIrst:  I have *no* clue about testing.
>>>>
>>>> Second, I am surprised that you want to structure it by function.  I
>>>> would have
>>>> thought that it could be structure by file at the most.  And then
>>>> there will
>>>> be tests that involve code from many files.
>>>>
>>>> But I guess
>>>>
>>>>>
>>>>> If you use one of these commands, all currently registered ERT tests
>>>>> are
>>>>> deleted, and files are reloaded (since you're likely to work on the
>>>>> tests, too).  To repeat the tests without reloading, you will use the
>>>>> ERT commands like `ert-results-rerun-all-tests', bound to `r' in the
>>>>> ERT
>>>>> results buffer.
>>>>>
>>>>>
>>>>>
>>>>> I choose ERT (git clone http://github.com/ohler/ert.git) because
>>>>> that's
>>>>> likely to go into Emacs core (or elpa.gnu.org).
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The idea is to search the directory structure from the current source
>>>>> file upwards for a directory named "tests/" if it exists.  Else ask
>>>>> the
>>>>> user.  Similar to what `add-change-log-entry' does.
>>>>>
>>>>> Below that directory, a tree like the source tree exists:
>>>>>
>>>>> project
>>>>>   +-- lisp/
>>>>>   |     +-- a.el
>>>>>   |     `-- b/
>>>>>   |         +-- b.el
>>>>>   |
>>>>>   `-- tests/
>>>>>         +-- a.el/
>>>>>         |     +-- tests.el
>>>>>         |     `-- a-defun.el
>>>>>         `-- b/
>>>>>             +-- b.el/
>>>>>                   +-- tests.el
>>>>>                   `-- b-defun.el
>>>>>
>>>>> If this setup exists, when editing defun-x in lisp/a.el,
>>>>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>>>>> (fallback: tests.el there) and execute all tests with selector
>>>>> "^a-defun".
>>>>
>>>> Well, OK, this is fine.  But under a.el and b.el there should also be
>>>> general tests that are not function dependent, and there should be a
>>>> place
>>>> to put tests that you do not want to assign to a specific file.
>>>>
>>>> We do have a "testing" directory already, you can use that.
>>>> I would prefer the tests to be in testing, not in lisp/testing
>>>> if possible. I would like to have the lisp directory contain
>>>> only code.  If possible.
>>>>
>>>> It would be OK to have a lisp subdirectory in testing,
>>>> just as it would be OK to have contrib/lisp in testing
>>>> for the contributed packages.
>>>>
>>>> PLEASE, go ahead.  I do not think you have write access
>>>> yet on repo - give me your user name and I'll activate you.
>>>>
>>>> - Carsten
>>>>
>>>>> `M-x org-test-test-buffer-file' in that same source file will load all
>>>>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>>>>
>>>>>
>>>>> Thus tests for
>>>>>    org-mode/lisp/org-protocol.el
>>>>> will be searched in the directory
>>>>>    org-mode/tests/lisp/org-protocol.el/*.el
>>>>>
>>>>>
>>>>> Once the basic route of testing is clear, I'd like to "translate" the
>>>>> existing tests for org-html.el to work with ERT, which will involve
>>>>> writing more tools (create output buffers, compare output with control
>>>>> files using ediff etc.).  I know Lennart Borgman has wrote that stuff
>>>>> for nXhtml already.  I hope we can use his stuff and help here.
>>>>>
>>>>> The directory org-mode/lisp/tests/ would not need to be part of the
>>>>> "official" Org mode package.  It could as well be checked out
>>>>> separately, if "tests" is part of org-mode/lisp/.gitignore (e.g.).
>>>>>
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>  Sebastian
>
> Footnotes: 
> [1]  http://github.com/eschulte/jump.el