Re: [Duplicity-talk] Failure running tests.

[Kenneth Loafman]

Scctt,

An answer on my last email, please.

I'm looking at the log you sent.  Still trying to figure out what's going on.

...Ken

On Thu, Sep 12, 2019 at 4:54 PM Scott Hannahs <address@hidden> wrote:

Ken,

I am willing to leave the unicode errors as just an old OS weirdness.  The others I am a bit concerned that I am not testing correctly or it isn’t passing the test.  This happens both on the old OS and the new OS with Python 3.7.

To take the second failed test (the first failure is a unicode problem in test_selection).

self = <testing.functional.test_replicate.ReplicateTest testMethod=test_replicate>

        basepython = os.environ.get(u'TOXPYTHON', None)

        if basepython is not None:

            cmd_list.extend([basepython])

            cmd_list.extend([u"-m", u"coverage", u"run", u"--source=duplicity", u"-p"])

        cmd_list.extend([u"../bin/duplicity"])

        cmd_list.extend(options)

        cmd_list.extend([u"-v0"])

        cmd_list.extend([u"--no-print-statistics"])

        cmd_list.extend([u"--allow-source-mismatch"])

        cmd_list.extend([u"--archive-dir=testfiles/cache"])

        if current_time:

            cmd_list.extend([u"--current-time", current_time])

        cmd_list.extend(self.class_args)

        if fail:

            cmd_list.extend([u"--fail", u"".__class__(fail)])

        cmdline = u" ".join([u'"%s"' % x for x in cmd_list])

which if I read this right gives me a command line:

...command: "../bin/duplicity" "file://testfiles/replicate_out" "testfiles/restore_out" "--restore-time" "100000" "-v0" "--no-print-statistics" "--allow-source-mismatch" "--archive-dir=testfiles/cache" < /dev/null

And then the error which seems way beyond my python knowledge.  I can find the fsdecode function in os.py but why it is getting a bad argument is beyond me.  I do not know why it should think that it will find the cache directory since that is not in the gzip tar file.  But maybe it was generated in a previous test?  In reading the test language I don’t see that but that may be my limitation.

b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/path.py", line 476, in <lambda>'
b'    util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))'
b"PermissionError: [Errno 1] Operation not permitted: b'testfiles/cache/3f6f90580e457c90e7202bc33cd9e363/duplicity-full-signatures.19700102T034640Z.sigtar.gz'"
b'During handling of the above exception, another exception occurred:'
b'Traceback (most recent call last):'
b'  File "../bin/duplicity", line 157, in <module>'
b'    if u"Forced assertion for testing" in util.uexc(e):'
b'  File "/sw/src/fink.build/duplicity-0.8.04-3/duplicity-0.8.04/duplicity/util.py", line 120, in uexc'
b'    return fsdecode(m)'
b'  File "/sw/lib/python3.7/os.py", line 821, in fsdecode'
b'    filename = fspath(filename)  # Does type-checking of `filename`.'
b'TypeError: expected str, bytes or os.PathLike object, not int'
...return_val: 1

Including the whole output including the build and testing phase.

-Scott

On Sep 12, 2019, at 1:55 PM, Kenneth Loafman <address@hidden> wrote:

Scott,

What C/C++ compiler and libraries?  I'm assuming Xcode.  The core of Python is C/C++, so the libraries could be a problem.

Honestly, unless Homrbrew is installing patches I don't know about, I'm lost as to what is different.

...Ken

On Thu, Sep 12, 2019 at 11:29 AM Scott Hannahs <address@hidden> wrote:

Ken,

Under Mac OS X 10.11.X you can run “ulimit -n 5120” which seems to be sufficient to do the testing phase.  It might be possible to use a smaller number but I haven’t checked.

Under Mac OS 10.14.X running "ulimit -n 5120” will give the error "-bash: ulimit: open files: cannot modify limit: Operation not permitted” if 5120 is more than the current setting.  This was a surprise to me.  So my shell script to run the "setup pytest.py” would fail immediately, since the default is 4096 and I can’t increase to 5120 under the newer Mojave system.  Once I worked around that, I got the errors I sent.

I run the build as a user called fink-bld, this user is not a privileged user.  But this is the user who unpacks the files and owns the entire folder structure where the build and testing are taking place.  This should work and makes sure the build is for a plain vanilla user and not using any of my personal preferences settings and can be built/tested by a non-administrator account.

I have asked that the build directory be kept at the end of testing and there is no folder “testifies” only the zipped version.  However I think the testing suite creates and removes that directory repeatedly.   So having it not exist at the end of testing seems normal.  The pytest goest through all the tests and just records the errors and does not stop.  Was that test supposed to be a case where the cache file does not exist?  There is no cache folder when I extract the testifiles.tar.gz archive manually so not finding a cache is normal and it should be regenerated during testing?

I used “fink” (of course) to install python 3.7, this is installed from source.  And is actually 3.7.4.  These should have the same modules installed on both OS systems.  There may be something in more recently defined unicode glyphs that is incompatible.  At some time, some glyphs (or whatever unicode calls them) have been redefined.  The A with a circle (or angstrom) is an example.  I think there were two different unicode identifiers for these.

-Scott

On Sep 12, 2019, at 12:01 PM, Kenneth Loafman <address@hidden> wrote:

Scott,

I've had access to ulimit from the first.  I'm not sure what you mean.

Who owns the directory where duplicity is testing?  If the testing user can't write to that directory most of the functional tests will fail.

I'm not sure about Python's unicode environment.  How did you install Python?  Was it from binary or source?  If source, how?

...Ken