Re: [Duplicity-talk] Duplicity-talk Digest, Vol 361, Issue 2

[Arjun]

I have faced this issue before, and fixed it by setting 

PYTHONUSERBASE=/usr/local

and if necessary, the PYTHONPATH. These would of course depend on where pip is 
installing the packages.

Quoting duplicity-talk-request@nongnu.org (2023-03-08 12:00:59)
> Send Duplicity-talk mailing list submissions to
>         duplicity-talk@nongnu.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> or, via email, send a message with subject or body 'help' to
>         duplicity-talk-request@nongnu.org
> 
> You can reach the person managing the list at
>         duplicity-talk-owner@nongnu.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Duplicity-talk digest..."
> 
> 
> Today's Topics:
> 
>    1. Re: Failures on backup jobs from cron (Russell Clemings)
>    2. Re: Failures on backup jobs from cron (edgar.soldin@web.de)
>    3. Re: Failures on backup jobs from cron (Kenneth Loafman)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 7 Mar 2023 14:03:58 -0800
> From: Russell Clemings <rclemings@gmail.com>
> To: Kenneth Loafman <kenneth@loafman.com>
> Cc: Discussion about duplicity backup <duplicity-talk@nongnu.org>,
>         edgar.soldin@web.de
> Subject: Re: [Duplicity-talk] Failures on backup jobs from cron
> Message-ID:
>         <CAN7RX0b7_QhnwkNBTGfc3buDHELCV+KogK0Xs6Jaz9JaetO+RQ@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Answers below.
> 
> On Tue, Mar 7, 2023 at 7:32 AM Kenneth Loafman <kenneth@loafman.com> wrote:
> 
> > Hi Russel,
> >
> > 1) Which version(s) of duplicity, Python, OS are you using?  Which work
> > and which don't.
> >
> 
> Duplicity 1.2.2, Python 3.8.3, CentOS v7.9.2009 STANDARD virtuozzo with
> cPanel 106.0.18
> 
> 
> >
> > 2) As to your problem with the cryptography module install, try:
> >
> >     $ python3 -m pip install --U cryptography::38.0.4
> >
> >     or ignore it completely if you are not using azure backend.
> >
> 
> I don't have azure but tried it anyway: ERROR: Invalid requirement:
> 'cryptography::38.0.4'
> 
> (Should "--U" be "-U"? I got "unknown option" with "--U".)
> 
> 
> >
> > 3) On the cron job, make sure all paths are set properly in crontab, and
> > make sure you don't have duplicate duplicity instances:
> >
> >     $ find /usr -name duplicity -type f
> >     $ find /usr -name duplicity -type d
> >
> 
> I do get multiple versions, but I've always included the full path in my
> script, e.g.:
> 
> crontab: /home/newton/duplicity-0.8.12.1612/s3backup/backup.sh
> backup.sh: /home/newton/duplicity-0.8.12.1612/bin/duplicity
> 
> 
> >     If you have more than one response from above, uninstall all, and
> > reinstall the one you want.
> >
> 
> I'm kind of afraid to uninstall anything until I can get a replacement
> working. Right now 0.8.12.1612 still works from the command line.
> 
> >
> > 4) If you installed from source, just remember that the source is
> > non-versioned.  We use the last tag as the version during release.
> >     There are multiple options both stable and latest:
> >
> 
> I did install from source. Maybe that's the problem.
> 
> Before I gave up on fixing the 0.8.12.1612 installation (which as I said
> still works from the command line, just not from cron), I was getting this
> error when running from the root cron:
> 
> ImportError: cannot import name '_librsync'
> 
> So I'm stuck. Unless there are other suggestions I guess I will try one of
> the options below and hope for the best. What's bewildering is everything
> had been working fine for several years, until it started failing on two of
> four servers a couple of weeks ago.
> 
> 
> >    - Stable snap builds - “sudo snap install duplicity —classic"
> >    - Latest snap builds - “sudo snap install duplicity —classic —edge"
> >    - Stable pip3 builds - “sudo python3 -m pip install duplicity"
> >    - Latest pip3 builds - “sudo python3 -m pip install --pre duplicity"
> >    - Stable duplicity PPA -
> >    
> > https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git
> >    
> > <https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-develop-git>
> >    - Latest duplicity PPA -
> >    
> > https://code.launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-develop-git
> >    - Stable tarball install - https://launchpad.net/duplicity/+download
> >    - Source - https://gitlab.com/duplicity/duplicity
> >
> >     Since the snap versions have all the pypi stuff resolved that may be
> > your best bet.
> >
> > I hope some of this helps.
> >
> > ...Thanks,
> > ...Ken
> >
> >
> >
> >
> > On Mon, Mar 6, 2023 at 4:19 PM Russell Clemings via Duplicity-talk <
> > duplicity-talk@nongnu.org> wrote:
> >
> >> OK, well, I gave up. Everytime I tracked down and included a dependency
> >> (via PYTHONPATH), I'd just get another failure from another missing
> >> dependency.
> >>
> >> So I decided that if I was going to spend that much time on it, I might
> >> as well just start fresh with the current version, following the
> >> instructions here:
> >>
> >> https://gitlab.com/duplicity/duplicity/-/blob/main/README.md
> >>
> >> It seems to have installed OK, except for an error with pip3 install -r
> >> requirements.txt:
> >>
> >> ---
> >>
> >> Collecting cryptography>=2.1.4 (from azure-storage-blob->-r
> >> requirements.txt (line 18))
> >>   Using cached
> >> https://files.pythonhosted.org/packages/fa/f3/f4b8c175ea9a1de650b0085858059050b7953a93d66c97ed89b93b232996/cryptography-39.0.2.tar.gz
> >>     Complete output from command python setup.py egg_info:
> >>
> >>             =============================DEBUG
> >> ASSISTANCE==========================
> >>             If you are seeing an error here please try the following to
> >>             successfully install cryptography:
> >>
> >>             Upgrade to the latest pip and try again. This will fix errors
> >> for most
> >>             users. See:
> >> https://pip.pypa.io/en/stable/installing/#upgrading-pip
> >>             =============================DEBUG
> >> ASSISTANCE==========================
> >>
> >>     Traceback (most recent call last):
> >>       File "<string>", line 1, in <module>
> >>       File "/tmp/pip-build-g7v1tzdv/cryptography/setup.py", line 18, in
> >> <module>
> >>         from setuptools_rust import RustExtension
> >>     ModuleNotFoundError: No module named 'setuptools_rust'
> >>
> >>     ----------------------------------------
> >> Command "python setup.py egg_info" failed with error code 1 in
> >> /tmp/pip-build-g7v1tzdv/cryptography/
> >>
> >> ---
> >>
> >> Upgrading pip as suggested didn't do anything:
> >>
> >> ---
> >>
> >> xxxxxx@xxxx2 [~/duplicity-main]# python3 -m pip install --upgrade pip
> >> Defaulting to user installation because normal site-packages is not
> >> writeable
> >> Requirement already satisfied: pip in
> >> /usr/local/python3.8/lib/python3.8/site-packages (23.0.1)
> >>
> >> ---
> >>
> >> Finally when I try to run a backup script, I get what I think are some
> >> boto errors:
> >>
> >> ---
> >>
> >> Attempt of list Nr. 1 failed. ClientError: An error occurred (403) when
> >> calling the HeadBucket operation: Forbidden
> >> Attempt of list Nr. 2 failed. AttributeError: 'NoneType' object has no
> >> attribute 'objects'
> >> Attempt of list Nr. 3 failed. AttributeError: 'NoneType' object has no
> >> attribute 'objects'
> >> Attempt of list Nr. 4 failed. AttributeError: 'NoneType' object has no
> >> attribute 'objects'
> >> Giving up after 5 attempts. AttributeError: 'NoneType' object has no
> >> attribute 'objects'
> >>
> >> ---
> >>
> >> Suggestions?
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Feb 27, 2023 at 2:51 PM Russell Clemings <rclemings@gmail.com>
> >> wrote:
> >>
> >>> Thanks. I'll try these and report back.
> >>>
> >>> On Mon, Feb 27, 2023 at 2:03 PM edgar.soldin--- via Duplicity-talk <
> >>> duplicity-talk@nongnu.org> wrote:
> >>>
> >>>> such issues usually are caused by cron not setting PATH env var  or
> >>>> wrongly. in my experience ,)
> >>>>
> >>>> to debug it, run 'which python3'
> >>>> 1. in cron. pipe to file or logger to retrieve the output
> >>>> 2. as root
> >>>> comparing both ourputs you will probably see that different python
> >>>> interpreters are used, with one not having the missing modules installed 
> >>>> in
> >>>> it's default location
> >>>>
> >>>> resolve by adding the PATH env var of the root shell at crontab top.
> >>>>
> >>>> good luck.. ede/duply.net
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On February 27, 2023 9:48:24 PM GMT+01:00, Scott Hannahs via
> >>>> Duplicity-talk <duplicity-talk@nongnu.org> wrote:
> >>>>>
> >>>>> You may need to set the PYTHONPATH variable when running from Cron or
> >>>>> other non-shell systems.
> >>>>>
> >>>>> https://www.geeksforgeeks.org/pythonpath-environment-variable-in-python/
> >>>>>
> >>>>> -Scott
> >>>>>
> >>>>> On Feb 27, 2023, at 14:37, Russell Clemings via Duplicity-talk <
> >>>>> duplicity-talk@nongnu.org> wrote:
> >>>>>
> >>>>> I've got a curious problem suddenly with duplicity on two of more four
> >>>>> servers.
> >>>>>
> >>>>> If I run a backup job as root, everything works fine. If it runs from
> >>>>> the root cron, I get this:
> >>>>>
> >>>>> First server:
> >>>>> Traceback (most recent call last):
> >>>>>   File "/home/xxxx/duplicity-0.8.12.1612/bin/duplicity", line 30, in
> >>>>> <module>
> >>>>>     from future import standard_library
> >>>>> ModuleNotFoundError: No module named 'future'
> >>>>>
> >>>>> Second server:
> >>>>> Traceback (most recent call last):
> >>>>>   File "/home/xxxx/duplicity-0.8.12.1612/bin/duplicity", line 39, in
> >>>>> <module>
> >>>>>     import fasteners
> >>>>> ModuleNotFoundError: No module named 'fasteners'
> >>>>>
> >>>>> Both modules are installed, and as I said, the backups run fine from
> >>>>> the command line. Also, it works fine from cron on two other servers. 
> >>>>> (And
> >>>>> I realize my duplicity is a few versions behind; updating is on my to-do
> >>>>> list.)
> >>>>>
> >>>>> Any pointers to help locate the problem are appreciated.
> >>>>>
> >>>>> --
> >>>>> ===============================================
> >>>>> Russell Clemings
> >>>>> <russell@clemings.com>
> >>>>> ===============================================
> >>>>> _______________________________________________
> >>>>> Duplicity-talk mailing list
> >>>>> Duplicity-talk@nongnu.org
> >>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>> Duplicity-talk mailing list
> >>>> Duplicity-talk@nongnu.org
> >>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >>>>
> >>>
> >>>
> >>> --
> >>> ===============================================
> >>> Russell Clemings
> >>> <russell@clemings.com>
> >>> ===============================================
> >>>
> >>
> >>
> >> --
> >> ===============================================
> >> Russell Clemings
> >> <russell@clemings.com>
> >> ===============================================
> >> _______________________________________________
> >> Duplicity-talk mailing list
> >> Duplicity-talk@nongnu.org
> >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >>
> >
> 
> -- 
> ===============================================
> Russell Clemings
> <russell@clemings.com>
> ===============================================
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> <https://lists.nongnu.org/archive/html/duplicity-talk/attachments/20230307/d9c07c1c/attachment.htm>
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 8 Mar 2023 12:57:27 +0100
> From: edgar.soldin@web.de
> To: Discussion about duplicity backup <duplicity-talk@nongnu.org>
> Subject: Re: [Duplicity-talk] Failures on backup jobs from cron
> Message-ID: <8013166d-d5b5-ab7c-e86b-b1ce26aaef3b@web.de>
> Content-Type: text/plain; charset=UTF-8; format=flowed
> 
> On 07.03.2023 23:03, Russell Clemings wrote:
> >     3) On the cron job, make sure all paths are set properly in crontab, 
> > and make sure you don't have duplicate duplicity instances:
> >
> >          $ find /usr -name duplicity -type f
> >          $ find /usr -name duplicity -type d
> >
> >
> > I do get multiple versions, but I've always included the full path in my 
> > script, e.g.:
> >
> > crontab: /home/newton/duplicity-0.8.12.1612/s3backup/backup.sh
> > backup.sh: /home/newton/duplicity-0.8.12.1612/bin/duplicity
> 
> that is irrelevant. installing python apps places parts of them in system 
> python library folders by default. having multiple versions may lead to one 
> version using modules from an incompatible version, hence to errors.
> 
> >          If you have more than one response from above, uninstall all, and 
> > reinstall the one you want.
> >
> >
> > I'm kind of afraid to uninstall anything until I can get a replacement 
> > working. Right now 0.8.12.1612 still works from the command line.
> 
> well, if that one works. remove all of the other ones and fix the cron issue 
> instead?
> 
> ..ede/duply.net
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 8 Mar 2023 09:37:33 -0600
> From: Kenneth Loafman <kenneth@loafman.com>
> To: Discussion about duplicity backup <duplicity-talk@nongnu.org>
> Cc: edgar.soldin@web.de
> Subject: Re: [Duplicity-talk] Failures on backup jobs from cron
> Message-ID:
>         <CAD-_Ma2zXgDdJK0yR65KcBbdWZaSNiXtmZRR0=GzU2Fk+K-uKA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> On Wed, Mar 8, 2023 at 5:58 AM edgar.soldin--- via Duplicity-talk <
> duplicity-talk@nongnu.org> wrote:
> 
> > On 07.03.2023 23:03, Russell Clemings wrote:
> > >     3) On the cron job, make sure all paths are set properly in crontab,
> > and make sure you don't have duplicate duplicity instances:
> > >
> > >          $ find /usr -name duplicity -type f
> > >          $ find /usr -name duplicity -type d
> > >
> > >
> > > I do get multiple versions, but I've always included the full path in my
> > script, e.g.:
> > >
> > > crontab: /home/newton/duplicity-0.8.12.1612/s3backup/backup.sh
> > > backup.sh: /home/newton/duplicity-0.8.12.1612/bin/duplicity
> >
> > that is irrelevant. installing python apps places parts of them in system
> > python library folders by default. having multiple versions may lead to one
> > version using modules from an incompatible version, hence to errors.
> >
> > >          If you have more than one response from above, uninstall all,
> > and reinstall the one you want.
> > >
> > >
> > > I'm kind of afraid to uninstall anything until I can get a replacement
> > working. Right now 0.8.12.1612 still works from the command line.
> >
> > well, if that one works. remove all of the other ones and fix the cron
> > issue instead?
> 
> 
> I'd remove them all and start over with 1.2.2.  It has fewer errors than
> 0.8.12 which is a few years old.
> 
> You can download a versioned tarball of 1.2.2 from here
> <https://launchpad.net/duplicity/1.0/1.2.2/+download/duplicity-1.2.2.tar.gz>.
> The one you got was for developers.
> 
> ...Ken
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> <https://lists.nongnu.org/archive/html/duplicity-talk/attachments/20230308/9104a895/attachment.htm>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Duplicity-talk mailing list
> Duplicity-talk@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> 
> ------------------------------
> 
> End of Duplicity-talk Digest, Vol 361, Issue 2
> **********************************************