Re: Python's native-inputs

[Nicolas Graves]

Answer in two different emails.

 On 2024-04-18 22:07, Maxim Cournoyer wrote:

> Hi Nicolas,
>
> Nicolas Graves via "Development of GNU Guix and the GNU System
> distribution." <guix-devel@gnu.org> writes:
>
>> Hi Guix,
>>
>> On some languages, there are a lot of unused native-inputs that are
>> development & linting dependencies much more than packages that are
>> actually used to build or test a package (I'm thinking at least Python
>> and Rust). These fall in the category of tools "useful" at run time, but
>> unecessary at build time.
>
> Indeed.

One more argument: it's important for user experience using python to
have these packages up-to-date, because users will want to have an
up-to-date lsp client and utilities.

>
>> Is there a clear policy about their removal? I've seen the case of
>> pre-commit in Python, and I've commited a series yesterday regarding
>> pylint, but there are a whole lot of them in Python, at least :
>
> [...]
>
>> These packages make a lot of sense when considering things like
>> `guix shell -D` but they are hampering some progress on Python packages
>> since they are everywhere and a small update in their inputs rebuilds
>> the whole python world (even though it has NO influence on the
>> functionality of any other package).
>>
>> What are the guidelines in this case?
>
> There aren't really any guidelines.  It's easy to avoid the linters, it
> makes sense to avoid them, but sometimes upstream makes it difficult to
> avoid running a linter as part of there test suite, in which case I
> think it's acceptable to keep them as native-inputs rather than come up
> with more patches to maintain.
>
>> I can propose a huge patch series (currently ~300 patches, and not
>> finished), to remove them, lint against them and remove them from the
>> importer as a default, but that's a big decision to make. IMO we should
>> have a dev-inputs field to handle these cases, but that's even more work.
>
> The situation is similar as for other test inputs such as pytest; it has
> an enormous amounts of dependents and thus cannot be easily upgraded on
> the master branch.  At least more up-to-date variants can be added since
> these are not in the transitive closure of any packages.

 That's true for some very essential packages (setuptools, pytest...)
 which would make the case to only update these packages once we want to
 update the whole python chain, i.e. when we update python. But IMO it
 shouldn't be the case for a package like black, isort or flake8.

>
> I don't think we should go out of our way to address annoyances that
> upstream have caused -- that'd be too much work to maintain in the long
> run.

 I think it depends on the case. For a package like python-dateutil,
 having pytest-cov as a native-input is bad, because it causes a rebuild
 of the whole chain (guix refresh -l python-pytest-cov went from ~2500
 to ~15 packages with the whole series, but in particular that was a big
 part of dependency removal). Now for small leaf packages, having
 pytest-cov as a native-input is not a big issue indeed, I agree in
 those cases we would better ignore it than fix it in the meantime.

 While making the patches, I realized it's not often that we need a
 convoluted patch. Most of the time, adding "-c" "/dev/null" to
 test-flags will do the trick to remove python-pytest-cov and it's even
 less annoyance for black, flake8 or mypy, if not other cases are often
 handled with a simple removal of the native-input in pyproject.toml, or
 setup.py. What's annoying is when packages are flagged as required when
 they're not.

 It could possibly even be handled in the build-system in the case of
 pytest, to avoid these "-c" "/dev/null". This is untested, but I think
 it'll work.
 
;; Add a pytest plugin for the pyproject-build-system to filter out some
;; arguments we want to be able to ignore.
;; (define (pytest-ignore-options group-options-alist)
;;   "This function converts an alist of (group . options) in a string that can
;; be instantiated as a python pytest plugin."
;;   (string-join
;;    (cons*
;;     "import pytest"
;;     (map
;;      (match-lambda
;;        ((group . options)
;;         (string-append
;;          (format #f "def pytest_addoption(parser):
;;     group = parser.getgroup(~s,'Guix ignored options')
;;     options = [~{~s, ~}]
;;     for option in options:
;;         group.addoption(option, action='count')"
;;                  group
;;                  options))))
;;      group-options-alist))
;;    "\n\n"))

;; (let ((cov-alist
;;        '(("cov" . ("--cov" "--cov-reset" "--cov-report" "--cov-config"
;;                    "--no-cov-on-fail" "--no-cov" "--cov-fail-under"
;;                    "--cov-append" "--cov-branch" "--cov-context")))))
;;   (display (pytest-ignore-options cov-alist)))

;; TODO Instantiate this file, add it to the check phase through the 
;; PYTEST_PLUGINS environment variable (possibly also PYTHONPATH or 
GUIX_PYTHONPATH ?)
;; reference: 
https://docs.pytest.org/en/7.1.x/how-to/writing_plugins.html#plugin-discovery-order-at-tool-startup
;; Using the environment variable should also set the right precedence :
;; if pytest-cov is in native-inputs, then use it; if not, ignore --cov* 
options.

 I'm not sure we can do the same for packages though, it seems that the
 proper is not easy to grasp.

 But in a near future, I wouldn't be surprised if we can tweak the
 pyproject-build-system to simply read the configuration and remove
 unwanted packages when they are present. It's definitely not available yet
 and my experiments didn't work, but python-pypa-build is headed in that
 direction for some of its utilities, and python-3.12 seems to also
 change the bootstrap order of its packages (python-pypa-build will be
 easier to bootstrap).

 TL;DR :
 - patch series in big progress, not done yet because I don't really
 know where to stop and massive rebuilds.
 - WDYT about tweaking the build-system for pytest?
 - not done : tweaking the pypi import to ignore those packages. I've
 done it in the juliahub import code, but I haven't delved into that
 though.
 - not done : adding those packages to guix lint - trivial. We should
 probably add a variable in pypi import that is exported and added into
 guix lint.

 Acting on the three (guix import, guix lint, guix build) can make it
 quite easy on maintainers in the end IMO.
 
-- 
Best regards,
Nicolas Graves