Re: [pkg.m] Load order of dependencies

[Philip Nienhuis]

Kai Torben Ohlhus wrote:
> Dear all,
>
> Since following the work on GNU Octave in 2013, I always notice a high
> tension when it comes to the pkg implementation and the organization of
> Octave Forge itself.  My humble perspective is, that the current
> implementations do a good enough job so far and there is lack of
> maintenance power in both Octave Forge and the pkg tool, which might
> stress developers as well, who spend time and effort in improvements:

Very true.

> On 12/10/19 5:39 AM, PhilipNienhuis wrote:
> > Lately I did some work on pkg.m so I got some feeling for how it
> sticks together.
>
> On 12/10/19 5:45 AM, Andrew Janke wrote:
> > Also, anybody who cares about this, come on by Packajoozle and give
> > suggestions if you want. https://github.com/apjanke/octave-packajoozle
> > Packajoozle loads its packages in dependency order (dependents before
> > dependers) for this reason.
>
>
> The original question by Juan was for the scenario:
>
>     A depends on {B depends on {D, E}, C}
>
> if the loading order by pkg should be (and in pkj is)
>
>     1. Load {D, E} undetermined order; functions to top of loadpath
>     2. Load {B, C} undetermined order; functions to top of loadpath
>     3. Load A                        ; functions to top of loadpath
>
> but the exact opposite seems to be implemented in pkg, right?  To me the
> stated order above seems more "natural" as well.

Sure. But as I wrote, pkg.m was written with initialization of packages 
using dependencies in mind and somewhere in this thread it was mentioned 
that then you need the other way round.
In L.46 of <pkg>/private/load_packages.m there is this comment:
  ## Load the packages, but take care of the ordering of dependencies.

I think in this thread it isn't very clear that the current Octave-Forge 
setup never took deliberate shadowing into account.
Until now shadowing was mainly a nuisance when OF functions were 
absorbed in core and the remaining OF package wasn't properly adapted to 
the new situation.
Now JuanPi has the wish to explicitly shadow a function in another OF 
package that is also a dependency. AFAIK that is new.

> On 12/12/19 10:26 AM, Juan Pablo Carbajal wrote:
> > As said many times pkg.m should be replaced [...] I am not convinced
> > pkj does a much better job, but it at least enjoys a more maintainable
> > structure.
>
> @Juan, did you use pkj and can confirm that it cannot handle your
> loading issue?  The dependency between geometry [1] and matgeom [2] will
> occur in the upcoming release [3], right?
>
>
> The next aspect that came as an argument for the current pkg implementation:
>
> On 12/11/19 1:57 AM, PhilipNienhuis wrote:
> > Moreover, dependencies aren't always so "linear". ISTR there were
> > (or are?) circular dependencies.
>  
> Is this really somewhere the case, or theoretical consideration?
> Personally, I agree with Andrew in this respect:

IIRC there were (years ago), like pkg A -> pkg B - > pkg C -> pkg A; but 
I never checked back.

One of the weak points in our current package system is that there's no 
"authority" overlooking dependency chains. Package maintainers have all 
freedom and I'm not sure that every maintainer carefully checks existing 
dependency trees before adding one to her/his package.
Equally weak is that a package maintainer can at her/his will remove 
functions that other packages rely on, thus breaking a dependency. I'm 
equally unsure that all package maintainers check for this, and AFAIK 
this isn't checked by forge admins either.

> On 12/11/19 5:55 AM, Andrew Janke wrote:
> > Circular dependencies; ouch.
>
> Thus I would not stress this, if it is an exceptional/non-occuring case
> in current OF packages.

I think (hope) it was an exception.

My other argument is mutually exclusive dependencies. That may also be 
an exception, but has happened as well.

I remember that at one time I thought I was in something looking very 
much like Windows' "dll hell" - I tried to update or install some 
package and next I know the rest of the evening I was occupied getting 
other packages updated/installed. Of course my mistake was that I tried 
to do it neatly but in hindsight I should have used the "nodeps" flag 
right away.

The whole point is that -as outlined above- there's no sound or robust 
dependency chain management whatsoever.

> @Philip, do you currently spend effort in improving the current pkg
> function, or should pkj by Andrew be a favorable long term replacement?
>   Agreeing on this might save time and effort.

I was just fixing bugs that were in the way and that were relatively 
easy to solve.
Other than that I have no preference.

But: is pkj compatible with all existing packages? or do existing 
packages need to adapt DESCRIPTION entries and/or other things?

Philip