Re: [GNUnet-developers] Proposal: Make GNUnet Great Again?

[Florian Dold]

>From my experience with working on GNU Taler, I tend to agree with the
arguments *against* multirepos, especially for GNUnet.

Multirepos tend to work well if either:

a) The language you're using has support for project-local package
dependencies.  This is the case for JavaScript (with npm/yarn), and IMHO
a reason why many small repos *can* work well for JS/TypeScript projects
and other languages supporting this approach.  But you do *not* want
developers to build multirepos from source, that's just very painful.
If we are heading towards multirepos with GNUnet, extra custom tooling
will IMHO be inevitable to avoid too much manual building, and I don't
think we want to maintain these custom tools.  An example for such
tooling would be Googles's Repo tool
(https://source.android.com/setup/develop /
https://source.android.com/setup/develop/repo).

b) The sub-projects are loosely coupled with a relatively stable API.
This is (mostly) the case for GNU Taler, where we have (semi-)stable
HTTP APIs, and there are no real source-level dependencies.  There are
exceptions for this (the Taler merchant depends on the headers of the
Taler exchange), but they are so few that it's still manageable.

c) There is some stable set of base-components, and the "ecosystem"
around it is compiled against the stable release version of this
component.  Maybe it can be argued that gnunet-gtk targets the more
stable features of GNUnet, and thus shall be separate?

Neither of these conditions are fulfilled for GNUnet.  I don't really
see what benefit we get from separating out components that have
"distinct use cases" into separate repos.  It doesn't make CI faster
(with BuildBot you can just trigger test cases also based on what files
changed).  Also if you look at FreeBSD, that's an example of a project
where the whole OS (including user space tools) is in one repo.  Why
shouldn't this work for the GNUnet?

I am sure with appropriate tooling, both mono- and multirepos can be a
pleasure to work with.  But as it stands now, moving to multirepos would
negatively impact productivity more than just keeping the monorepo.

Just another random aspect:  Configuration hell.  What if I (as a
developer) want to build my GNUnet component with a certain ./configure
feature/compiler flag enabled/disabled?  Then I have to do this in all
GNUnet dependencies manually, unless there's special tooling to do just
that.

We should also keep in mind that for the end user, it doesn't matter how
the repo(s) are/is structured, they can always get the same set of
packages in their distro.  Multiple source repos can produce a single
package, or a single repo can produce multiple distro-level packages.

- Florian

On 2/10/19 1:02 AM, Amirouche Boubekki wrote:
> I think splitting the codebase will be a pain for gnunet.
> 
> The only *good* reasons for manyrepos are social or ego politics "this
> is my lawn" or legal. The only one that applies to gnunet is legal
> because one needs to fill a gnu form to be able to contribute.
> 
> I am biased toward monorepo by experience dealing with big project
> (100k+ SLOC) and the only time it made sens to split the project into
> many repositories because it was different teams / workflow (social) and
> different legal terms for the various services/daemons, at previous
> $WORK, they had to fork gentoo to make it work.
> 
> Otherwise, each time I saw another repository it was a source of pain:
> 
> - Need to manage several versions
> - git submodule workflow is not good enough, it doesn't track branch, I
> personally I never remember how to know the branch of a commit, plus it
> requires some more git-fu to bump the submodule.
> - refactoring anyone?
> - generally speaking manyrepos at small scale is more work
> 
> And again, it requires somehow to track down every versions (what works
> with what) and you end up with another repository (or distribution) with
> another build system that puts everything together. Continuous
> Integration can do that? Where is the code of the CI? Another repo? More
> versions, more git clone more grep across repositories / directories not
> even in sync.
> 
> Popularity arguments:
> 
> a) Ok, everybody know GAFAM love monorepos and that is a also a source
> of pain (dedicated team and software). That said, gnunet is not the size
> of any GAFAM, hence it will not suffer from monorepo pain points. 
> 
> b) Github and Javascript made the manyrepos popular for various ego
> reasons and because JavaScript is not good. I won't take inspiration
> from that part of the JavaScript noosphere. gnunet-leftpad anyone?
> 
> c) Now, there is GNOME. GNOME is famous for its bazaar model of
> development and also famous for the adoption of meson (maybe even its
> inception) or its previous incarnation jhbuild. Anyway, even if GNOME
> and GNU (which is also a bazaar) success is appealing, gnunet is not GNU
> or GNOME. From my point of view the bazaar development model scales
> better / more easily in a socially distributed setting. Also why Linux
> is still a single repository?
> 
> Le sam. 9 févr. 2019 à 18:16, Schanzenbach, Martin
> <address@hidden <mailto:address@hidden>> a écrit :
> 
> 
> 
>     > On 9. Feb 2019, at 17:13, Christian Grothoff <address@hidden
>     <mailto:address@hidden>> wrote:
>     >
>     > On 2/9/19 5:04 PM, Schanzenbach, Martin wrote:
>     >> I have some inline comments as well below, but let us bring this
>     discussion down to a more practical consensus maybe.
>     >> I think we are arguing too much in the extremes and that is not
>     helpful. I am not saying we should compartmentalise
>     >> GNUnet into the tiniest possible components.
>     >> It's just that I think it is becoming a bit bloated.
>     >>
>     >> That being said, _most_ of what is in GNUnet today is perfectly
>     fine in a single repo and package.
>     >> For now, at least let us not add another one (gtk) as well?
>     >>
>     >> Then, we remain with
>     >>
>     >> - reclaim (+the things reclaim needs wrt libraries)
>     >> - conversation (+X)
>     >> - secureshare (+X)
>     >> - fs (+X)
>     >>
>     >> as components/services on my personal "list".
>     >> I suggest that _if_ I find the time, I could extract reclaim into
>     a separate repo as soon as we have a CI and I can
>     >> test how it works and we can learn from the experience.
>     >> Then, we can discuss if we want to do the same with other
>     components, one at a time, if there is consensus and a person that
>     >> would be willing to take ownership (I am pretty sure we talked
>     about this concept last summer as well).
>     >
>     > Maybe you could start with extracting the SecuShare components? That
>     > should do for a first "experience", and be a bit more effective at
>     > reducing bloat as well ;-).
> 
>     Well, I could, but our secushare people are quite active so maybe
>     there are volunteers (if they agree with the proposal at all).
>     Regarding "bloat". If we want to effectively eliminate bloat than
>     let's look at numbers:
> 
>     File Sharing:
>     src/fs: 36918 (!) LOC in .c files
>     src/datastore/cache: ~15k LOC in .c files
> 
>     Conversation:
>     src/conversation: 10538 LOC in .c files
> 
>     SecuShare:
>     src/psyc* : ~17000 LOC in .c files (altough I am not sure about this
>     because theoretically psyc is a general use protocol, no?)
>     src/social: 9447 LOC in .c files
>     src/multicast: 5633 LOC in .c files
> 
>     Reclaim:
>     src/reclaim* : ~6500 LOC in .c files
> 
>     Now, considering that fs is practically always built for everybody
>     and SecuShare and reclaim are experimental, it hurts the most for
>     devs that actually compile from source.
>     Everything combined are 110000+ LOC which is 22% of the codebase
>     (~500k, oO). Considering that there is a significant redundancy in
>     transport/ (75k) at the moment, this number is probably closer to 25%.
>     Granted, this is a lot less than I expected ;), but maybe
>     illustrates the dimensions.
> 
> 
>     >
>     > That said, splitting of reclaim seems also much less problematic than
>     > fs/conversation, and if you then integrate reclaim with the libgabe
>     > tree, the overall number of downloads/installation for reclaim
>     wouldn't
>     > go up, so that would certainly kill my argument of making the
>     > installation more complex (might indeed simplify it, as one
>     doesn't have
>     > to remember to install libgabe before GNUnet to get reclaim).
> 
>     Could do, but libgabe has some nasty additional deps (libpbc and
>     gmp) which we _might_ eventually get rid of completely by
>     implementing GNS-based encryption.
> 
> 
>     _______________________________________________
>     GNUnet-developers mailing list
>     address@hidden <mailto:address@hidden>
>     https://lists.gnu.org/mailman/listinfo/gnunet-developers
> 
> 
> _______________________________________________
> GNUnet-developers mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>