Re: [GNUnet-developers] Towards a new formalized release policy

[Nils Gillmann]

Nils Gillmann transcribed 15K bytes:
> Christian Grothoff transcribed 9.8K bytes:
> > On 03/30/2018 07:45 PM, Catonano wrote:
> > > Hello
> > > 
> > > this is the first time I write on this mailing list
> > 
> > Welcome ;-).
> > 
> > > I should probably introduce myself but Iì m not exact6ly comfortable
> > > with introductions
> > 
> > Not really necessary, we are very open and not data hungry ;-).
> > 
> > > I sent some patches to Ng0 when they were migrating the documentation
> > > from the old format to TexInfo>> My patches eliminated a ton of errors 
> > > and warnings that the texinfo
> > > tools were erupting at the time
> > 
> > Great, thanks. You are right that the texinfo now _builds_. But that
> > does not mean that the structure is great (easy to follow, logical) or
> > that all of the text is current.  That said, this is not something any
> > single person can fix anymore: I have not used half the platforms listed
> > for build instructions in a long time and some (completely undocumented)
> > subsystems (looking in the direction of the SecuShare and Fraunhofer
> > teams here) I am not sufficiently familiar with.
> > 
> > > I thought the work on the documentation was finished
> > 
> > Well, technically it never will be ;-).
> 
> The conversion is finished, the real hard work began when it was
> finished. Last winter I considered the initial export and fixes phase
> done and started more fixing grammar and general additions I wanted to
> make ever since I started with this.
> 
> > > If it' s not, I could try to contribute some more
> > > 
> > > Is the documentation process stuck ? Where, exactly ?
> > 
> > (1) devs taking the time to document their contributions
> > (2) someone taking the lead at going over the text, removing
> >     clearly outdated stuff (3.7.7 "the shorten zone" no longer exists!,
> 
> Well the codebase is really big. I'd love to get feedback on what can
> be thrown out, really.
> 
> >     do we need build instructions for Debian 7.5 if we have some
> >     for Debian 8? What about Ubuntu 12.04 vs. 14.04?),
> 
> No. I've been meaning to generalize this. So you get a general instruction
> list (self contained) to build our dependencies and also point out issues
> etc in that list for each dependency.
> After that I want Debian, Ubuntu, gentoo, openbsd, windows, etc in a list
> as small as possible and with a language as easy to follow as possible.
> 
> That's another thing I want to fix in the long run, language requirements
> for reading the Manual. I'm still in the introduction chapter with this,
> as I'm occasionally reading the text to people and correct it to what they
> will understand better.
> 
> >     updating/clarifying things they can update, and *bothering*
> >     people (me, this list) to help them where they cannot!
> 
> Yeah, the thing with the list is, and I've pointed this out before:
> Start using it. I have little to no motivation to write to the list
> because like 95% of the messages happen offlist or continue offlist.
> I could dump my masterplan for the documentation on the list, but I'm
> pessimistic if it would be useful at all.
> 
> > (3) Fixing some ugly layout issues (like the page *numbered* 1 in the
> >     PDF "Introduction" with 1 sentence)
> 
> Yep.
> 
> > As you write below: you don't know how to build the project right now,
> > or how to get an indication of where the coverage is lacking. Well, both
> > would be good things to document. Hartmut worked on getting GNUnet to
> > build on Guix.  We should document how in that very manual.
> 
> I maintain a collection of packages for Guix for building GNUnet that
> are not yet announced. Maybe I can hardlink them into a more public
> repository and we could use that aswell? Wldhxx suggested to document
> both instlaling and developing with Nix (and then Guix?).
> 
> > We have
> > configure options to do code coverage analysis to see where tests are
> > missing.  
> 
> Speaking of missing configure options: being able to build the documentation
> without too much dependencies. For my experimental in progress system *and*
> for the new gnunet.org server we need to build the documentation. At the
> moment my gnunet-doc package, which is only a bit different from the
> guix-env.scm, takes too much time to build because you have no convenient
> way to just build the docs (texi outputs, man, etc).
> Someone should finish --enable-documentation and make it ignore every
> folder except for 'doc' or 'doc/documentation'.
> 
> > That's documented in section 5.6 "Build-system". Is that a
> > good place? Is there a better way to structure the documentation to make
> > this information easier to find? An index? Other chapter/section
> > structure?  That's what needs help ;-).
> 
> Just my view on this, as since the convertion to Texinfo I've been the
> primary committer to the doc/documentation tree.
> 
> Writing good Documentation takes time. To rewrite an entire Documentation
> takes even more time. If I were paid for this, I'd focus on it and would

What I meant by this was more like: If I had a reason to make documentation
a top priority you'd see more commits on the documentation. I can't contribute
to the C parts of the code at the moment, but I like working on the tex based
ones. In general I think we can agree that time being short is a good reason
why this takes long ;)

> probably be done in a reasonable long time. I like working on this, really.
> My main issue is shared time, and therefore too little time.
> That said, I've started a more practical approach: I'm setting up systems
> and getting in touch with Operating Systems (the same way I did with Guix
> back then) to document the process and minimize the parts in the Install
> Handbook.
> In the ideal case you could simply add GNUnet from package repositories
> and be done. Some systems need finetuning, like lynX told me a while back
> that on BSD the VPN function doesn't work. I want to debug this with both
> FreeBSD and OpenBSD (one of them first) to extend the list of systems.
> I have lots of feedback. Like we should really, really, really drop the
> mingw approach. As far as I know Devan wanted to work on this in the future.
> Dropping mingw from the documentation would decrease its size.
> 
> I can take my time to send more feedback and what needs to be done to the
> list if you want?
> 
> > >  
> > > 
> > > 
> > >     2) As for the release _criteria_, I had proposed a few simple minimal
> > >     requirements everybody seemed to agree upon at the time: (1) passing
> > >     testcases, (2) no compiler warnings / serious issues found in static
> > >     analysis, (3) passes 'acceptance' test where we manually try key
> > >     features in the GUI(s).  I think I also had as highly desirable (4)
> > >     working/passing CI/BB and (5) new Web site with current documentation,
> > >     but I'm (in principle) willing to forgo those. Also, we can decide to
> > >     cut out subsystems (psyc, multicast, psycstore, etc.) if those do not
> > >     pass tests and nothing else depends upon them.  So if we get this, I'm
> > >     generally happy to 'make dist' a new TGZ, which is 'making a release'.
> > > 
> > >     3) What you are discussing is more the *development* process.  We
> > >     already have branches.  We have seen how merging branches for a 
> > > release
> > >     can create wonderful additional chaos and delays because the branch
> > >     worked for the dev, but not on other systems --- and merging 100 
> > > patches
> > >     from a branch (as usual with insufficient unit tests) then makes for 
> > > fun
> > >     debugging when you actually want to get a release done.  So without
> > >     better CI and better tests, they can do about as much harm as good. I 
> > > am
> > >     all for "do not break master" and "only commit new code that builds 
> > > and
> > >     passes tests to master". That won't fix the strange 
> > > existing/random-ish
> > >     test failures we do have for a while now. For that, it would take 
> > > better
> > >     tests, and people with the time to write them, and write them well.
> > > 
> > > 
> > > Again, I could try to contribute some tests
> 
> If you want to dive deep into GNUnet, there's also the thing that
> python related tests in our codebase need to be rewritten to python3.
> It will be less work to port them in the process to a more generalized
> common test framework.
> Many python functions we have can also be generalized into something
> like "gnunet-python-common", which our python bindings (gnunet-python)
> could use to be reduced in size.
> I could explain more to you or anyone with python knowledge.
> 
> > Given what is hitting me right this second, there is one other
> > test-related issue: some tests are failing non-deterministically. Those
> > are particularly awful, and so fixing this by either having the
> > underlying bugs be caught deterministically by other tests, or fixing
> > the tests to be more well-behaved is just as important as improving
> > coverage.  Oh, and _often_ the bug is in the test case (i.e. timeout too
> > small, works 95/100 times). Which is a really ugly situation as it makes
> > it hard to know if a change actually fixed (or introduced) an issue. So
> > just running the tests locally a few times to see if some behave oddly
> > on _your_ system can be a good indicator, as this non-determinism can
> > really depend on the specific system. GNUnet doesn't have threads, but
> > we still have concurrency (multi-process), so we cannot escape
> > non-determinism...
> > 
> > > I could use an introduction about how to build and run the project right
> > > now, preferably using Guix based tools/environments
> 
> I have my own way to build GNUnet on guix and done some more recent work with 
> it,
> maybe I could provide some insight if you tell me where you are stuck.
> 
> > Hartmut Goebel <address@hidden> and
> > Andreas Enge <address@hidden> have both worked on this in the
> > past, they know more than I do, but I don't know if they read this.
> > Maybe squeeze some documentation out of them? Might help if you promise
> > to add it to the texinfo ;-).
> > 
> > > And then if there's any indication of where is the coverage lacking,
> > > that coudl be useful
> > 
> > See section 5.6.  Let me know if it is unclear and you are unable to
> > improve the text yourself ;-)
> > 
> > >  
> > > 
> > >     Today, we sometimes have bugfixes in a branch not backported to 
> > > master,
> > >     or branches that have not been rebased to master lacking bugfixes from
> > >     master. Wonderful. As maintainer, it's hard enough for me to keep 
> > > track
> > >     of mainline/master and my own branches/developments. I cannot also
> > >     manually cherry-pick bugfixes from branches, and so far *many*
> > >     developers have been really shitty at merging their branches in a 
> > > timely
> > >     fashion (and by "timely", I can point to examples in the time range of
> > >     within a few years).
> > > 
> > > 
> > > I' m sorry to learn that
> > 
> > Well, I was naturally ranting a bit to hope the situation won't repeat
> > (as often) ;-).
> > 
> > > 
> > >     So please, do feel free to use branches, but more importantly, write
> > >     good tests, make sure they pass, and merge if they do. Also, do setup 
> > > a
> > >     CI, make sure the CI runs the tests on a wide array of systems, make
> > >     sure master *passes* the tests, and _then_ we can impose a 'do not 
> > > break
> > >     master' regime for commits. 
> > > 
> > > 
> > > What does it take to setup a CI ?
> > > 
> > > I hear that Gitlab has some powerful CI tools, could they help ?
> > 
> > That is dvn's plan. Even though my recent experience with extracting
> > information from core dumps from a Gitlab CI that Tim setup for MHD was,
> > eh, painful, compared to what we have today with the BuildBot.
> > 
> > > Again, can I help with this ?
> > 
> > From my perspective dvn & company are "in charge" of this. I heard dvn
> > is a bit busy this month, and I'm sure they'd appreciate qualified help.
> >  In case they don't read this - and if this is what you want to help
> > with, check with <address@hidden>. My understanding of the vision for the
> > NG-CI is that the result will mainly be a Git repo with the GitLab CI
> > configuration. So this should make collaboration easy, but I don't know
> > how far things are here.
> > 
> 
> 
> 
> 
> > _______________________________________________
> > GNUnet-developers mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/gnunet-developers
> 
> 
> _______________________________________________
> GNUnet-developers mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnunet-developers