Re: [GNUnet-developers] Discussion, and Help Wanted: Moving to Gitlab for Git, CI, and Issues

[Schanzenbach, Martin]

Thanks for the writeup.
My comments below.

> On 5. Apr 2019, at 21:20, Devan C. dvn <address@hidden> wrote:
> 
> Signed PGP part
> Hello my fellow GNUnetians,
> 
> As some of you know, I have been pushing for and working on getting us
> to CI/CD system based on Gitlab CI. This is pretty much ready to start
> using and building out the pipelines for now.
> 
> In a previous thread[0] there was a decision to give Gitlab a try, as
> more than just our CI system, and to migrate our repos from Gitolite to
> our own Gitlab instance.
> There was less agreement regarding the idea to move from MantisBT to
> Gitlab's issue tracker, but in the end we decided to try importing all
> of the >2k Mantis tickets into Gitlab.[1]
> 
> This email is a status update on the overall progress of moving to
> Gitlab.
> 
> Current Gitlab status:
> 
> - Gitlab is running and accessible at `https://gitlab.gnunet.org`
> 
> - Registration is open. There are no guarantees on uptime, or even
>  data retention (though I don't expect data to disappear).
> 
> - wldhx has kindly offered two "Gitlab Runners" for running CI jobs.
>  These will be added as shared runners, to be used by any projects on
>  the instance. This may be changed later to only be shared by projects
>  under the GNUnet namespace.
> 
> **TODO**
> 
> - Setup email. Used for registration, password resets,
>  notifcations, and interaction (eg. issue threads).
> 
> - Currently run using containers with docker-compose. Will switch to
> using systemd services to with the containers, removing docker-compose.
> 
> - Daily remote backups. Perhaps 'firefly.gnunet.org' could be the backup
>  site, hmm?
> 
> - Change configured hostname (in Gitlab) to 'gitlab.gnunet.org'.
> 
> - Setup redirect from 'git.gnunet.org' -> 'gitlab.gnunet.org'
> ----
> 
> Current [MantisBT -> Gitlab Issues] status:
> 
> Exporting:
> - Mantis can export to CSV and "Excell" XML
>  - These do not contain comments (bugnotes). It looks like there might
>    be a possibility to enable them via a configuration option[2]. Not
>    sure who all has admin access, whom I could coordinate with. Maybe
>    easier if I can get admin rights? Grothoff, what do you think?
> 
> Importing:
> - I have found only a couple of scripts[3,4] for this. They are both out
>  of date, for old versions of both softwares. I have tried both to no
>  avail. [4] is the most promising; It's not so old.
>  I would really appreciate any help working on this.
> 
> I suppose this means that we will continue to use Mantis, and disable
> issues in Gitlab for now. Any protests or ideas?

That would be very sad. And fwiw, I think this is because Mantis does not offer 
proper export/APIs...
We should migrate to gitlab issues if we decide for it. Even if that means a 
hard break with legacy Mantis issues.

> ----
> 
> Migrating from Gitolite:
> 
> For those whom are not aware, we currently use gitolite for all of the
> lovely repos in our collection. We will need to copy all of the repos to
> Gitlab, as well as duplicate permissions, and setup githooks.
> 
> 1. Create namespaces/groups on our Gitlab
> 
> 2. Clone repos. This can be done via the web interface "Import" option
> when creating a new repo, or the new remote can be added, and then
> pushed. The little script found here can help with getting all the
> repos from Gitolite[5]
> 
> 3. Setup redirects. eg.  https://gnunet.org/git ->
> https://gitlab.gnunet.org
> 
> 4. Manually replicate permissions. Will need a Gitolite admin's help
> on this.

Can't we just have uses use their own namespaces for development?
I mean, the power of gitlab is that users work in their own forks and create 
pull requests.
Then, the "main" project repo does some testing/ci magic and if everything 
passes, it is (automatically) merged.
No need to give everybody access manually!

> 
> 5. Setup githooks. We have quite a few githooks setup, so we will
> need to recreate those.
> 
> After all of that is done, I think we should be ready to switch over
> to Gitlab for at least the git management and CI/CD.
> ----
> 
> That brings us to the final update: The CI System...
> 
> - We have a couple of small runners (thanks wldhx).
> 
> - We have some very basic '.gitlab-ci.yml'[6] files, defining jobs.
>  - I will begin expanding these in the coming weeks.
> 
> **TODO**
> 
> - As we build out a matrix of pipelines, we will need more resources.
> 'firefly.gnunet.org' is a logical option. In the past I've seen it
> utilized heavily by experiments. As long as we are okay with dedicating
> some CPU and RAM to runners, then I will begin setting them up.
> 
> - Setup Gitlab Container Registry [7] for storing our CI artifacts.
> 
> - Expand our '.gitlab-ci.yml' files to include e2e tests, builds for
>  multiple architecures, and continuous delivery of packages for various
>  package managers.
> ----
> 
> Wow, so that's a lot of text. A lot of people have been asking me about
> the status of Gitlab, and if and how they can help with CI. I hope this
> gives people a thorough update, and answers. I really believe Gitlab can
> be a useful software suite, despite its shortcomings. My hopes are that
> it will help increase the feedback loop between development and testing,
> as well as make it easier and more welcoming for new contributors.
> 
> 
> Be well, and Happy Hacking!
> - Devan
> 
> 
> [0] https://lists.gnu.org/archive/html/gnunet-developers/2019-01/msg00071.html
> [1] https://lists.gnu.org/archive/html/gnunet-developers/2019-01/msg00082.html
> [2] 
> http://www.mantisbt.org/docs/master/en-US/Admin_Guide/html-desktop/#admin.config.display
> [3] https://gitlab.kitware.com/utils/mantis-to-gitlab
> [4] https://github.com/timwiel/mantis2gitlab
> [5] 
> https://tutorials.technology/tutorials/89-show-all-repositories-of-a-git-server.html
> [6] https://docs.gitlab.com/ce/ci/quick_start/
> [7] https://docs.gitlab.com/ce/user/project/container_registry.html
> 
>

signature.asc
Description: Message signed with OpenPGP