[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Savannah-cvs] [143] SavannahInternals: more information
From: |
Assaf Gordon |
Subject: |
[Savannah-cvs] [143] SavannahInternals: more information |
Date: |
Wed, 18 Feb 2015 00:41:57 +0000 |
Revision: 143
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=143
Author: agn
Date: 2015-02-18 00:41:49 +0000 (Wed, 18 Feb 2015)
Log Message:
-----------
SavannahInternals: more information
Modified Paths:
--------------
trunk/sviki/SavannahInternals.mdwn
Modified: trunk/sviki/SavannahInternals.mdwn
===================================================================
--- trunk/sviki/SavannahInternals.mdwn 2015-02-16 01:45:36 UTC (rev 142)
+++ trunk/sviki/SavannahInternals.mdwn 2015-02-18 00:41:49 UTC (rev 143)
@@ -83,9 +83,8 @@
calls a cgi script on `www.gnu.org` to request a website update.
6. A cron job on `www.gnu.org` regularly checks for update requests,
and `cvs updates` the requested repositories.
-7. Unlike code repositories (which use cron+database access), mailing list
- creation is performed by `frontend` running a remote command on
- `lists.gnu.org` using ssh.
+7. A cron job on `internal` regularly query the database for mailing list
+ updates. It then creates the mailing lists on `lists.gnu.org`.
This is done because `lists.gnu.org` is managed by the FSF, not Savannah.
8. Publishing a new tarball release for GNU packages is performed by an
automated process, explained in the
@@ -295,10 +294,9 @@
are done by the script). New mailing lists are added as new records in the
tables. Change requests to mailing lists are done by changing the
`status` field (see source code for possible values).
-* A cron job on `frontend` reads the changes in the `mail_group_list` and
- executes the necessary changes on `lists.gnu.org` using ssh, then updates
- the database with the result of the change.
- See
[/backend/mail/sv_mailman.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in).
+* A cron job on `internal` reads the changes in the `mail_group_list` and
+ executes the necessary changes on `lists.gnu.org`. See "cron jobs on
+ internal" section below.
### MySQL Database
@@ -357,15 +355,15 @@
file, ensuring it will not be overwritten. The manually added
information is the owner name/email for some repositories. Example:
- $ cat /srv/git/project-list
- ...
- administration.git Savannah+Hackers
- anubis.git address@hidden
- autoconf.git address@hidden
- automake.git address@hidden
- autostrap.git Sylvain+Beucler
- coreutils.git Jim+Meyering
- ...
+ $ cat /srv/git/project-list
+ ...
+ administration.git Savannah+Hackers
+ anubis.git address@hidden
+ autoconf.git address@hidden
+ automake.git address@hidden
+ autostrap.git Sylvain+Beucler
+ coreutils.git Jim+Meyering
+ ...
These names (e.g. `Jim Meyering`) will appear
in the *owner* field when viewing the gitweb page of the project
@@ -381,15 +379,15 @@
mentioned project, writes an entry in `cgitrepos`. Example:
- $ cat /srv/git/cgitrepos
- ...
- repo.group=GNU Core Utilities
- repo.url=coreutils.git
- repo.path=/srv/git/coreutils.git
- repo.desc=GNU coreutils
- repo.readme=README.html
- repo.owner=Jim Meyering
- ...
+ $ cat /srv/git/cgitrepos
+ ...
+ repo.group=GNU Core Utilities
+ repo.url=coreutils.git
+ repo.path=/srv/git/coreutils.git
+ repo.desc=GNU coreutils
+ repo.readme=README.html
+ repo.owner=Jim Meyering
+ ...
* `generate_log_accum.pl` - This script updates the CVS hooks in savannah
repositories.
@@ -402,54 +400,186 @@
`<project>/CVSROOT/{loginfo,commitinfo}` files.
* Example of `commitinfo` for source-code (not web) CVS repository:
- $ cat /sources/pspp/CVSROOT/commitinfo
- #<savane>
- DEFAULT /usr/local/bin/commit_prep.pl -T pspp_70 %p
- #</savane>
- ALL $CVSROOT/CVSROOT/no-commits %{s}
+ $ cat /sources/pspp/CVSROOT/commitinfo
+ #<savane>
+ DEFAULT /usr/local/bin/commit_prep.pl -T pspp_70 %p
+ #</savane>
+ ALL $CVSROOT/CVSROOT/no-commits %{s}
* Example of `loginfo` for source-code (not web) CVS repository:
- $ cat /sources/pspp/CVSROOT/loginfo
- #<savane>
- DEFAULT /usr/local/bin/log_accum.pl --mail-to address@hidden
--send-diff --file-prefix pspp_70 --config /etc/log_accum-sources.config %p
%{sVv}
- #</savane>
+ $ cat /sources/pspp/CVSROOT/loginfo
+ #<savane>
+ DEFAULT /usr/local/bin/log_accum.pl --mail-to address@hidden
--send-diff --file-prefix pspp_70 --config /etc/log_accum-sources.config %p
%{sVv}
+ #</savane>
* Example of `loginfo` for web CVS repository:
- $ cat /srv/cvs/web/datamash/CVSROOT/loginfo
- #<savane>
- ALL echo 'Triggering webpages update...'; cat > /dev/null; curl
http://www.gnu.org/new-savannah-project/new.py -s -F type=gnu -F
project=`basename %r`
- #</savane>
+ $ cat /srv/cvs/web/datamash/CVSROOT/loginfo
+ #<savane>
+ ALL echo 'Triggering webpages update...'; cat > /dev/null; curl
http://www.gnu.org/new-savannah-project/new.py -s -F type=gnu -F
project=`basename %r`
+ #</savane>
* `sv_groups` - This script creates new repositories for projects in the
savannah database
* reminder: when a new project is approved/created on
savannah, or when a project administrator selects a new repository
feature the `frontend` php code only updates the database. this
- script picks up the changes and executes them).
+ script picks up the changes and executes them.
* file is
[/backend/accounts/sv_groups.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/accounts/sv_groups.in).
+ * The script queries the `groups` table, and creates the needed
+ repositories.
+ Example: For project `XXX`, if `groups.use_git=1`, it look for
+ `vcs:/srv/git/XXX.git`. If it doesn't exist, create it with
+ `git init` and additional savannah-specific configurations (e.g.
+ permissions, hooks, xattrs).
+ * Repository initialization and configuration is delegated to the
+ relevant perl module. See
+
[Git.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Git.pm),
+
[Cvs.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Cvs.pm),
+
[Bzr.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Bzr.pm),
+
[Svn.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Svn.pm),
+
[Hg.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Hg.pm),
+
[Download.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Download.pm).
+ * Two types of CVS repositories can be created:
+ `cvs` (Using `CvsMakeArea` in `CVS.pm`) - source code repository;
+ `webcvs` (Using `WebCvMakeAreaSavannahGNU` in `CVS.pm`) - webcvs
+ repository (content will be available on www.gnu.org/www.nongnu.org).
+ * The script also contains legacy code to create unix groups on the
+ local machine (using `/usr/sbin/groupadd`). This part is not used any
+ more, since groups and users are delegated to `nsswitch/libnss-mysql`
+ (see [[UserAuthentication]]).
+ * FIXME: What as "Area Attic" (mentioned in the code)?
+ * The script takes parameteres determining which items to create (e.g.
+ `--only-git`).
+ * On `vcs` host, the git/hg/bzr/cvs/cvsweb repositories are created.
+ * On `download` host, the download (and legacy 'arch') are created.
-
### Cron jobs on download
-FIXME: work-in-progress
-
stored in `download:/etc/cron.d/sv`:
*/30 * * * * root sv_groups --cron --only-download --only-arch
+* `sv_groups` - This script creates new repositories for projects in the
+ savannah database. See description for `sv_groups` in the above section.
+
### Cron jobs on frontend
-FIXME: work-in-progress
+* `frontend:/etc/cron.d/sv_export` [non-functional] - contains the
+ following commands:
-* `frontend:/etc/cron.d/sv_export` - unused?
-* `frontend:/etc/cron.daily/sv_list_groups` - unfinished. publish
- authoritative list of gnu/nongnu projects on savannah.
+ # XML data export
+ 02 * * * * root sv_export --xml-path /var/cache/savane
-FIXME: where's the script which creates lists on `lists.gnu.org` (
-calls `sv_mailman`)?
+ # Remove old data export
+ 30 1 * * * root sv_export_cleaner --xml-path /var/cache/savane
+ # Get list of Git repositories
+ 3,33 * * * * root scp vcs.sv.gnu.org:/srv/git/cgitrepos
/etc/savane
+
+
[sv_export](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/export/sv_export.in)
+ is an old feature which exports project data into XML.
+ FIXME: remove this?
+
+* `frontend:/etc/cron.daily/sv_list_groups` [unfinished] - an attempt to
+ publish an authoritative list of active projects hosted on GNU Savannah.
+ The script contains the following code:
+
+ mysql savane -B -e 'SELECT group_type.type_id, group_type.name,
+ unix_group_name, group_name
+ FROM groups, group_type
+ WHERE groups.type = group_type.type_id
+ AND groups.status="A"
+ ORDER BY type;' \
+ > /var/www/overlay/cooperation/groups.tsv
+
+ FIXME: complete this nice feature, and expose it with apache
+ configuration. Also possibly with JSON data, and more fields.
+
+### Cron jobs on internal
+
+stored in `internal:/etc/cron.d/sv`:
+
+ # Assign uidNumber and gidNumber inthe database (and rebuild cache for
+ # libnss-mysql-bg). 7211 was the latest group created by the old
+ # sv_groups cron - maybe one day we'll move everything to the 70000
+ # range. Then create homedirs and authorized_keys files.
+ */5 * * * * root sv_assign_uid_gid --min-gidNumber=77212
+
+
+ # New list creation
+ */5 * * * * root sv_mailman --cron
+
+ # Database cleaning
+ */25 * * * * root sv_cleaner --cron
+ 48 12 2 * * root sv_cleaner --cron --big-cleanup
+
+ # Reminder
+ 4 4 * * * root sv_reminder --cron
+ 25 18 * * * root sv_reminder --cron
+
+
+* `sv_assign_uid_gid` - creates unique user ids and group ids for
+ users/projects in the savannah system.
+ * reminder: across all savannah hosts, unix user/group ids are managed
+ with nsswitch/libnss-mysql (see [[UserAuthentication]] for details).
+ * Source code:
+
[/backend/accounts/sv_assign_uid_gid.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/accounts/sv_assign_uid_gid.in).
+ * When a new user (or project) is created on savannah, the php code
+ on `frontend` does not assign uid/gid.
+ * This script picks up new users (SQL: `user.uidNumber is NULL`) and
+ new projects (SQL: `groups.gidNumber is NULL`) and assigns them ids.
+
+* `sv_mailman` - queries the `mail_group_list` table, and creates/updates
+ the mailing lists on `lists.gnu.org`.
+ * Source code:
+
[/backend/mail/sv_mailman.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in).
+ * This script is tightly-coupled with the php code in
+
[/frontend/php/mail/admin/index.php](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/frontend/php/mail/admin/index.php).
+ * The `index.php` code sets the `mail_group_list.status` field to the
+ requested new status, and the `sv_mailman` script executes the
+ changes and updates the database.
+ * `sv_mailman` calls `internal:/usr/sbin/newlist`. The `newlist`
+ script is a stub which delegates the call to `lists.gnu.org` using
+ SSH. See [[MailSystem]] for more details.
+ The gist of `internal:/usr/sbin/newlist` is:
+
+ #!/bin/bash
+ list_full_name=$1
+ admin_mail=$2
+ password=$3
+ cat <<EOF | ssh -T -i ~/.ssh/id_dsa.lists.gnu.org address@hidden
+ COMMAND=newlist
+ LIST_FULL_NAME=$list_full_name
+ ADMIN_MAIL=$admin_mail
+ PASSWORD=$password
+ EOF
+
+
+* `sv_cleaner` - cleans up left-overs from interrupted php code?
+ * source code:
+
[/backend/misc/sv_cleaner.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/misc/sv_cleaner.in).
+ * FIXME: investigate this script.
+
+* `sv_reminder` - sends email reminders to users.
+ * source code:
+
[/backend/misc/sv_reminder.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/misc/sv_reminder.in).
+ * The help message says:
+
+ This script will send reminder to users in some specific cases.
+ Case 1: an item is about to start or finish, all people from the
item
+ notification list should be warned.
+ Case 2: an user decided to receive a batch of open item assigned to
+ him each x days, minimum priority being > 5
+ Case 3: a project admin decided that members assigned to some item
of
+ should receive a batch each x days, minimum priority > 5
+ (Case 4: a project admin decided that members assigned to some item
+ whose statut/resolution havent changed for some period should
receive
+ a batch each x days)
+
+ * FIXME: investigate this script.
+
### www.gnu.org update mechanism
### Mailing list configuration
@@ -467,5 +597,3 @@
### cvsweb hooks on vcs
-### other scripts on vcs
-
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Savannah-cvs] [143] SavannahInternals: more information,
Assaf Gordon <=