Re: GNUmakefile.team additions

[Yavor Doganov]

В 18:34 +0200 на 15.02.2010 (пн), George Zarkadas написа:
> I also have a question (in part because I intend to address in the
> future the issue with fuzzy, deleted, etc translations): is it
> considered good practise to have (in a standard place) separate
> scripts for each task that are called from within the makefile, or it
> is better to incorporate everything inside the makefile?

It depends.  If the recipe becomes complex and unwieldy to
read/grok/edit in the makefile, it's sometimes better to encapsulate
it in a separate program/script.  For this case, that's not warranted
IMO.

> As a background information, the workflow I use in general is:
>   1. make sync VCS=yes
>   2. make report
>   3. apply changes to already existent files, insert and translate new .po 
> files
>   4. make format
>   5. make writeback
>   6. cd to www, walk affected dirs and cvs commit (my) individual files

That looks fine to me.  FWIW, my workflow is essentially the same,
except that 1) I do it with a daily cronjob, so I never run `sync'
manually; 4) as a matter of habit I run M-x po-wrap for every file I
edit; 5) I `cp' files manually and commit them in a batch in `www', so
this new target is a convenience for me.

> On my system (ubuntu 9.10) the `$(MSGCAT) -o $$file $$file' line
> *always* updates the file's timestamp (as seen by ls -l).

Ah, yes, I wrote it backwards -- just like plain `cat', `msgcat'
always updates the file's timestamp.  But:

> This leads to unnecessary increase of CVS revision numbers on the
> local (team) repository and it is why the inner if exists; it is not
> a speed optimisation.

I can't believe this.  CVS (and any other VCS, for that matter, except
GNU Arch, TTBOMK) does not perform a commit if there are no actual
differences.  Observe:

$ cd $wwwdir/gnu/po
$ msgcat -o gnu.bg.po gnu.bg.po
$ cvs up
cvs update: Updating .
$ cvs diff
cvs diff: Diffing .
$ cvs ci -m "Bogus commit that should not happen."
cvs commit: Examining .

Perhaps you are confused that CVS lists the files as modified in the
commit message window as spawned by $EDITOR after doing `cvs ci'
without the `-m' argument?  If so, that's because of a dumb limitation
of CVS: it doesn't know which files it will actually commit before it
consults the server.  When you run `cvs commit', it crawls the tree
and determines the files it should *check* based on timestamp
differences, but when you actually perform the `commit' operation and
CVS consults the server, and finds out there are no differences, it
does nothing.  Kinda stupid and confusing, but things are what they
are.

Since there's zero chance `www' (or any other site, if one day we make
GNUN a general purpose tool) to switch to GNU Arch, this check is
useless.

P.S.  In fact GNU Arch is even dumber, it will commit and add a
      spurious rev regardless of the timestamp.

> * Assuming that all .po files are submitted formatted to www, as they
> ought to, if line width is > 80 then the file was touched with a
> po-editor (which they are known to not break msgids), so it needs to
> be formatted.

I performed some simple tests on all files at philosphy/po, and
surpisingly to me, the overall performance with the wc check is
faster.  So we'll keep it.

Some minor comments:

  - As per '(coreutils)wc Invocation' the -L option is
    locale-dependent, so we reset the locale.
  - Usually in makefiles, scripts, etc. the long options are used as
    they're far more descriptive; the short options are good for
    interactive use.
  - The check for the max line length is very often TRUE even if the
    file is properly wrapped.

> * Another implicit assumption of the code is that `make sync VCS=yes'
> has already been run; this is what I do always before starting an
> editing run of my team files.

That's what you do and what everyone should do, but it does not follow
that everyone does it in practice :-)  Anyway, adding a dependency on
`sync' would be annoying, me thinks.

As an exapmple of a bad `make writeback' use: consider a mass-change
in www, like substituting "gnu.org/foo" with "foo.fsf.org" in many
files.  Usually, some of us will do this job for all files in *www*,
be it .html, .po, or .LANG.html (that have no master .LANG.po).  There
is no way to _merge_ the change back to your www-LANG repo with the
existing targets, and no way to be aware of it (apart of being
subscribed to www-commits or announcement being made on
{www,trans-coord}-discuss).  But since `writeback' is nothing more
than an automation over manual `cp' operation, there's nothing holding
us to implement it, with the constraints described above in mind.

> == About writeback: ==
> 
> Yes, this is a dangerous target, but if implemented correctly it will
> be a great convenience especially for large teams with many edits: it
> will replace a lot of cp commands

I agree.

> and it will allow to implement *safely* an even more dangerous
> target (say, `export'?), ie the upload of all edits to www in one
> step.

I don't know what exactly you mean here; please elaborate.

> * push        * pushback      * rewrite      * store
> * resync      * backsync      * syncback   * source

Both push and publish (as suggested by Ali) sound good to me.  `push'
might be slightly confusing for teams that use a DVCS, so let's make
it `publish'.

Ali: Interactivity won't gain you much (if anything), because to make
a sound decision you'd want to inspect the differences between the two
files.  So I think that asking for confirmation before the copy
operation would be basically nothing but a nuisance.


In conclusion, I intend to commit the changes below (some extra
testing is appreciated).  If somehing is buggy, we can always correct
it at once, of course.

2010-02-18  George Zarkadas  <address@hidden>  (tiny change)
            Yavor Doganov  <address@hidden>

        Implement format/publish/clean targets in GNUmakefile.team.
        * GNUmakefile.team (MSGCAT, pubwmsg): New variables.
        (format, publish, clean): New phony targets.
        * doc/gnun.texi (New Translation): Recommend rewrapping of
        comments.
        (GNUmakefile.team Variables) <Targets in GNUmakefile.team>:
        Document the new rules.
        * NEWS:
        * THANKS: Update.

--- GNUmakefile.team    7 Jan 2010 14:43:04 -0000       1.9
+++ GNUmakefile.team    18 Feb 2010 14:46:26 -0000
@@ -43,6 +43,7 @@
 # Adjust these variables if you don't have the programs in your PATH.
 MSGMERGE := msgmerge
 MSGFMT := msgfmt
+MSGCAT := msgcat
 CVS := cvs
 SVN := svn
 BZR := bzr
@@ -52,6 +53,9 @@
 
 translations := $(shell find -name '*.$(TEAM).po' | sort)
 log := "Automatic merge from the master repository."
+# Warning message for the `publish' rule.
+pubwmsg := "Warning (%s): %s\n  does not exist; (either obsolete or \`cvs\
+update\' in $(wwwdir) needed).\n"
 
 # Determine the VCS.
 REPO := $(shell (test -d CVS && echo CVS) || (test -d .svn && echo SVN) \
@@ -155,3 +159,50 @@
            | egrep '(fuzzy|untranslated)' \
              && echo "$${file#./} needs updating." || true ; \
        done
+
+# Helper target to rewrap all PO files; avoids spurious diffs when
+# they get remerged by the official build.
+.PHONY: format
+format:
+       @echo Formatting .po files with msgcat:
+       @for file in $(translations); do \
+         if [ `LC_ALL=C <$$file wc --max-line-length` -gt 80 ]; then \
+           $(MSGCAT) -o $$file $$file && echo "  $${file#./}"; \
+         fi; \
+       done
+
+# Helper target to copy all (supposedly) modified files to the `www'
+# master repository.  A warning is printed if the corresponding
+# directory in `www' cannot be found, or if the template is missing
+# (which in almost all cases means that the original article has been
+# renamed or deleted).
+.PHONY: publish
+publish: format
+       @echo Copying edited .po files back to $(wwwdir):
+       @for file in $(translations); do \
+         wwwfdir=$(wwwdir)`dirname $$file`/po ; \
+         wwwfpot=$${wwwfdir}/`basename $${file/.$(TEAM).po/.pot}`; \
+         wwwfile=$${wwwfdir}/`basename $$file`; \
+         if [ ! -d $$wwwfdir ]; then \
+           printf $(pubwmsg) "$${file#./}" "directory $$wwwfdir"; \
+           continue; \
+         fi; \
+         if [ ! -f $$wwwfpot ]; then \
+           printf $(pubwmsg) "$${file#./}" "template $$wwwfpot"; \
+           continue; \
+         fi; \
+         if [ $$file -nt $$wwwfile ]; then \
+           cp $$file $$wwwfile && echo "  $${file#./}"; \
+         fi; \
+       done
+
+# Helper target to delete common auto-generated files.
+.PHONY: clean
+clean:
+       @echo -n Deleting unnecessary auto-generated files...
+       @for file in $(translations); do \
+         $(RM) $$file~; \
+         $(RM) $${file/.po/.mo}; \
+         $(RM) $$file.bak; \
+       done
+       @echo " done."
Index: NEWS
===================================================================
RCS file: /sources/trans-coord/trans-coord/gnun/server/gnun/NEWS,v
retrieving revision 1.24
diff -u -r1.24 NEWS
--- NEWS        7 Jan 2010 14:43:04 -0000       1.24
+++ NEWS        18 Feb 2010 14:46:26 -0000
@@ -1,5 +1,10 @@
 GNUnited Nations NEWS - User visible changes.
 
+* Changes in GNUnited Nations 0.5 (2010-??-??)
+
+** New GNUmakefile.team targets: format, publish, clean.
+   Please refer to the manual for details.
+
 * Changes in GNUnited Nations 0.4 (2009-11-11)
 
 ** New script `gnun-clear-previous'.
Index: THANKS
===================================================================
RCS file: /sources/trans-coord/trans-coord/gnun/server/gnun/THANKS,v
retrieving revision 1.3
diff -u -r1.3 THANKS
--- THANKS      17 Dec 2009 12:26:44 -0000      1.3
+++ THANKS      18 Feb 2010 14:46:26 -0000
@@ -1,15 +1,17 @@
 Several people have reported bugs and/or provided useful suggestions
 and ideas.  Many thanks for their contributions:
 
-+------------------+-----------------+
-|Ali Servet Dönmez |   address@hidden|
-+------------------+-----------------+
-|Andrea Pescetti   | address@hidden|
-+------------------+-----------------+
-|Dominik Smatana   |   address@hidden|
-+------------------+-----------------+
-|Hossam Hossny     |   address@hidden|
-+------------------+-----------------+
++------------------+---------------------------+
+|Ali Servet Dönmez |             address@hidden|
++------------------+---------------------------+
+|Andrea Pescetti   |           address@hidden|
++------------------+---------------------------+
+|Dominik Smatana   |             address@hidden|
++------------------+---------------------------+
+|George Zarkadas   |address@hidden|
++------------------+---------------------------+
+|Hossam Hossny     |             address@hidden|
++------------------+---------------------------+
 
 Local Variables:
 mode: text
Index: doc/gnun.texi
===================================================================
RCS file: /sources/trans-coord/trans-coord/gnun/server/gnun/doc/gnun.texi,v
retrieving revision 1.17
diff -u -r1.17 gnun.texi
--- doc/gnun.texi       15 Feb 2010 10:12:10 -0000      1.17
+++ doc/gnun.texi       18 Feb 2010 14:46:27 -0000
@@ -973,6 +973,11 @@
 header after entering the necessary information (this is done by
 simply pressing @key{TAB} in PO mode).
 
+It is recommended that you wrap all lines in the comments to be less
+than 80 lines; that looks better from a purely aesthetic point of view
+and improves the performance of @file{GNUmakefile.team}'s @code{publish}
+rule (@pxref{GNUmakefile.team Variables}).
+
 There are some special messages that appear in the POT and PO:
 
 @table @samp
@@ -1488,6 +1493,65 @@
 @item report
 Verifies which translations are complete, and prints a list (with
 statistics) of those that need to be updated.
+
address@hidden format
+A convenience rule to rewrap all files upto the standard length.  Most
+PO editors leave the @code{msgstr} as a single long line after it has
+been edited, but @acronym{GNUN} will automatically rewrap the file to
+the standard line length when it is processed.  Wrapping long lines in
+PO files is a good pracice as it avoids unnecessary revisions.
+
+This rule checks for all translations that have lines longer than 80,
+and runs @command{msgcat} accordingly to reformat them.  For that
+reason, it is recommended that you wrap all long lines in the comment
+fields (e.g. the file header, including the copyright statement, and any
+other comments for specific messages), because @code{make format} will
+unnecessarily invoke @command{msgcat} for any file that has a longer
+line, wherever it may occur.
+
address@hidden publish
+The @code{publish} rule's task is to copy all modified files to the
+official @dfn{www} repository.  It depends on the @code{format} target
+to ensure that all files are rewrapped to the standard line length
+limit, but deliberately does not depend on @code{sync VCS=yes}.
+Usually, one would run @code{make publish} when one or a bunch of PO
+files are in a satisfactory condition to be published, and this rule is
+just a convenience to avoid multiple manual @command{cp} invocations.
+As a rule of thumb, before running this rule it is sane to run
address@hidden and correct any @dfn{fuzzy} messages and other problems, if
+necessary.
+
+The @code{publish} rule does not depend on @code{sync} explicitly,
+because that would be a nuisance for offline operations and basically
+unnecessary when the committer is fairly confident that there are no
+changes to (re-)merge.  A hard dependency on @code{sync} would slow down
+the operation considerably.
+
+As usual, when committing to the official repository, it is always a
+good practice to examine the output of @command{cvs diff}.
+
+Invoking @code{make publish} prints warnings and does not actually copy
+the affected file if the sub-directory in ``www'' is non-existent or the
+corresponding @file{.pot} is missing.
+
+Typically, after an editing session (whether it involves actual editing
+or just merging contributions from team members), one would do:
+
address@hidden
+$ make sync VCS=yes
+$ make publish
+$ cd @var{wwwdir}
+$ cvs up
+(Add all new translations, if any.)
+  $ cvs add @var{file} @dots{}
+$ cvs diff
+$ cvs commit -m "Some descriptive message."
address@hidden example
+
address@hidden clean
+Deletes all backup and auto-generated files that some PO editors leave
+behind, address@hidden@var{file}.po~}, @address@hidden and
address@hidden@var{file}.mo}.
 @end table
 
 @code{make VCS=yes} is the recommended command to be run periodically.