gnupload improvements

[Sergey Poznyakoff]

Hello,

I'd like to propose several improvements in gnupload. The patch
is attached.

This version of gnupload has been successfully used with alpha.gnu.org,
ftp.gnu.org and download.gnu.org.ua.

Additional documentation about its usage is available from:

  http://puszcza.gnu.org.ua/cookbook/?func=detailitem&item_id=136

The list of changes follows:

1. Ability to upload to download.gnu.org.ua (uploads via sftp).

2. Configuration file.

At startup, gnupload searches the current working directory for a file
named .gnupload. If found, this file is parsed according to the
following rules:

    * Empty lines are removed.
    * Lines beginning with a # (comments) are removed.
    * Newline characters are replaced by spaces.
    * The resulting string is prepended to the actual command line.

This allows to spare typing most commonly used options. E.g.:

# A sample .gnupload file
--to ftp://ftp.gnu.org.ua:tar
--user 55D0C732
# End of file

3. Support for removal and creating symlinks.

Three command line options are introduced for this purpose. The
--rmsymlink option causes removal of the symlinks listed in the command
line. The --symlink option creates symbolic links. It takes even number
of arguments, e.g.:

  --symlink a b c d

will create two symbolic links: a -> b and c -> d.

Finally, the --symlink-re option allows to create symbolic links along
with uploading new releases. For example, the following command:

  gnupload --to ... --symlink-re foo-1.2.tar.gz

uploads the file foo-1.2.tar.gz and creates a symbolic link:
foo-latest.tar.gz -> foo-1.2.tar.gz. As an optional argument, this
option allows to specify a sed expression to transform file names into
link names.

4. Any number of operations can be specified in a single invocation,
e.g.:

gnupload --to alpha.gnu.org:tar \
         --delete tar-1.20.90.tar.gz tar-1.20.90.tar.bz2 \
         --rmsymlink tar-latest.tar.gz tar-latest.tar.gz2 \
         -- tar-1.20.91.tar.gz

(double-dash in this case is needed to separate files to upload
from --rmsymlink arguments).     

5. Debugging features:

The --dry-run option causes gnupload to print what it would have done,
without actually uploading anything. It also prints the contents of the
created directive file(s).

The --to command option allows for the argument in form DIR:SUBDIR,
where DIR is an absolute directory name (--to /tmp:tar). This instructs
gnupload to copy the created files to /tmp. This is useful for debugging
the script itself and for debugging the software implementing automated
upload procedures.

Regards,
Sergey

Index: lib/gnupload
--- orig/gnupload       2008-12-28 15:44:04.000000000 +0200
+++ lib/gnupload        2009-01-24 15:54:33.000000000 +0200
@@ -1,9 +1,9 @@
 #!/bin/sh
 # Sign files and upload them.
 
-scriptversion=2008-11-12.21
+scriptversion=2009-01-23.23
 
-# Copyright (C) 2004, 2005, 2006, 2007, 2008  Free Software Foundation
+# Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009  Free Software Foundation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -24,97 +24,192 @@ set -e
 
 GPG='gpg --batch --no-tty'
 to=
-delete=false
+DRY_RUN=
+SYMLINK_FILES=
+DELETE_FILES=
+DELETE_SYMLINKS=
+COLLECT_VAR=
+DBG=
 
-usage="Usage: $0 [OPTIONS]... FILES...
+usage="Usage: $0 [OPTIONS]... [COMMAND] FILES... [[COMMAND] FILES...]
 
-Sign all FILES, and upload them to (or delete them from) selected
-destinations, according to
+Sign all FILES, and upload them to selected destinations, according to
 <http://www.gnu.org/prep/maintain/html_node/Automated-FTP-Uploads.html>.
 
+Commands:
+  --delete                 delete FILES from destination
+  --symlink                create symbolic links
+  --rmsymlink              remove symbolic links 
+  --                       treat the remaining arguments as files to upload
+  
 Options:
   --help                   print this help text and exit
   --to DEST                specify one destination for FILES
                            (multiple --to options are allowed)
   --user NAME              sign with key NAME
-  --delete                 delete FILES from destination instead of uploading
+  --symlink-re[=SED-EXPR]  use SED-EXPR to create symbolic links
+  --dry-run                do nothing, show everything
   --version                output version information and exit
 
+If --symlink-re without SED-EXPR is given, symlink target name is
+created by replacing version information with the word \`-latest',
+e.g.:
+
+  foo-1.3.4.tar.gz -> foo-latest.tar.gz
+
 Recognized destinations are:
   alpha.gnu.org:DIRECTORY
   savannah.gnu.org:DIRECTORY
   savannah.nongnu.org:DIRECTORY
   ftp.gnu.org:DIRECTORY
                            build directive files and upload files by FTP
+  download.gnu.org.ua:{alpha|ftp}/DIRECTORY
+                           build directive files and upload files by SFTP
   address@hidden:DIRECTORY    upload files with scp
 
-Deletion only works for ftp.gnu.org and alpha.gnu.org (using the
-archive: directive).  Otherwise it is a no-op.  Deleting a file foo also
-deletes foo.sig; do not specify the .sig explicitly.
-
-Simple single-target single-file examples:
-  gnupload --to alpha.gnu.org:automake automake-1.8.2b.tar.gz
-  gnupload --to ftp.gnu.org:automake automake-1.8.3.tar.gz
-  gnupload --to alpha.gnu.org:automake --delete automake-oops.tar.gz
+If the file .gnupload exists in the current working directory, its contents
+is prepended to the actual command line options.  Use this to keep your
+defaults.  Comments (#) and empty lines in .gnupload are allowed.
+
+Examples:
+1. Upload automake-1.8.2b.tar.gz and automake-1.8.2b.tar.bz2 to two sites:
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+           automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
 
-Multiple-target multiple-file example:
+2. Same as above, but also create symbolic links to automake-latest.tar.*:
   gnupload --to sources.redhat.com:~ftp/pub/automake \\
            --to alpha.gnu.org:automake \\
+          --symlink-re \\
+           automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
+
+3. Symlink automake-1.8.2b.tar.gz to automake-latest.tar.gz and
+automake-1.8.2b.tar.bz2 to automake-latest.tar.bz2 on both sites:
+
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+          --symlink automake-1.8.2b.tar.gz automake-latest.tar.gz \\
+                    automake-latest.tar.bz2 automake-latest.tar.bz2
+
+4. Delete automake-1.8.2a.tar.gz and .bz2, remove symlink
+automake-latest.tar.gz and upload automake-1.8.2b.tar.gz:
+
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+          --delete automake-1.8.2a.tar.gz automake-1.8.2a.tar.bz2 \\
+          --rmsymlink automake-latest.tar.gz \\
+          -- \\
            automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
 
 Report bugs to <address@hidden>.
 Send patches to <address@hidden>."
 
+# Read local configuration file
+if [ -r .gnupload ]; then
+  echo "$0: Reading configuration file .gnupload"
+  eval set -- "`sed 's/#.*$//;/^$/d' .gnupload | tr '\n' ' '` $*"
+fi
+    
 while test -n "$1"; do
   case $1 in
-    --delete)
-      delete=true
-      shift
-      ;;
-    --help)
-      echo "$usage"
-      exit $?
-      ;;
-    --to)
-      if test -z "$2"; then
-       echo "$0: Missing argument for --to" 1>&2
-        exit 1
-      else
-        to="$to $2"
-        shift 2
-      fi
+  -*) COLLECT_VAR=
+      case $1 in
+        --help)
+          echo "$usage"
+          exit $?
+          ;;
+        --to)
+          if test -z "$2"; then
+            echo "$0: Missing argument for --to" 1>&2
+            exit 1
+          else
+            to="$to $2"
+            shift 2
+          fi
+          ;;
+        --user)
+          if test -z "$2"; then
+            echo "$0: Missing argument for --user" 1>&2
+            exit 1
+          else
+            GPG="$GPG --local-user $2"
+            shift 2
+          fi
+          ;;
+       --delete)
+          COLLECT_VAR=DELETE_FILES
+          shift
+          ;;
+       --rmsymlink)
+         COLLECT_VAR=DELETE_SYMLINKS
+         shift
+         ;;
+        --symlink-re=*)
+          SYMLINK_EXPR=${1##--symlink=}
+          shift
+          ;;
+        --symlink-re)
+          SYMLINK_EXPR='s|-[0-9][0-9\.]*\(-[0-9][0-9]*\)\?\.|-latest.|'
+          shift
+          ;;
+        --symlink)
+          COLLECT_VAR=SYMLINK_FILES
+          shift
+          ;;
+        --dry-run|-n)
+          DRY_RUN=1
+          shift
+          ;;
+        --version)
+          echo "gnupload $scriptversion"
+          exit $?
+          ;;
+       --)
+         shift
+         break
+         ;;
+        -*)
+          echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2
+          exit 1
+          ;;
+      esac
       ;;
-    --user)
-      if test -z "$2"; then
-       echo "$0: Missing argument for --user" 1>&2
-        exit 1
+  *)  if test -z "$COLLECT_VAR"; then
+        break
       else
-        GPG="$GPG --local-user $2"
-        shift 2
+        eval $COLLECT_VAR=\"\$$COLLECT_VAR $1\"
+       shift
       fi
       ;;
-    --version)
-      echo "gnupload $scriptversion"
-      exit $?
-      ;;
-    -*)
-      echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2
-      exit 1
-      ;;
-    *)
-      break
-      ;;
-  esac
+  esac 
 done
 
-if test $# = 0; then
-  echo "$0: No file to upload or delete" 1>&2
+dprint() {
+  echo "Running $*..."
+}
+
+if test -n "$DRY_RUN"; then
+  DBG=dprint
+fi
+
+if test -z "$to"; then
+  echo "$0: Missing destination sites" >&2
   exit 1
-else
-  :
 fi
 
-if $delete; then :; else
+if test -n "$SYMLINK_FILES"; then
+  if test -n "`echo "$SYMLINK_FILES" | sed 's/[^ ]//g;s/  //g'`"; then
+    echo "$0: Odd number of symlink arguments" >&2
+    exit 1
+  fi
+fi  
+
+if test $# = 0; then
+  if test -z "${SYMLINK_FILES}${DELETE_FILES}${DELETE_SYMLINKS}"; then
+    echo "$0: No file to upload" 1>&2
+    exit 1
+  fi
+else
   # Make sure all files exist.  We don't want to ask
   # for the passphrase if the script will fail.
   for file
@@ -122,8 +217,15 @@ if $delete; then :; else
     if test ! -f $file; then
       echo "$0: Cannot find \`$file'" 1>&2
       exit 1
-    else
-      :
+    elif test -n "$SYMLINK_EXPR"; then
+      linkname=`echo $file | sed "$SYMLINK_EXPR"`
+      if test -z "$linkname"; then
+        echo "$0: symlink expression produces empty results" >&2
+        exit 1
+      elif test $linkname = $file; then
+        echo "$0: symlink expression does not alter file name" >&2
+        exit 1
+      fi
     fi
   done
 fi
@@ -143,8 +245,7 @@ read -r passphrase
 stty echo
 echo
 
-# Nothing to sign if deleting.
-if $delete; then :; else
+if test $# -ne 0; then
   for file
   do
     echo "Signing $file..."
@@ -153,59 +254,143 @@ if $delete; then :; else
   done
 fi
 
+# mkdirective DESTDIR BASE FILE STMT
+# Arguments: See upload, below
+mkdirective() {
+  stmt="$4"
+  if test -n "$3"; then
+    stmt="
+filename: $3$stmt"
+  fi
+  
+  cat >${2}.directive<<EOF
+version: 1.1
+directory: $1
+comment: gnupload v. $scriptversion$stmt
+EOF
+  if test -n "$DRY_RUN"; then
+    echo "File ${2}.directive:"
+    cat ${2}.directive
+    echo "File ${2}.directive:" | sed 's/./-/g'
+  fi
+}
+
+mksymlink() {
+  while test $# -ne 0
+  do
+    echo "symlink: $1 $2"
+    shift
+    shift
+  done
+}
+
+# upload DEST DESTDIR BASE FILE STMT FILES
+# Arguments:
+#  DEST     Destination site;
+#  DESTDIR  Destination directory;
+#  BASE     Base name for the directive file;
+#  FILE     Name of the file to distribute (may be empty);
+#  STMT     Additional statements for the directive file;
+#  FILES    List of files to upload.
+upload() {
+  dest=$1
+  destdir=$2
+  base=$3
+  file=$4
+  stmt=$5
+  files=$6
+  
+  rm -f $base.directive $base.directive.asc
+  case $dest in
+    alpha.gnu.org:*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $DBG ncftpput ftp-upload.gnu.org /incoming/alpha $files 
$base.directive.asc
+      ;;
+    ftp.gnu.org:*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $DBG ncftpput ftp-upload.gnu.org /incoming/ftp $files $base.directive.asc
+      ;;
+    savannah.gnu.org:*)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $DBG ncftpput savannah.gnu.org /incoming/savannah/$destdir $files
+      ;;
+    savannah.nongnu.org:*)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $DBG ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files
+      ;;
+    download.gnu.org.ua:alpha/*|download.gnu.org.ua:ftp/*)
+      mkdirective "${destdir#*/}" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      for f in $files $base.directive.asc
+      do
+        echo put $f
+      done | $DBG sftp -b - puszcza.gnu.org.ua:/incoming/${destdir%%/*}
+      ;;
+    /*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $DBG cp $files $base.directive.asc ${dest%%:*}
+      ;;
+    *)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $DBG scp $files $dest
+      ;;
+  esac
+  rm -f $base.directive $base.directive.asc
+}
+
+#####
+# Process any standalone directives
+stmt=
+if test -n "$SYMLINK_FILES"; then
+  stmt="$stmt
+`mksymlink $SYMLINK_FILES`"
+fi
+
+for file in $DELETE_FILES
+do
+  stmt="$stmt
+archive: $file"
+done   
+
+for file in $DELETE_SYMLINKS
+do
+  stmt="$stmt
+rmsymlink: $file"
+done   
+
+if test -n "$stmt"; then
+  for dest in $to
+  do
+    destdir=`echo $dest | sed 's/[^:]*://'`
+    upload "$dest" "$destdir" "`hostname`-$$" "" "$stmt"
+  done
+fi
+
+# Process actual uploads    
 for dest in $to
 do
   for file
   do
-    # Prepare arguments.
-    if $delete; then
-      echo "Removing $file from $dest..."
-      files=  # nothing to upload if deleting
-      directive="archive: $file"
-    else
-      echo "Uploading $file to $dest..."
-      files="$file $file.sig"
-      directive="filename: "`basename -- "$file"`
-    fi
+    echo "Uploading $file to $dest..."
+    stmt=
+    files="$file $file.sig"
     destdir=`echo $dest | sed 's/[^:]*://'`
-
-    case $dest in
-      alpha.gnu.org:*)
-       rm -f $file.directive $file.directive.asc
-       cat >$file.directive<<EOF
-version: 1.1
-directory: $destdir
-$directive
-EOF
-       echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive
-        ncftpput ftp-upload.gnu.org /incoming/alpha $files $file.directive.asc
-       rm -f $file.directive $file.directive.asc
-       ;;
-      ftp.gnu.org:*)
-       rm -f $file.directive $file.directive.asc
-       cat >$file.directive<<EOF
-version: 1.1
-directory: $destdir
-$directive
-EOF
-       echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive
-        ncftpput ftp-upload.gnu.org /incoming/ftp $files $file.directive.asc
-       rm -f $file.directive $file.directive.asc
-       ;;
-      savannah.gnu.org:*)
-        # We only know how to implement delete for {ftp,alpha}.gnu.org.
-        $delete \
-        || ncftpput savannah.gnu.org /incoming/savannah/$destdir $files
-       ;;
-      savannah.nongnu.org:*)
-        $delete \
-        || ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files
-       ;;
-      *)
-        $delete \
-       || scp $files $dest
-       ;;
-    esac
+    if test -n "$SYMLINK_EXPR"; then
+      linkname=`echo $file | sed "$SYMLINK_EXPR"`
+      stmt="$stmt  
+symlink: $file $linkname
+symlink: $file.sig $linkname.sig"
+    fi
+    upload "$dest" "$destdir" "$file" "$file" "$stmt" "$files"
   done
 done