[bug #20932] locale2lang-0.1 - BUG + fix: fallback to language_only extracted from Accept-Language http header is needed (fwd)

[boud]

hi samizdat-devel,

i think the bug + fix below could solve a practical problem for many
non-english speaking indymedia collectives or other independent media
groups: "activist spam" which someone posts as identical articles, in
English, on several dozen different local indymedia sites. This sort
of article is sometimes serious and sometimes more like conspiracy
theory, but AFAIK the people doing it usually have "en-US"  in their
browser http Accept-Language header.  If the mono option is enabled
by sysadmin and the user chooses this option:
 https://savannah.nongnu.org/patch/?6167
and if his/her preferred language is non-English, then s/he will 
not even notice the presence of the "activist spam" article.

This could possibly imply less intervention or less urgent intervention
is needed by moderators (depending on the editorial policy, of course):
the decision and filtering of what to read (ignoring non-preferred languages
rather than just not preferring them) is made by the reader, not by
an editorial collective de facto deciding on behalf of the whole local
activist community.  (Of course, ignoring real spam is not a good idea.
For that we have the Antispam class in antispam.rb .)

Anyway, read on if you're interested. :)

cheers
boud



[bug #20932] locale2lang-0.1 - BUG + fix: fallback to language_only
    extracted from Accept-Language http header is needed


URL:
  <http://savannah.nongnu.org/bugs/?20932>

                 Summary: locale2lang-0.1 - BUG + fix: fallback to
language_only extracted from Accept-Language http header is needed
                 Project: Samizdat
            Submitted by: boud
            Submitted on: Wednesday 08/29/2007 at 22:04
                Category: None
                Severity: 3 - Normal
                  Status: Works For Me
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any

    _______________________________________________________

Details:

PROBLEM:
Even though RFC 2616 recommends that user clients (e.g. firefox)
should recommend to their users to have a backup generic language
without a country code (e.g. "en" in addition to "en-US"),
in practice most users do not do this.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4

In particular, for non-english language samizdat sites, this
means that people who have only "en-US" sent by their browser
end up getting the default local language of the site. Their
article then gets published with message.language = the local
language, not "en", since formally speaking, they state that
they prefer "en-US" to the local language, but they are not
interested in "en".

This implies that if people want to add a local translation
rather than hiding an article, then moderator intervention
is required to change the language (unless the user chose
open editing).

Moreover, the monolanguage patch https://savannah.nongnu.org/patch/?6167
(still under
development) will fail to exclude these type of articles under
the mono option, since their language is wrongly tagged (except
for a pedantic interpretation of their request).

For these reasons, i'm putting this as a bug (with a proposed
fix) rather than a patch.

PROPOSED SOLUTION:
This requires a reasonably modern version of ruby gettext, e.g.
debian 1.7.0-1 or later.  Copying gettext/locale_object.rb
into an older installation and using an appropriate require
statement is a hack to avoid a full installation of a recent
gettext.

The idea is that if a requested accept-language in the list
is not found, then parse off the language part of it and try
that instead. This could potentially create multiple entries
of the same language, but i suspect that shouldn't be a
problem.


--- s070818/samizdat/lib/samizdat/engine/request.rb     2007-08-14
01:16:53.000000000 +0200
+++ /usr/lib/ruby/1.8/samizdat/engine/request.rb        2007-08-29
23:02:06.869866760 +0200
@@ -165,8 +173,17 @@
       accept.scan(/([^ ,;]+)(?:;q=([^ ,;]+))?/).collect {|l, q|
         [l, (q ? q.to_f : 1.0)]
       }.sort_by {|l, q| -q }.each {|l, q|
-        @accept_language.push l if config_lang.include? l
+#        @accept_language.push l if config_lang.include? l
+        if config_lang.include? l
+          @accept_language.push l
+        else
+          # try converting full locale (language tag) to ISO-639 language
only
+          # http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4
+          lang_only = Locale::Object.new(l).language
+          @accept_language.push lang_only if config_lang.include? lang_only
+        end
       }
+
     # lang cookie overrides Accept-Language
     lang = cookie('lang') and config_lang.include? lang and
       @accept_language.unshift lang


FUTURE EXTENSIONS:
The relations between human languages and how close or distant
they are are well studied. A measure of the distance between
different languages could potentially be used as a backup to
find the likely closest language that a user would prefer
rather than just taking what is considered the "language"
part of the locale/Accept-Language string.

Since the "narratives" which claim different national identities
often try to claim sharp distinctions between closely related
languages, this could potentially be a quite politically
sensitive issue. This is not surprising, and is not IMHO an
argument against doing this: an RDF engine specifically aimed
for grassroots, non-authoritarian media is necessarily going
to challenge artificial linguistic barriers if it's to get
somewhere near doing its task.

In any case, users with their own notions of language preferences
would still be able to state this by all the presently available
methods; adding a language metric would only be used as a
fallback.

COMMENT:
The Locale:: module could probably also be used to check the
config files for valid languages and warn about invalid
languages/locales.



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Wednesday 08/29/2007 at 22:04  Name: 070829_locale2lang-0.1  Size: 997B
  By: boud

<http://savannah.nongnu.org/bugs/download.php?file_id=13832>

    _______________________________________________________

Reply to this item at:

  <http://savannah.nongnu.org/bugs/?20932>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.nongnu.org/