Re: [O] Citation syntax: a revised proposal

[Tory S. Anderson]

+1

Thanks for the work substantiating the idea. 

Richard Lawrence <address@hidden> writes:

> Hi everyone,
>
> Since discussion seems to have petered out on the previous thread (see:
> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
> go back over the discussion and write up a concrete proposal for
> citation syntax.
>
> This proposal represents my attempt to formulate a syntax that is easy
> to read, easy to parse, and covers all the use-cases that people
> mentioned as being important.  It is surely not perfect, but I learned a
> lot from the previous thread, and I hope something like this will serve
> the community's needs.
>
> The proposal is below, both inline (for easy quoting) and attached (for
> easy reading).  To keep it relatively short, I have mostly not explained
> my reasoning for the choices I made, but I am happy to do so here if
> anyone has questions.
>
> I welcome feedback, comments, criticisms, and objections on any point.
> However, since we've already had a long discussion about this, I
> respectfully request that we try to keep this thread focused.  To that
> end, I suggest:
>
>   1) If you have criticisms or objections, please try to indicate
>      whether you think they are `substantive' (e.g., you see a problem
>      that would prevent you from using this syntax, or prevent Org from
>      implementing it) or not (e.g., you would prefer a slightly
>      different but equivalent way of expressing something).
>
>   2) If you wish to express an opinion about the proposal without
>      offering further comments, let us know by just replying with +1
>      (meaning you'd like to see this syntax, or something reasonably
>      similar to it, be adopted), 0, or -1 (meaning you'd prefer not to
>      see this syntax or anything similar to it adopted).
>
> I guess this is my Valentine to the Org community. :) Thanks for reading!
>
> Best,
> Richard
>
> #+TITLE: Citation syntax, a revised proposal
> #+DATE: <2015-02-14 Sat>
>
> #+AUTHOR: Richard Lawrence
> #+EMAIL: address@hidden
>
> #+LANGUAGE: en
> #+SELECT_TAGS: export
>
> #+EXCLUDE_TAGS: noexport
>
> * Citation syntax
> ** Requirements
> A citation is a textual reference to one or more individual works,
> together with other information about those works, grouped together in
> a single place.  
>
> Within a citation, each reference to an individual work needs to be
> capable of containing:
>   1) a database key that references the cited work
>   2) prefix / pre-note
>   3) suffix / post-note
>      
> Whole citations also need:
>   4) address@hidden a way of specifying whether the citation is in-text or
>      parenthetical
>   5) a way of representing a common prefix and suffix, if the citation
>      is a multi-cite
>   6) a way of specifying whether the citation should produce a
>      complete bibliography entry in-place
>   7) an extensible way of specifying formatting properties to export
>      filters and/or specific export backends 
>      
> ** Citation definitions
> *** Citation keys; bibliography references vs. complete entries 
> A citation key consists of a unique label preceded by a flag, which is
> optionally preceded by a hyphen.
>
> The flag is either `@' or `&'.  `@' indicates that the citation should
> produce a normal reference to the bibliography entry for the cited
> work (in whatever style the document uses), located elsewhere.
>
> The `&' flag indicates that the citation should produce a complete
> bibliography entry for the cited work in the place where the citation
> appears.
>
> The optional hyphen (`-') indicates that the author's name should be
> suppressed from the rendered citation.  (Note that this is only useful
> in author-X citation styles; it should have no effect in numeric
> styles.)
>
> *** Basic citations: Parenthetical vs. in-text 
> There are two basic types of citation: /parenthetical/ and /in-text/.
> Each of these may contain references to one or more individual works.
>
> The difference between parenthetical and in-text citations is
> expressed using parentheses around the /first/ citation key.  A
> parenthetical citation has such parentheses around the first citation
> key; an in-text citation lacks them.  (Parentheses around non-initial
> keys are permitted for visual consistency and to keep the grammar
> simple, but have no meaning.)
>
> A citation thus consists in general of a bracketed list, beginning
> with `cite:', of one or more individual references, each of which:
>   - may contain a prefix,
>   - must contain a citation key, which may or may not be surrounded by `(...)'
>   - and may contain a suffix
> Individual references are separated by semi-colons.
>
> There are also two special cases to make simple-but-common uses very
> easy to type and read:
>   1) a parenthetical citation for a single work with no prefix and
>      suffix may be written by just surrounding the key with brackets,
>      like: address@hidden
>   2) an in-text citation for a single work with no prefix and suffix
>      may be written as a /bare/ key, without brackets, like: @Doe99.
> (Thus, in both of the `simple' cases, one less level of bracketing is
> required.)
>
> Prefix and suffix text are regular Org text, which are allowed to
> contain various kinds of Org markup (see the grammar below for a
> complete list).
>
> *** Multi-cite citations 
> Multi-cite citations are distinguished from basic parenthetical and
> in-text citations by the presence of an optional common prefix or
> common suffix (which may not contain keys).  If present, the common
> prefix must occur before the first individual reference, and the
> common suffix must occur after the last individual reference.  The
> common prefix and suffix are separated from the individual references
> by semi-colons.
>
> *** Examples of main citation syntax
> Basic parenthetical citation:
>
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. [cite: (@Doe99)]
> #+END_QUOTE
>
>
> Basic parenthetical citation using special-case syntax:
>
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. address@hidden
> #+END_QUOTE
>
>
> Parenthetical citation with multiple works and prefix and suffix:
>
> #+BEGIN_QUOTE
> The nineteenth century was in fact lovely [cite: see (@Doe99) p. 44;
> @Smith2000 has a review].
> #+END_QUOTE
>
>
> Basic in-text citation with a suffix:
>
> #+BEGIN_QUOTE
> As [cite: @Doe99 p. 44] says, the nineteenth century was very interesting.
> #+END_QUOTE
>
>
> In-text citation using special-case syntax:
>
> #+BEGIN_QUOTE
> @Doe2000 explains that the twentieth century was even more interesting. 
> #+END_QUOTE
>
>
> In-text citation with author suppressed:
>
> #+BEGIN_QUOTE
> As Doe explained in his address@hidden, the twentieth century was somewhat
> less interesting than previously thought.
> #+END_QUOTE
>
>
> Parenthetical citation with full-entry key:
>
> #+BEGIN_QUOTE
> A complete bibliography entry follows in parentheses. [cite: (&Doe99)]
> A complete bibliography entry follows in parentheses. [&Doe99]
> #+END_QUOTE
>
>
> In-text citation with full-entry key:
>
> #+BEGIN_QUOTE
> A complete bibliography entry follows: [cite: &Doe99].
> A complete bibliography entry follows: &Doe99.
> #+END_QUOTE
>
>
> Full-entry in-text citation, in a footnote:
>
> #+BEGIN_QUOTE
> Doe exhibits unusual scholarship.[fn:: &Doe99.]
> #+END_QUOTE
>
>
> In-text citation, with a complete bibliography entry minus the author
> in a footnote, plus a suffix:
>
> #+BEGIN_QUOTE
> @Doe99 exhibits unusual scholarship.[fn:1]
>
> [fn:1] [cite: -&Doe99 Cf. especially section 4.]
> #+END_QUOTE
>
>
> In-text multi-cite:
>
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; @Smith1998;
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
>
> Parenthetical multi-cite:
>
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; (@Smith1998);
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
>
> *** Syntax for extensions 
> Additional information can be supplied in a citation that may affect
> how export filters or particular backends format it.
>
> This additional information may be supplied following the brackets of
> a citation between the following delimiters: `%%( ... )'.
>
> (Note: I am proposing that this expression go /after/ the main
> citation brackets both because it visually separates this extra
> information from the main citation, and in order to avoid imposing any
> further syntactic restriction on suffixes.)
>
> At least for now, any information supplied this way is /strictly the
> user's responsibility/ to interpret (e.g., using an export filter).
> This means that citations that have information like this are not
> portable and might not be exported correctly:
>   - in other users' setups
>   - by particular backends
>   - by future versions of Org
>
> I will not deal with the details of how this additional information
> should be syntactically represented, since this has not really been
> discussed.  But I suggest that, to deal with the complexities of
> additional information in full generality, something like a complete
> Lisp list is required.  Thus, I suggest that this additional
> information simply be represented as a Lisp list.  (Besides
> generality, this has the benefit of making the syntax easy to parse:
> the parser can just call Elisp's read function with a marker after the
> `%%'.)
>
> I provide these examples merely to illustrate the possibilities here:
>
> #+BEGIN_QUOTE
> @vonNeumann1930 %%(:type genitive :capitalize t) model can only handle
> a limited range of observed cases.
>
> @McCarthy1950 %%('s) clever use of Lisp syntax was also used to
> express the Saxon genitive.
>
> For more, see Ref. @Doe99 %%(:type refnum :follow-to "some.pdf").
>
> Even more complicated examples occur after Doe's famous article from
> [cite: @Doe99] %%(:type date-only).
>
> And in [cite: @Doe2000] %%(:attr_latex (:format-string
> "\citeyear{%KEY}") :attr_html (:only-fields (month year))), Doe
> finally realized that arbitrary complexity was a powerful but
> double-edged sword.
>
> @_aParticularlyUGLYkey:is-this-one %%(:overlay "Nice Display")
> #+END_QUOTE 
>
> ** Grammar
> This section formally documents the syntax of citations discussed
> above.  
>
> To represent the syntax of citations, we need a category of /citation/
> objects, which require the following properties (the names here are not
> important and could be changed):
>   - is-parenthetical (boolean; nil means is in-text)
>   - common-prefix (text)
>   - common-suffix (text)
>   - references (list)
>   - extra-info (list)
>
> Each reference in the list of references should be a plist with the
> following properties:
>   - prefix (text)
>   - suffix (text)
>   - key (string)
>   - is-parenthesized (boolean; t means key was parenthesized; only
>     significant for the first reference in a citation)
>   - suppress-author (boolean; t means author name should not be output)
>   - is-full (boolean; t means a full bibliography entry should be
>     output in-place) 
>
> The category of citations has the following grammar:
>   - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
>   - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
>     CITATION-LIST whose first individual INDIVIDUAL-REFERENCE is a
>     PARENTHESIZED-KEY
>   - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
>     CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.
>   - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
>     brackets, optionally followed by an EXTRA-INFO clause.
>   - A SIMPLE-IN-TEXT is a BARE-KEY, optionally followed by an
>     EXTRA-INFO clause
>   - A CITATION-LIST has the format
>        [cite: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX] 
> EXTRA-INFO
>     where the initial PREFIX, final SUFFIX, and EXTRA-INFO clause are
>     optional.  At least one INDIVIDUAL-REFERENCE must be present. 
>   - An INDIVIDUAL-REFERENCE has the format: 
>        PREFIX KEY-MAYBE-PARENS SUFFIX
>     The KEY-MAYBE-PARENS is obligatory, and the prefix and suffix
>     are optional.
>   - A KEY-MAYBE-PARENS is either a BARE-KEY or PARENTHESIZED-KEY
>   - A BARE-KEY is a KEY with immediately-preceding whitespace
>   - A PARENTHESIZED-KEY is a KEY immediately surrounded by `(' and `)'.
>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>     `&' followed by a string of characters which begins with a letter
>     or `_', and may contain alphanumeric characters and the following
>     internal punctuation characters:
>        :.#$%&-+?<>~/
>   - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and
>     KEY-MAYBE-PARENs) which may contain only the following Org
>     objects:
>     - bold
>     - code
>     - entity
>     - italic
>     - latex-fragment
>     - line-break
>     - strike-through
>     - subscript
>     - superscript
>     - underline
>     - superscript
>     (Note that this list could be extended somewhat if necessary.)
>   - An EXTRA-INFO clause consists of data not specified by this
>     grammar, in between `%%(' and `)'
>
> ** Outstanding issues
> It seems to me that there are potential problems with the above
> proposal in a number of areas, but I cannot tell how serious they are,
> or what changes (if any) should be made to solve them.  I don't
> pretend that this is an exhaustive list:
>   1) *Nesting.*  I have favored LaTeX compatibility for in-text
>      citations with multiple references; but this means there is no
>      way to `nest' citations.  Thus, there is no way to express (in
>      the main syntax) what Pandoc expresses as:
>         @Doe99 [p. 34; see also @DoeRoe2000]
>      which renders like:
>         Doe (1999, p. 34; see also Doe and Roe 2000)
>      Instead, since a citation is in-text or parenthetical as a whole,
>      the equivalent in the above syntax
>         [cite: @Doe99 p. 34; see also @DoeRoe2000]
>      should render like:
>       Doe (1999, p. 34), see also Doe and Roe (2000).
>      I am not certain if Pandoc-like output is important in this case.
>      The few people who commented on this said that it was not. 
>   2) *Limitations on prefixes and suffixes.*  There may be legitimate
>      uses of `@', `;', `]', etc. inside prefix or suffix text that the
>      above syntax does not allow.  Examples might include:
>      - use of semi-colons as part of the prefix/suffix text
>      - footnotes, links, or timestamps inside a prefix/suffix
>      I am not certain how important these cases are.  If they are
>      important, some of them might be able to be worked around with
>      entities.
>   3) *Edge cases.* The above syntax may make it possible to express
>      things that don't make sense, or would be too difficult to
>      export.  The only one I can think of is that it is possible to
>      mix `@'-style and `&'-style keys in the same citation.  I am not
>      sure if this should be forbidden; it may sometimes make sense.
>      It may also be possible to express things that external tools,
>      such as citeproc-js, don't know how to process.  I do not have a
>      good sense of what, if anything, falls into that category, and
>      what should be done about it.
>   4) *Citation commands.*  Rather than introduce an explicit
>      representation for different citation commands/types, I have used
>      different parts of the syntax to express the common distinctions
>      that people mentioned.  I suggest that, for now, anything beyond
>      these basic distinctions be left to the user-extension syntax.
>      However, if it becomes clear in the future that there is a need
>      to add a representation for a command to the main syntax, there
>      is a natural place to do so: immediately after the `cite:' tag
>      (as Nicolas suggested).
>
> Also, I have not said anything in this proposal to address how other
> document metadata should be represented, which has not been discussed
> much on the list.  I think this should be discussed separately.
>