Re: CSL-JSON support for =parsebib=

[Titus von der Malsburg]

Hi all,

I’m the maintainer of bibtex-completion, helm-bibtex, and ivy-bibtex.  My name 
is actually Titus, not Theo ;)

Cool to see that the ecosystem around academic writing in org mode is 
developing so nicely.  I use org mode for this purpose every single working day 
and it’s amazing already.  I have to confess, though, that I haven’t been 
keeping up with recent developments.  I just saw the recent thread about the 
citation syntax.  (Thanks to Bruce D’Arcus for pointing me to it.)  Is there a 
good place where I can read up on the current efforts and plans regarding 
citations, bibliographies and so on (I mean other than reading the last couple 
of months of the mailing list archive)?

Regarding the symbols vs. string issue:  I don’t have a strong opinion, but 
personally tend to favor a conservative solution that avoids braking changes.  
First, it’s difficult to predict how switching to symbols is going to affect 
other software including custom code written by users.  Second, JSON key names 
can contain spaces and other weird stuff.  So strings are perhaps a more 
natural choice anyway.  (It appears that you can actually configure the JSON 
parser to use strings instead of symbols.  See variable `json-key-type`.)  
Third, as you say, it would also be nice to maintain compatibility with 
bibtex.el.  Finally, it’s not necessarily clear that avoiding the conversion to 
strings saves sufficiently many CPU cycles to justify the effort.  (But this 
may be a non-issue anyway, if the JSON parser can return strings directly.)

Having said that, I’d be happy to merge a PR that that implements the switch to 
symbols in bibtex-completion if that’s the consensus.  Touches a substantial 
number of lines, but should nonetheless be relatively straightforward.

Regarding support for CSL-JSON: bibtex-completion is currently very 
BibTeX-oriented and uses fairly low-level parsing functions from parsebib.  We 
could add similar support for CSL-JSON but things would become messy.  (It’s 
already a bit ugly, I have to say, which is entirely my fault.)  It might be 
more elegant to have a higher-level API in parsebib.  This API could perhaps 
even abstract away from the underlying format (BibTeX, CSL-JSON, or others in 
the future?).  This would substantially simplify matters in bibtex-completion, 
but would also enable many other cool uses of parsebib.

Some rough ideas for such an API (just for illustration):
- A function that returns all entries in a .bib or CSL-JSON file.
- A function that returns an entry with a specific key (or multiple entries).
- Functions for resolving strings and cross-references.

So much for now.

  Titus


On 2021-05-07 Fri 11:17, Joost Kremers wrote:
> Hi,
>
> [Cc-ing Theo von der Malsburg]
>
> Now that Org is getting support for Citeproc, it could be useful to add 
> support
> for the CSL-JSON format for bibliographic data to Emacs. Therefore, after a
> friendly request from Denis Maier, I have added support for this format to the
> =parsebib= library.
>
> Since =parsebib= is used by =bibtex-completions=, which in turn is used by
> =bibtex-actions=, =helm-bibtex=, =ivy-bibtex=, =org-ref= and 
> =org-roam-bibtex=,
> this is a first step in making bibliographic data in =.json= format directly
> available to Org users, without the need of any BibTeX conversion.
>
> [Boy, look at me doing the marketing speak! :D ]
>
> Anyway, this really is the first step. =bibtex-completion= will need to be
> modified in order to make use of the new functionality, and the same may be 
> true
> of the packages based on it.
>
> At this point, the new code isn't merged into =master= yet. It is available in
> the =wip/csl= branch of =parsebib='s Github repo:
>
> https://github.com/joostkremers/parsebib/tree/wip/csl
>
> The README has most of the details. I appreciate any and all comments,
> suggestions and tips.
>
> For those maintaining packages based on =parsebib=, I have at least one
> question: currently, =parsebib= returns a BibTeX entry in the form of an alist
> of =(<field> . <value>)= pairs, where both =<field>= and =<value>= are 
> strings.
> A CSL-JSON entry is returned as an alist, but the =<field>= names are symbols,
> not strings.
>
> It would be extremely impractical to return the JSON data with strings as 
> field
> names, because the JSON parsing libraries in Emacs return symbols, so 
> converting
> them would take time. Plus, those libraries also expect symbols when 
> serialising
> Elisp data to JSON. (Which I intend to make use of in Ebib later on.)
>
> It would be easier to modify the BibTeX output to return field names as 
> symbols.
> I originally chose strings, because that's what =bibtex.el= uses, making it a
> little easier to integrate with it.
>
> So the question: would it be helpful to make this change to the BibTeX data, 
> so
> that the data from both sources uses the same format? Or would it be better to
> keep it as it is, even if that means that BibTeX data and JSON data isn't
> compatible?
>
> TIA
>
> Joost
>
>
> -- 
> Joost Kremers
> Life has its moments