Re: Extensions to the `base64' Program

[Pádraig Brady]

On 31/08/15 14:09, Matt Walker wrote:
> Dear coreutils Development Team,
> 
> I recently ran into a problem when using the base64 program to decode strings 
> received from the GMail API.  Here is a relevant Stack Overflow post:
> 
> http://stackoverflow.com/questions/24812139/base64-decoding-of-mime-email-not-working-gmail-api
> 
> And here is a trouble string:
> 
> IFN0YXkgbW9yZSBvcmdhbml6ZWQgd2l0aCBHbWFpbCdzIGluYm94DQpbaW1hZ2U6IEdvb2dsZV0NCg0KSGkgTWF0dA0KDQpHbWFpbCdzIGluYm94IHB1dHMgeW91IGluIGNvbnRyb2wNCg0KW2ltYWdlOiBJbmJveCB2aWRlb10gPGh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNoP3Y9Q0ZmN2RsZXdKdXM-DQpNZWV0IHRoZSBpbmJveA0KDQpHbWFpbCdzIGluYm94IHNvcnRzIHlvdXIgZW1haWwgaW50byBjYXRlZ29yaWVzIHNvIHlvdSBjYW4gc2VlIHdoYXQncyBuZXcgYXQNCmEgZ2xhbmNlLCBkZWNpZGUgd2hpY2ggZW1haWxzIHlvdSB3YW50IHRvIHJlYWQgd2hlbiBhbmQgdmlldyBzaW1pbGFyIHR5cGVzDQpvZiBlbWFpbHMgdG9nZXRoZXIuIFdhdGNoIHRoZSB2aWRlbw0KPGh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNoP3Y9Q0ZmN2RsZXdKdXM-DQpbaW1hZ2U6IFNvY2lhbCB0YWJdDQpDaG9vc2UgeW91ciBjYXRlZ29yaWVzDQoNClRoZSBTb2NpYWwgYW5kIFByb21vdGlvbnMgY2F0ZWdvcmllcyBhcmUgb24gYnkgZGVmYXVsdC4gQWRkIGNhdGVnb3JpZXMgbGlrZQ0KVXBkYXRlcyBhbmQgRm9ydW1zIG9yIHJlbW92ZSBjYXRlZ29yaWVzIHRvIGhhdmUgdGhvc2UgZW1haWxzIHNob3cgdXAgaW4NCnlvdXIgUHJpbWFyeSBpbmJveC4gTGVhcm4gaG93IHRvIGNob29zZSBjYXRlZ29yaWVzDQo8aHR0cHM6Ly9zdXBwb3J0Lmdvb2dsZS5jb20vbWFpbC8_aGw9ZW4mcD1pbmJveGNhdGVnb3JpZXNfYWxsPg0KW2ltY
WdlOiBDdXN0b21pemVdDQpDdXN0b21pemUgeW91ciBpbmJveA0KDQpJZiB5b3Ugc2VlIGEgbWVzc2FnZSB5b3Ugd2FudCBpbiBhIGRpZmZlcmVudCBjYXRlZ29yeSwgeW91IGNhbiBtb3ZlIGl0DQp0aGVyZS4gT24gbW9iaWxlIGRldmljZXMsIHlvdSBjYW4gZXZlbiBjaG9vc2Ugd2hpY2ggY2F0ZWdvcmllcyBjcmVhdGUgYQ0Kbm90aWZpY2F0aW9uLiBNb3JlIGN1c3RvbWl6YXRpb24gdGlwcw0KPGh0dHBzOi8vc3VwcG9ydC5nb29nbGUuY29tL21haWwvP2hsPWVuJnA9aW5ib3hjYXRlZ29yaWVzX2FsbD4NCg0KDQoNClRvIGxlYXJuIG1vcmUgYWJvdXQgR21haWwncyBpbmJveCwgY2hlY2sgb3V0IHRoZSBoZWxwIGNlbnRlcg0KPGh0dHBzOi8vc3VwcG9ydC5nb29nbGUuY29tL21haWwvP2hsPWVuJnA9aW5ib3hjYXRlZ29yaWVzX2FsbD4gb3Igd2F0Y2ggdGhlDQp2aWRlbyA8aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g_dj1DRmY3ZGxld0p1cyZhbXA-Lg0KDQoNCltpbWFnZTogR21haWwgaWNvbl1IYXBweSBlbWFpbGluZywNClRoZSBHbWFpbCBUZWFtDQrCqSAyMDE1IEdvb2dsZSBJbmMuIDE2MDAgQW1waGl0aGVhdHJlIFBhcmt3YXksIE1vdW50YWluIFZpZXcsIENBIDk0MDQzDQo=
> 
> The problem is that `base64' doesn't support the RFC 4648 standard.  An 
> obvious work around is to do something akin to "cat <file> | sed 's/-/+/' | 
> sed 's|_|/|' | base64 --decode" or whatever (forgive the double sed please).  
> However, it would be more GNU-y I think to support the "web" or "url-safe" 
> version of Base64 encoding directly in the program as an extension.
> 
> If nobody feels like doing it, I could probably tackle the problem.  I just 
> want to ensure people want this first.
> 
> Some follow up thoughts if the change is deemed acceptable: should decode try 
> to be "smart" and decode according to the first decoding that makes sense, or 
> should there possibly be a "--smart-decode" option, or should we just have 
> "--encoding=web" or whatever and call it a day?

This is one of those marginal decisions.
I'm 60:40 against adding that because:

  It's easy to achieve with existing tools:
    b64_url_encode() { base64 | tr '+/' '-_'; }
    b64_url_decode() { tr -- '-_' '+/' | base64 -d; }

  Generally we avoid adding functionality unless it gives
  a functional advantage to include. That's not the case here
  and even performance wise, base64 doesn't generally process
  large amounts of data where the extra data copy would be significant.

  There's talk of adding base32 util, with which
  it would be nice to keep the same args as base64,
  and a --url option would be inappropriate there.

  There are probably other variants of base64 in use
  (like also transforming the '=' padding char for
  easier to read URLs), and so using the external tr
  solution is more general.

thanks,
Pádraig.