Re: "here strings" and tmpfiles

[Chet Ramey]

On 4/10/19 7:18 PM, Daniel Kahn Gillmor wrote:
> On Wed 2019-04-10 16:16:44 -0400, Chet Ramey wrote:
>> Is it just that people have not realized all along that most shells,
>> certainly all historical shells, that implement here documents use temp
>> files to do it? It's really only the ash-based shells (not an insignificant
>> portion of the shells in use, for sure) that use pipes exclusively.
> 
> Yes, i think most people have not realized that, all along.  I certainly
> didn't, and i've been using bash for decades now.

It's not just bash.

>> I'd love to see these concerns articulated concretely, so we can analyze
>> the risk in terms of the temp-file-on-disk implementation.
> 
> I tried to do that at the start of this thread, but i'm happy to try
> gain.
> 
> data written to the local filesystem can be discovered by someone
> analyzing the disk controller data path, or by someone with access to
> the underlying storage medium.

If you have someone who can access that data path, or someone with access
to physical storage, I suggest that you have much greater concerns than
whether or not you can read data in a shell script. This is data that is,
for the most part, already in the script itself.

> So anyone who puts anything sensitive into a here document or herestring
> using bash is triggering a potential leak of that sensitive data.  While
> the contents of many here documents are clearly not sensitive, or are
> destined for the filesystem anyway, *some* data placed in heredocs *are*
> sensitive, and are intended to be ephemeral. 

Here's the thing. If this had been a concern, it would never have happened.
Shells have always used temp files for here documents. You can't just chalk
it up to people not knowing -- the risk has been deemed acceptable.


>> I'm sure Jason has been advocating that nobody use here documents since the
>> Bourne shell introduced them, since that's the position he advocates above.
> 
> AFAICT, Jason only recently became aware of this, so he certainly hasn't
> been advocating no one use here docuemnts  the Bourne
> shell introduced them:

I'm exaggerating for effect, with maybe some snark. His position is, in
effect, that nobody should use here documents or here strings, since, by
his definition, every shell except ash descendents does it insecurely. I'm
sure it's more nuanced than that, and I admit I'm exaggerating for effect.
The intent is to show that people have known about this for 40 years and
to incorporate that into the risk calculation.


> He's not the only person who has used them in an attempt at simplicity
> or cleanliness only to find that this is a risk if sensitive data is
> included in them.
> 
>> Unnecessary hyperbole diminshes the force of your argument.
> 
> i wasn't intending to use unnecessary hyperbole

(It was the original message from Jason.)

> -- if you want to give
> guidance that most people can follow, it's necessary for the guidance to
> be clear and simple.  Complicated guidance gets ignored, or people get
> confused and can't follow it closely.

Starting off with things like "buggy, vulnerable, scary" and "don't use
herestrings" (or, by extension, here-documents) is unnecessary hyperbole
in my book. That's why I'm trying to solicit genuine use cases and have a
risk management discussion.

> fwiw, i'm not here to have an argument with you or to "win" at anything
> -- i'm trying to help improve bash and the ecosystem around it, the same
> as you are.  I certainly have no interest in having this be a contest.
> If you see this discussion as an argument, please let me know and i'll
> stop following up on this thread and adjust my expectations accordingly.

There's no "winning" here. It's a question of analyzing risk and deciding
where to place this issue in a priority-ordered list. An implementation has
a cost, even if I'm not the one who does it, and should have a
corresponding benefit.


> 
>> The real question is why the ephemerality of heredocs and herestrings is
>> important -- what is the use case that requires it, as opposed to it simply
>> being an implementation detail?
> 
> This is an implementation detail that isn't well-documented. 

Implementation details are rarely documented. I would argue that they
don't need to be.


>>> Either of these implementations would be great, though i note that how
>>> bash handles the extra fork (whether the parent or the child does the
>>> pipe writing) might complicate the use of the wait builtin (or any
>>> wait(-1) syscall) in the spawned process (e.g. similar to this bug
>>> report: https://bugs.debian.org/920455)
>>
>> It wouldn't really affect that. The reason `wait' waits for process
>> substitution processes is that they set $!, making them "known to the
>> shell" and subject to wait without arguments.
>>
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/wait.html#tag_20_153
> 
> This behavior is actually different between bash 4.4.18 and 5.0, but i
> think this is a separate discussion, so i'll defer it to a different
> thread to avoid confusion here :) 

Yes. It was reported as a bug against bash-4.3, back in 2015, and I agreed
with it, so the behavior changed.

http://lists.gnu.org/archive/html/bug-bash/2015-03/msg00076.html

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

signature.asc
Description: OpenPGP digital signature