Re: making a file sparse - in-place?

[Pádraig Brady]

On 01/24/2014 12:47 AM, Bernhard Voelker wrote:
> Inspired by a recent post on util-linux ML [1], talking about turning
> a file into a sparse file in-place, i.e. not using a 2-step approach
> like `cp --sparse file file2 && mv file2 file`), I thought, hey, don't
> we have this in coreutils already?
> 
> a)
> Therefore, I tried
>   $ dd if=file of=file conv=sparse
> 
> ... well, the data in 'file' was lost because the file got truncated
> by the open(...,O_TRUNC) call - even without a warning.
> POSIX [2] does not say anything about what dd(1) should do if the
> input and output file are the same.
> Is there a reason why dd should not deny to take "if=file of=file"
> without "conv=notrunc", or why we shouldn't warn at least?
> (Okay, the latter is not much of use because it's too late then
> anyway with a simple warning.) So I'd favor outputting an error
> diagnostic and exit(EXIT_FAILURE) in this case.

Fair point. Maybe.

> b)
> Then, I tried
>   $ dd if=file of=file conv=sparse,notrunc
> to avoid truncating the output file. That didn't corrupt the data,
> but the file still was not sparse afterward.
> What's the reason for conv=sparse not to work in this situation?
> BTW: generally, writing to the same file seems to work, e.g.:
>   dd if=file of=file conv=ucase,notrunc

To deallocate the zeros we'd have to use fallocate(FALLOC_FL_PUNCH_HOLE).
Also for efficiency reasons it would be nice to detect holes efficiently.
We can do this with the current fiemap code, but really we should try
and use the new SEEK_HOLE functionality available in the kernel.

> c)
> As a last method, I tried
>   $ cp --force --backup --sparse=always file file
> Most interestingly, the *backup* file was sparse afterward:
>   $ du file*
>   2688    file
>   256     file~
> That's surprising, and I'd almost consider this a little bug.
> But from the implementation of cp(1), that behavior is quite clear.
> 
> Apart from my tries above: do we already have a possibility in coreutils
> to turn a file to a sparse file in-place?

nothing calls FALLOC_FL_PUNCH_HOLE currently

> [1] http://article.gmane.org/gmane.linux.utilities.util-linux-ng/8415

I responded to this about the possibility of adding to dd (or fallocate(1))
rather than adding a new utility to do this.

> [2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/dd.html

thanks!
Pádraig.