Re: dd: copy blocks in reverse order

[Ivan Pozdeev]

> On 06/12/15 18:32, Ivan Pozdeev wrote:
>>> On 06/12/15 15:20, Ivan Pozdeev wrote:
>>>> Hello Coreutils,
>>>>
>>>> The subj is a requirement if I'm copying between overlapping regions, and 
>>>> the
>>>> destination is further in the media than the source.
>>>>
>>>> The specific task I'm having is to move a partition a number of sectors 
>>>> forward.
>>>>
>>>> To be completely clear, the algorithm is: copy a block, then seek to the
>>>> previous block (i.e. 2*bs bytes back).
>>>>
>>>> I don't currently see a use case for setting this for input/output 
>>>> independently.
>>>>
>>>> An idea for the option name is `d'.
>>>>
>>>> There is a workaround for my particular case - `dd if=<block_device> 
>>>> bs=<2*shift_bytes> | dd
>>>> of=<block_device> seek=<shift_blocks>' but it requires a few times 
>>>> 2*shift_bytes of memory.
>> 
>>> I see dd_rescue (and ddrescue) have "operate in reverse" options.
>>> The reason stated being:
>> 
>>> "If you have one spot of bad sectors within the partition,
>>> it might be a good idea, to approach this spot from both sides."
>> 
>> I already looked into dd_rescue. It can only copy a full device,
>> its purpose being data rescue rather than tinkering.
>> 
>>> For your use case you could compute the size and offset of the overlap,
>>> and use 2 dd invocations to copy the overlap area, then the rest.
>> 
>> Not applicable. The overlap is >1/2, so its destination is also within the
>> overlapped area.
>> 
>>> Or perhaps more simply use a loop to iterate in reverse
>>> using an appropriately large blocksize to minimize dd invocations.
>> 
>> Looked into this, too. 78000000+ invocations in my case.
>> I don't even want to think of how long this would take.
>> 
>>> So while this is useful it is a bit of an edge case
>>> and there are alternatives using other tools or multiple dd invocations.
>> 
>> dd's sole job is to "take blocks from here, put them into there".
>> Unless there's another tool whose sole job is to shuffle the input,
>> the order of taking/putting seems to be an intrinsic trait of that job.
>> Though I agree that it's the minority of cases where this trait matters.
>> 
>> There's yet another argument:
>> As you outlined, currently one has to resort to different methods for 
>> different
>> data sizes/overlap percentages, which are far from being obvious (e.g. I came
>> with my stated workaround only after having composed the letter). Basically,
>> what I'm suggesting is the "one obvious way" ((c)Python Zen) - a method whose
>> chief advantage is being immediately apparent (as well as being highly
>> applicable and readable in code).
>> 
>>> I'd be 60:40 against adding it.

> Good arguments thanks.  Worth considering.

> Another option for this case where the read and write offsets are close,
> would be to read into a circular buffer, and write while maintaining
> a minimum number of blocks (the overlap) in the buffer.

> We previously talked about adding such a util to coreutils:
> http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html

> A simple untested patch to debian's buffer(1) attached might achieve this
> in a more optimum fashion by supporting read and write sizes > overlap size.
> That would be used something like:

> dd bs=... if=... | buffer -s 1m -P 50 | dd seek=... of=...

> cheers,
> Pádraig.

My aforementioned workaround didn't work. The 2nd `dd' still overwrites
some sectors before the 1st one reads them.

The line with `buffer' just above (the System Rescue CD that I'm using has
`mbuffer' instead) didn't work, either: -P delays writes initially but doesn't
protect from buffer underruns.

You didn't specify which specific source your patch is against so I couldn't 
test it.
Yet, due to the previous paragraph, I suspect it won't help, either.

Even `parted' and `gparted' are no good! The first one doesn't support moving, 
the
2nd one has 1M granilarity and doesn't show/allow to set actual sector numbers.

-- 
Best regards,
 Ivan                            mailto:address@hidden