Re: Why the memory usage of sort does not seem to increase as the input file size increases?

[Pádraig Brady]

On 05/26/2014 04:38 PM, Peng Yu wrote:
> Hi,
> 
> I tried "sort" on some large file. But the memory usage of "sort" does
> not seem to be large. This seems to be strange to me, as I think that
> sort need to see all the data before completing the sorting process.
> Shouldn't the memory usage of "sort" increase as the input size
> increases? Thanks.

Sort takes a divide and conquer approach,
by sorting parts of the input to temporary files,
and then merging the results with a bounded amount of memory.

sort currently defaults to using a large memory buffer
to minimize overhead associated with writing and reading
temp files, so you may be seeing just this large memory
allocation each time.

The memory allocation can be controlled with --buffer-size

thanks,
Pádraig.