Re: sort: memory exhausted with 50GB file

[Paul Eggert]

Leo Butler <address@hidden> writes:

> I don't know if this is relevant, but I have extracted the 2nd through 1000th 
> character in the 50GB file, and there appears to be garbage (unprintable 
> chars) 
> in the first line. The remainder of the extract looks fine. Moreover, I split 
> the file into 500MB chunks, sorted these and then merge sorted the pairs. It 
> appears that the 500MB chunks produced by split have been stripped of '\n' 
> and 
> are garbage, as are the sorted files.

Hmm, it sounds like your input data has some very long lines, then.
That would explain at least part of your problem, then.  'sort' needs
to keep at least two lines in main memory to compare them: if single
input lines are many gigabytes long, then 'sort' must consume many
gigabytes of memory, regardless of what parameter you specify with '-S'.