Re: sort: memory exhausted with 50GB file

[Paul Eggert]

Leo Butler <address@hidden> writes:

> sort rapidly chews up about 40-50% of total physical memory

That's weird.  It shouldn't do that.  It doesn't do that on my machine
(Debian stable x86, coreutils 6.10, compiled with GCC 4.2.2).  Memory
usage goes up to 250 MB (as requested) and stays there.  'sort'
creates temp files of size 201,061,873 bytes, each containing
2,545,087 lines.  Here's the command I used to try to reproduce the
problem:

  awk 'BEGIN {for (i=0;i<467289720;i++) print "-16 -2 -14 -5 1 1 0 
0.3080808080808057 0 0.1540404040404028 0.3904338415207971"}' |
  sort -S 250M -k 6,6n -k 7,7n -k 8,8n -k 9,9n -k 10,10n -k 11,11n -T /tmp -T 
$HOME/junk -o /tmp/foo

I suppose it could be a locale problem.  What's the output of the
"locale" command?

Can you run 'sort' under GDB, and see what the stack backtrace looks
like when 'sort' fails?

Are you compiling for x86 or x86-64?