Segmentation fault on x86_64 linux

[bel]

Dear Ghostscript developers !

It seems I found a bug in Ghostscript compiled for Linux x86_64 platform.
It reveals on several most popular Linux Distributions, both with GNU and
with ESP Ghostscript. I also reported this bug to ESP Ghostscript as STR 
#1540.


Short Description
-----------------

CUPS test page beeing processed with cms color correction tables,
resolution 150 dpi and above and device=bitcmyk causes segmentation fault
on x86_64 Linux.

Commands caused segmentation fault:

cat full_file.ps | gs -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin
cat small_file.ps | gs -sOutputFile=- -sDEVICE=bitcmyk -r1200x600 > res.bin

Files full_file.ps and small_file.ps are attached to this letter.


Detailed Description
-----------------
Bug was noticed on the following x86_64 Linux distributions:
- SuSE Linux 9.3 (ESP Ghostscript 8.15.0)
- SuSE Linux 10.0 (ESP Ghostscript 8.15.0)
- Mandriva Linux Limited Edition 2005 (ESP Ghostscript 8.15.1)
- Mandriva Linux 2006.0 (ESP Ghostscript 8.15.1)
- Fedora Core release 3 (GNU Ghostscript 7.07)
- Fedora Core release 4 (GNU Ghostscript 7.07)
- Fedora Core release 5 (ESP Ghostscript 8.15.1)
- Ubuntu 5.10 (ESP Ghostscript 7.07.1)

The file full_file.ps is the standard CUPS test page with embedded
color correction tables, these tables cause ghostscript to segfault.

The file small_file.ps is attempt to reduce full_file.ps to minimal
size. This file causes segfault on SuSE 9.3 / x86_64 distribution.

Futher investigation on Mandriva 2005 LE with ESP Ghostscript 8.15.1 
compiled
from source showed the segfault occured somewhere near set_color_ht_le_4
function located in src/gxcht.c file. The py parameter passed to this 
function
sometimes have negative value. Simple check like the following

if(py < 0) py = 0;

prevents Ghostscript from the segmentation fault. But this is a workaround,
not the solution.

I should also note that this problem was never seen on 32-bit Linux (i386)
distributions.


Best Regards,

Konstantin Belyanski,

address@hidden