bug#49300: Report 3 UBSan integer related errors found by an automatic tool

[Paul Eggert]

On 6/30/21 10:07 AM, He Jingxuan wrote:

> bug1: basenc
> command: cat basenc1.stdin | basenec --z -
> relevant error message: ../src/basenc.c:635:25: runtime error: left shift of 
> 128 by 24 places cannot be represented in type ‘int'
>
> bug2: basenc
> command: cat basenc2.stdin | basenc --z - -d
> relevant error message: ../src/basenc.c:770:18: runtime error: signed integer 
> overflow: 41760500 * 85 cannot be represented in type ‘int'

These two bugs were fixed in coreutils 8.32; please try that instead.

> bug3: seq
> command: seq 3 1 1.
> relevant error message: ../src/seq.c:185:21: runtime error: unsigned integer 
> overflow: 2 + 18446744073709551615 cannot be represented in type 'unsigned long'

This diagnostic is a false alarm. It is complaining about this statement:

          ret.width += (fraction_len == 0
                        ? -1
                        : (decimal_point == arg
                           || ! ISDIGIT (decimal_point[-1])));

where ret.width (of type size_t) is 2. But (size_t) 2 + (int) -1 has 
well-defined behavior in C: in the normal case where size_t does not 
promote to int, it is equivalent to (size_t) 2 + (size_t) -1, and this 
must equal 1. And in the hypothetical case where size_t promotes to int, 
it is equivalent to (int) 2 + (int) -1 which also must equal 1. So there 
is no error here.

That being said, coreutils uses size_t too often; it should use idx_t 
(from Gnulib) to avoid common mistakes when using unsigned values. I'll 
add that to my list of things to do.