[Octave-bug-tracker] [bug #61565] nchoosek broken for integer inputs

[Arun Giridhar]

Follow-up Comment #14, bug #61565 (project octave):

Oh wow. I was not expecting them to saturate at intmax without a warning.

For integer inputs, should we also saturate at intmax of the same class as the
inputs, or should we return a doublle that is at least in the correct order of
magnitude? I think the second is more useful and much less surprising to the
user, but would like feedback on whether it would break Matlab compatibility.

Thinking of something like this: calculate C = prod (numer, "native"), and if
it saturates at the corresponding intmax or exceeds flintmax("single"),
depending on the class of input, then we can recalculate it as a double and
return it with a warning that it is beyond intmax or the single-precision
flintmax. If the inputs are double already, we retain the current check for
the double-precision flintmax. Thoughts on this?

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?61565>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/