I have already done it in the case of equality constraint matrices. Because
Octave gave an error message that said something.
In the other case I didn't know why this error appeared. Just tried to give
some info to whoever does the optim package.
since I don't know anything specific about the desired operation of the lsqlin function, I'll just mention that there are a lot of things we think Matlab should do. That said, m-code compatibility is one of the primary objectives of Octave, so the primary task should probably be to remove the error and ensure the NAN output, as you said. Octave does have code and other options that extend functionality beyond what Matlab does. it's possible this could be one of those cases, but only if mathematically justified within the function.