[Octave-bug-tracker] [bug #61472] AddressSanitizer crash in MEX tests

[John W. Eaton]

Follow-up Comment #27, bug #61472 (project octave):

After looking at this a little more, I'm not sure what the best solution is.

We only mix malloc with delete in the case of returning array data from MEX
functions.  Those arrays are either double/float or pairs of double/float used
to represent complex numbers (and then, we only mix malloc/delete when using
the new interleaved complex storage layout that is enabled when using the
-R2018a option for compiling MEX files).

With double data, there are no destructors to call.  With complex, the
destructors don't actually do anything.  I know it works for double arrays
because I've tested that.  But it fails for complex.  Did I not test that?!? 
I guess not.  I'm attaching a simple mex file to demonstrate.  Using stable
Octave without my proposed changes, try


octave> mkoctfile -mex -R2018a freetest.c
octave> x = freetest ();
octave> clear x  ## EXPECT error and crash


Note that it should work if you don't use interleaved complex data (omit the
-R2018a option or use -R2017b instead).

I'm also attaching a simpler stand-alone program that demonstrates the
problem.  When I run it, I see:


$ g++ -O2 vec-dtor.cc
$ ./a.out
new + delete[] for double: ok
malloc + delete[] for double: ok
new + delete[] for complex: ok
malloc + delete for complex: ok
malloc + delete[] for complex: free(): invalid pointer
Aborted


If we could find a simple workaround for the visibility problem, I would have
no objection to using the polymorphic allocator when using a compiler that
supports it.

Otherwise, I think the best solution for version 7 is to only allow malloc and
delete to be mixed when returning real-valued arrays from MEX functions.  For
complex data, we can just go back to copying data even when using interleaved
complex storage.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?61472>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/