Segfault on mmx instruction due to wrong alignment

[predaeus]

libNewton.so segfaults on writing to memory with

Code: Select all

0f 28 44 24 40    movaps 0x40(%esp), %xmm0


some time after a call to raycast().

The offset 0x40 added to the the content of %esp is not aligned at the 16 byte boundary when the segfault happens, it is aligned at an 8 byte boundary instead. movaps requires a 16 byte alignment.

If I add some unrelated lines of code to my application after that call, it does not segfault. The difference in code likely changes the location of %esp and magically aligns the memory address at the 16 byte boundary, therefore preventing the segfault.

I am running on an AMD64 here with Linux 32bit using the latest (as of current) Newton 2.0.

Please look into this issue if it is a bug or inform me of mistakes on my side.

[Julio Jerez]

can you post a demo that reproduce the crash, I can debug it now using the .so libraries?

[predaeus]

The minimal test case I could come up with is too large to be attached here. I've sent it to you Julio (the email on the title page) with more information about the problem and how I did debug it.

[Julio Jerez]

send me a PM

[predaeus]

Sorry, I can't figure out how to attach it to a PM instead, so I just uploaded it to a file host:
http://rapidshare.com/files/191702954/cerberon.rar.html

It should run if you unrar the file and keep the .dae in the same directory. It basically loads the Collada file and uses that for the collision mesh. Then performs a simple raycast and somewhere after this is when the segfault happens. I removed all the OpenGL etc. code. So it would just print some debug messages and then quit if it would succeed.

I just could not reproduce the problem when commenting the collada loading part and just providing a dummy mesh inside the application.
And my assembly skills are too bad to manipulate the ESP register without breaking other things.

Unfortunately you can't debug this with gdc. I've used zero for debugging in Ubuntu:
http://www.zero-bugs.com/2.0/free_download.html

More specifically I did it like this:

1) $ export LD_LIBRARY_PATH=~/projects/Cerberon/lib (the .so is there)
2) $ zero ./cerberon
3) click CONTINUE
4) segfault occurs and you can click OK to close the message dialog
5) it jumps to the assembly location 'movaps 0x40(%esp), %xmm0'

If you want to re-run the application you can

6) click PROGRAM->RESTART

If you can't get it to run, the relevant assembly parts are here
it segfaults at the first movaps call (b6d15074 in this example):

Code: Select all

b6d15052:    ff 51 20     call 0x20(%ecx) <dgBody::RayCast+0x627>
b6d15055:    d9 9c 24 58 03 00 00     fstp 0x358(%esp)
b6d1505c:    d9 84 24 94 03 00 00     fld 0x394(%esp)
b6d15063:    d9 84 24 58 03 00 00     fld 0x358(%esp)
b6d1506a:    db f1     fcomi %st1, %st0
b6d1506c:    0f 83 3b fa ff ff     jae 0xb6d14aad
b6d15072:    dd d9     fstp %st1
b6d15074:    0f 28 44 24 40     movaps 0x40(%esp), %xmm0
b6d15079:    0f 29 84 24 b0 00 00 00     movaps %xmm0, 0xb0(%esp)
b6d15081:    0f 28 8c 24 b0 00 00 00     movaps 0xb0(%esp), %xmm1
b6d15089:    0f 28 c1     movaps %xmm1, %xmm0


When the segfault occurs, ESP + 0x40 is 8 byte aligned and not 16 byte
aligned.

You can also pipe the libNewton.so assembly into file libNewton.s by doing

7) objdump -d libNewton.so > libNewton.s

And then look at the content of libNewton.s.

[Julio Jerez]

Is this linux 64?

[predaeus]

Linux 32bit, compiled and tested on Ubuntu 8.10.