Newton crashing

Report any bugs here and we'll post fixes

Moderators: Sascha Willems, Thomas

Newton crashing

Postby Bird » Wed Oct 11, 2017 5:39 pm

Hi Julio,

I've been playing around with the latest github version and keep getting crashing in the app I'm working on. Here's a video:
https://youtu.be/jZymxHB49FA

All the dynamic bodies in the scene are Convex Hull collision shapes. The crash doesn't happen as much when there's lower body counts.

My initiialization code looks like this

Code: Select all
   NewtonWorldSetUserData(world, this);
   NewtonSetSolverModel(world, 1);
   NewtonInvalidateCache(world);
   NewtonSetThreadsCount(world, 1);
   NewtonSelectBroadphaseAlgorithm(world, 0);
   NewtonSetPerformanceClock(world, dGetTimeInMicrosenconds);
   NewtonSetNumberOfSubsteps(world, 2);


And the stack dump is
---First crash stacktrace:
******* STACKDUMP *******
stack dump [0] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 501 dgWorldDynamicUpdate::SortClusters
stack dump [1] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicssimplesolver.cpp L: 40 dgWorldDynamicUpdate::ResolveClusterForces
stack dump [2] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 638 dgWorldDynamicUpdate::CalculateClusterReactionForcesKernel
stack dump [3] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 139 dgWorldDynamicUpdate::UpdateDynamics
stack dump [4] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 925 dgWorld::StepDynamics
stack dump [5] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 996 dgWorld::RunStep
stack dump [6] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgmutexthread.cpp L: 63 dgMutexThread::Execute
stack dump [7] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgthread.cpp L: 169 dgThread::dgThreadSystemCallback
stack dump [8] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 247 std::_LaunchPad<std::unique_ptr<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64>,std::default_delete<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64> > > >::_Run
stack dump [9] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 210 std::_Pad::_Call_func
stack dump [10] d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp L: 115 thread_start<unsigned int (__cdecl*)(void * __ptr64)>
stack dump [11] BaseThreadInitThunk
stack dump [12] RtlUserThreadStart

I know it's a long shot but do you have any idea what could be causing this.?

-Bird
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Wed Oct 11, 2017 6:00 pm

It look like is happen in function dgWorldDynamicUpdate::SortClusters
how frequent does I happens?

I am doin a refactoring of function because a change I made a while back is no worth keeping since
is too complex and in general is makin the solve slower.
I will do doin that this weekend. but I have to compare to several versions back.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Wed Oct 11, 2017 8:56 pm

It look like is happen in function dgWorldDynamicUpdate::SortClusters
how frequent does I happens?


It's pretty random but definitely more frequent as more bodies are added to the scene.
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Thu Oct 12, 2017 3:26 pm

Yes I would assume it would be very randome,
since I am planning to remove that algorithm this weekend, let us no worry about and just do the change and test again.
hopefully I can post here gain monday so that you can try again
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Julio Jerez » Mon Oct 16, 2017 9:52 am

I believe this is a memory problem, that was added to 3.15 some time in the last few month.
I think I can fix it, but it will be a waste of time.

I start to reintegration 3.13 to 3.14 starting from when I started making the large changes.
there is a complication because I changed the directory structure and rename some files, so is difficult to compare.
I made a version of 3.13 with the same directory structure and renamed the files to match 3.15 to make comparisons easier.

One think I noticed is that is not the only problem, 3.15 has gotten much slower too.
I already merged the major part starting from low to high level.
so far I merged the low level system, math library, containers the low level collision and things like and things lake that

So far there are no major differences between 3.13 and newer 3.13a, but there is still big different with 3.14
Now I need to merge broad phase and solver, I suspect the problem in the solver, but merge from top down is much harder.
anyway this will take another weekend.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Mon Oct 16, 2017 9:03 pm

Ok, thanks for the update. Good luck. :)
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Mon Oct 23, 2017 2:11 pm

ok I found some subtle bug that were cause undeniable effect in the solver and I fix them, It turned out this bug came what I change the operator overloaded in the math library.

one case was in function
void dgWorldDynamicUpdate::IntegrateVelocity() const
where I have a code that did a software round to zero, of the velocity when the it stat to become a denormal value, I theory this is support in hardware in some cpu, but not all, so I did in code using logical operation. when I change the math library thet function gto screwed up and when bodies gone to rest that start getting a lithe jolt, the reason was that in 3.14 I made the zero velocity a small number (1e-4 unit /second) instead of (1.e-18 as is was in 3.13)
this cause that critical island never rest cause great performance slow down.

the sucond bug was a mermory over run in this function, that migh be what cause yor crash/
void dgWorldDynamicUpdate::SortClusters(const dgBodyCluster* const cluster, dgFloat32 timestep, dgInt32 threadID) const

I am still done the comparison, because I also found some redundancies like two variable that hold the body acceleration and that can cause bat side effect.

anyway please sync and let me know if still happens.
if is happen do no panic because I have no made the changed that I said I was going to do,
I first want to find these discrepancies with 3.13 before moving forward.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Mon Oct 23, 2017 7:33 pm

Hi Julio,

Looks like you made some good progress! This version is way more stable for me. It took a very long time to crash and this crash was not in dgWorldDynamicUpdate::SortClusters like the other crashes

Here's the stacktrace

---First crash stacktrace:
******* STACKDUMP *******
stack dump [0] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 624 dgContactSolver::CalculateIntersectingPlane
stack dump [1] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 827 dgContactSolver::CalculateClosestPoints
stack dump [2] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 1333 dgContactSolver::CalculateConvexCastContacts
stack dump [3] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1443 dgWorld::CalculateConvexToConvexContacts
stack dump [4] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcollisionconvexpolygon.cpp L: 620 dgCollisionConvexPolygon::CalculateContactToConvexHullContinue
stack dump [5] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1842 dgWorld::CalculateConvexToNonConvexContactsContinue
stack dump [6] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1549 dgWorld::CalculateConvexToNonConvexContacts
stack dump [7] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1079 dgWorld::CalculateContacts
stack dump [8] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1198 dgWorld::CollideContinue
stack dump [9] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 413 dgWorldDynamicUpdate::SpanningTree
stack dump [10] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 199 dgWorldDynamicUpdate::BuildClusters
stack dump [11] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 90 dgWorldDynamicUpdate::UpdateDynamics
stack dump [12] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 916 dgWorld::StepDynamics
stack dump [13] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 987 dgWorld::RunStep
stack dump [14] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgmutexthread.cpp L: 63 dgMutexThread::Execute
stack dump [15] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgthread.cpp L: 169 dgThread::dgThreadSystemCallback
stack dump [16] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 247 std::_LaunchPad<std::unique_ptr<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64>,std::default_delete<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64> > > >::_Run
stack dump [17] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 210 std::_Pad::_Call_func
stack dump [18] d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp L: 115 thread_start<unsigned int (__cdecl*)(void * __ptr64)>
stack dump [19] BaseThreadInitThunk
stack dump [20] RtlUserThreadStart

---End of first stacktrace
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Tue Oct 24, 2017 3:11 pm

umm a crash here should never ever happens.
stack dump [0] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 624 dgContactSolver::CalculateIntersectingPlane

are your shape very oblong on he extreme?
I can no imaging the code crashing there under any circumstance, I do not think I have ever seen a crash on the function, some else must have gone wrong.
can you make another video that show the collision shapes.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Wed Oct 25, 2017 2:39 pm

https://www.youtube.com/watch?v=Xn4k0dTdNkA&feature=youtu.be

Ok, here's the same group of complex meshes but rendered using the convex hull geometry from the NewtonCollision shape. The video ends as the crash happens. Looks like Newton is crashing in the same spot. Crashing does not happen very often when there's fewer bodies in the scene

dgphysics\dgcontactsolver.cpp L: 624 dgContactSolver::CalculateIntersectingPlane

******* STACKDUMP *******
stack dump [0] c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional L: 212 std::_Func_impl<<lambda_1a43963cd9b0b9024ac23dcf90cf0ba8>,std::allocator<int>,void>::_Do_call
stack dump [1] c:\hurleyworks\code5\projects\products\dreamer\modules\g3logger\core\g3log.cpp L: 181 g3::internal::saveMessage
stack dump [2] c:\hurleyworks\code5\projects\products\dreamer\modules\g3logger\core\logcapture.cpp L: 32 LogCapture::~LogCapture
stack dump [3] c:\hurleyworks\code5\projects\products\dreamer\modules\g3logger\core\crashhandler_windows.cpp L: 111 `anonymous namespace'::exceptionHandling
stack dump [4] c:\hurleyworks\code5\projects\products\dreamer\modules\g3logger\core\crashhandler_windows.cpp L: 134 `anonymous namespace'::vectorExceptionHandling
stack dump [5] RtlNormalizeString
stack dump [6] RtlLookupFunctionEntry
stack dump [7] KiUserExceptionDispatcher
stack dump [8] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 624 dgContactSolver::CalculateIntersectingPlane
stack dump [9] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 827 dgContactSolver::CalculateClosestPoints
stack dump [10] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgcontactsolver.cpp L: 1333 dgContactSolver::CalculateConvexCastContacts
stack dump [11] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1443 dgWorld::CalculateConvexToConvexContacts
stack dump [12] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1075 dgWorld::CalculateContacts
stack dump [13] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgnarrowphasecollision.cpp L: 1198 dgWorld::CollideContinue
stack dump [14] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 413 dgWorldDynamicUpdate::SpanningTree
stack dump [15] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 199 dgWorldDynamicUpdate::BuildClusters
stack dump [16] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworlddynamicupdate.cpp L: 90 dgWorldDynamicUpdate::UpdateDynamics
stack dump [17] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 916 dgWorld::StepDynamics
stack dump [18] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgphysics\dgworld.cpp L: 987 dgWorld::RunStep
stack dump [19] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgmutexthread.cpp L: 63 dgMutexThread::Execute
stack dump [20] e:\code5\opensource\frameworks\newton-dynamics\sdk\dgcore\dgthread.cpp L: 169 dgThread::dgThreadSystemCallback
stack dump [21] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 247 std::_LaunchPad<std::unique_ptr<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64>,std::default_delete<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),dgThread * __ptr64> > > >::_Run
stack dump [22] c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread L: 210 std::_Pad::_Call_func
stack dump [23] d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp L: 115 thread_start<unsigned int (__cdecl*)(void * __ptr64)>
stack dump [24] BaseThreadInitThunk
stack dump [25] RtlUserThreadStart
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Thu Oct 26, 2017 2:07 pm

ok this is a second time and is start to happen after I committee the fix for the first bug.
I think I know what happens.
neither trace stack is not saying that this is a memory violation, and the was my fear, because the will mean that there is a bug of that function which has been working for almost 15 years without ever crashing.

what I belie is happing is that the fix that I made did sole the problem of the stack size if the function the sorted the joint, however is barely made it, is just push the bug to the next function call.

I am talking about the fix that change function alloca
Code: Select all
//#define DG_ALLOCA_SIZE (sizeof (dgVector))
//#define dgAlloca(type, size) (type*) alloca ((size) * sizeof (type))
#define dgAlloca(type, size) (type*) ((dgVector*) alloca (size * sizeof (type) + 256))



the other function was allocation a smaller stack size (four time smaller), but because the allocation is conservative, the padding was enough to mask the memory overrun.
your scene is big enough to make the padding insignificant, so that expose the crash.
fixing the bug mean the stack is much bigger so it is possible a caller function will run out of space.

this was not problem before because when I was using thread, the stack size was set a thread creation time, but c++ 11 std thread does no allow to set the stack size of a thread.
what this mean is that is time to abandon alloca, and add a stack container to the world

I will made that change this weekend, meantime could you make me a favor,

in file: ...\newton-dynamics\sdk\dgCore\dgTypes.h
line 172 //#define DG_USE_THREAD_EMULATION
uncomment out that define, so that the game run on the main thread.

this will force the engine to run of the stack size of the main thread which I suspect is much larger (1 meg ) while child threads are 250k)
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Thu Oct 26, 2017 4:17 pm

in file: ...\newton-dynamics\sdk\dgCore\dgTypes.h
line 172 //#define DG_USE_THREAD_EMULATION
uncomment out that define, so that the game run on the main thread.

this will force the engine to run of the stack size of the main thread which I suspect is much larger (1 meg ) while child threads are 250k)


My app design has Newton as a separate module running on it's own thread so the calling thread to Newton is not the process main thread. I gave it a try anyway and still got the crash in the same place after a while.
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Thu Oct 26, 2017 4:54 pm

Oh I see that change will not do anything them.
I will make the stack allocator change this weekend and hopefully this will take care of that bug.

It it about time we get rid of alloca anyway, that function can be debugged it, and is some case it is in fact a slow function because it always prove the stack.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Newton crashing

Postby Bird » Wed Jan 03, 2018 10:23 am

Hi Julio,

I just tried the latest build and no more crashing!
Bird
 
Posts: 623
Joined: Tue Nov 22, 2011 1:27 am

Re: Newton crashing

Postby Julio Jerez » Wed Jan 03, 2018 10:50 am

ok I forget about that, yes I found the bug that was causing it and fixed.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles


Return to Bugs and Fixes

Who is online

Users browsing this forum: No registered users and 3 guests