weird behavior using different sized time steps

A place to discuss everything related to Newton Dynamics.

Moderators: Sascha Willems, walaber

weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 10:16 am

We have recently integrated newton and our vehicle model so we can do articulated vehicles and will very likely express more of the model in newton primitives next year.

At the moment our setup is fairly simple: newton rigid bodies represent the body of the tractor vehicle and the body of the trailer vehicle. These bodies are joined by a Newton joint. Everything else is still expressed in our model and generates torques and forces on the newton bodies. We call NewtonUpdate at our simulation frequency with the appropriate timestep size and via a PreUpdate listener our model is updated.

If we run a simulation at 60Hz (note that this is a very very low frequency) everything seems to be fine and the vehicle seems to drive as usual with or without a trailer hitched. However, if we run the simulation at 500Hz or 1000Hz, the behavior changes: the drive wheels will spin up but the body will only move very very slowly. At lower frequencies the effect is present as well, but less pronounced. Or in other words, decreasing the update time also seems to decrease body velocities (?).

Our model takes timestep size as a parameter and has no problem running at higher frequencies and the two main newton functions involved here (NewtonUpdate and the PreUpdate listener) do too. So we're at a loss what is happening here. Is there anything in Newton or in NewtonUpdate that would/could cause this behavior?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby JoeJ » Tue Nov 14, 2017 12:06 pm

I remember something similar and assumed it is due damping, but never checked it.
Until Julio answers you could try this. IIRC there are dafault damping values and to disable you need to give pointers to a zero vector on body creation. There should be linear and angular damping.
User avatar
JoeJ
 
Posts: 1453
Joined: Tue Dec 21, 2010 6:18 pm

Re: weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 12:26 pm

Indeed! we use NewtonCreateDynamicBody and NewtonBodySetMassMatrix when creating the bodies. Neither has a parameter for damping. I never saw NewtonBodySetLinearDamping and NewtonBodySetAngularDamping, so those were both defaults. Setting them to zero seems to completely solve our issue. Thanks! :D

Don't quite understand why this is not time step size independent though.
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Tue Nov 14, 2017 12:33 pm

you simulate a 500 htz or more?
that seems an over kill, It seems that 120, 180 should be reasonable.

but any way like Joe said you will need to disable newton default drag.
Code: Select all
   dVector drag(0.0f);
   NewtonBodySetLinearDamping(m_body, 0);
   NewtonBodySetAngularDamping(m_body, &drag[0]);

I hop you are no sue variable frame rate that will be a problem, iterative numerical algorithm do that cope well with variable frame rate sine the time step becomes a unknown random variable.this goes for both Integration and linear systems.

this can easily be demonstrated using any numerical integration and expanding and writing the taylor expansion, basically
Code: Select all
y' = f(t)
them the expansion is   
y(t + dt) = f(t) + f'(t) * dt + f''(t) * dt^2  + f'''(t) * dt^3 + .....

sat the integration is first order as most numerical integration system are.
y = y0 + f'(t) * dt + f''(t) * e(dt^2)

now when dt is constant the term e(dt^2) is constant, so the integration error due to time is constant
but if dt is variable, then error is self is a function of t that add random noise to the system in the form of random acceleration and velocity.

this is a generic proof, but you can check out more specific derivation in wikipedia,
the Verlet which uses forward and backwrod difference is very popular and eassy to understand.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 2:37 pm

Hi Julio,

no, we use fixed time step size. Currently, in our model/application there are some aspects that benefit from higher rates.

But since you mentioned integration, I've tried to find out more details recently but could not find any definitive information. What kind of integrator/solver is newton using, and how is it influenced by NewtonSetSolverModel / NewtonSetSolverConvergenceQuality?

What is the default for NewtonSetSolverConvergenceQuality? And does it only affect joints? or does it also affect integrator for (unconnected) rigid bodies?

We value accuracy over computational time (the latter you can throw a bigger cpu at :wink:), what would be a practical value for the number of iterations? Can we get metrics on how many iterations were needed?

What doesNewtonSetNumberOfSubsteps, or rather the substeps, do?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Tue Nov 14, 2017 3:36 pm

ok few think.

NewtonSetSolverModel Is the number of iteration that the iterative lcp system uses when calculation contact reaction forces , the defult is 4

NewtonSetSolverConvergenceQuality is now deprecated,
in newton 1.xx I had a closed from direct solver by that was too slow for scene with many objects
this was very good because the complened I has were about performance not about accuracy

in 2.xx I start experimentation with Iterative LCP and the open a can of worm because I start getting complains about performance and accuracy. so to go around that
I start adding functionality to set the converge rate of the clock diagonal solver,
NewtonSetSolverConvergenceQuality and that what that function is.
I kept tweaking until 3.13 but I wasn't ever satisfied with it

I kwon having an iteration solve that at the core also has an iteration solve is what I always complement all these game engine are doing, and althought I get to be very good, I ken it have to go.

so for 3.14 the block diagonal solver is exact, and the high level solve is iterative. and function
NewtonSetSolverConvergenceQuality is now deprecated. This may make large scenes a little slower
but so far I have no seen that problem.

In Any case the last commit has the three solvers, 1.xx, 2.xx and 3.14
there are not the exact versions because I fix bugs alone the way, by the are mathematically equivalent. Is some is different the can be interchange.

The result so far and by far the best quality/performance of newton I have ever seen.

The doe no apply to joint since they are always exact unless the end application set the to be sift joints.
to summarize you do no have to worry about NewtonSetSolverConvergenceQuality converger quality is
in theory machine precision, I said in theory because is I let the solve solver until machine precision error, the is become thousand of time slower because of arithmetic on denormal number is thousand of time slower. so I set the precision to 1.0e-7

What kind of integrator/solver is newton using, and how is it influenced by NewtonSetSolverModel / NewtonSetSolverConvergenceQuality?
Newton use and RK4 with a modification integration method.
The modification is the weigh factor of the derivatives, which I use 1/4, 1/4. 1/4. 1/4
instead of the normal 1/6, 1/6, 1/6, 1.3
The is not a hack the weigh coefficient are in fact arbitrary, the only requirement is that the add to 1.0, the latest are the result for a east square optimization on the coefficient, but the made the evolution far more expensive for very lethality gain in accuracy.
for a free body the integration is a predictor corrector.
these are the reason Newton is slower than the other engine, I too care about accuracy and correctness. Remarkably with all those extra stuff newton is fasten than most main stream game engines.

finally;
We value accuracy over computational time (the latter you can throw a bigger cpu at ), what would be a practical value for the number of iterations? Can we get metrics on how many iterations were needed?


you should be fine with the default value which is 4,

What doesNewtonSetNumberOfSubsteps, or rather the substeps, do?

That what I want to suggest you look at, instead of take control of the number of passes and doing
you can use that function to subsample the time step.
that what I use in the demoes.
I set the update at 60.0f

and I set the number of substep to 2, this esi the equivalent fo
calling Newton Update (1.0 / 120.0) twice

in your case say you want to simulate at 300, on initialization you call
doesNewtonSetNumberOfSubsteps (5)

and in the update loop you call
NewtonUdate (1/60)

you may say is the same, but it really insect because ether are internal states that are preserve across calls. specially when running the engine asynchronous or when doing CCD.

I hope this was clear.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 5:47 pm

thanks for the detailed response!

so, if the solver model number of iterations is about contact reaction forces changing it would not have any effect in simulations not containing collisions? (ie. our current simple situation of having at most two bodies that are jointed)

We're on git commit 776a1464a3512096a7ef734ed7bb709363491e5b, which should be 3.14. What do you mean by that it has all three solvers, are they selectable, or are all three still present in the code, but just one is actually used?

I thought I remembered newton using (a variant of) RK4, but could not find it. Good to know.

Ok, clear about the substeps. And that could be very useful in our case. There are some expensive calculations (such as intersections with terrain) that do not need to run at very high frequencies, while for the model higher frequencies can be beneficial. Once we move more of the model into newton, we could update at two different rates very easily.

I am quite interested in what internal state is preserved. Our application is multithreaded where the simulation runs in a different thread than the visualisation. Therefore we use a rotating state buffer that makes it safe for the simulation to write a new state, while the other threads are reading the previous state. This means that before each newton update in the pre update listener we set the newton state (body position, orientation and velocities) from the buffered model state. However, if there is additional internal state, that state might get lost, or perhaps worse be incorrect.

Note that we don't do joints yet in this way (tractor-trailer combination only functions singlethreaded yet). Haven't had time to really look at it, but I do not see functions for getting/setting joint state at first glance. Is this possible?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Tue Nov 14, 2017 6:46 pm

if the solver model number of iterations is about contact reaction forces changing it would not have any effect in simulations not containing collisions? (ie. our current simple situation of having at most two bodies that are jointed)

This is correct, but is hard to make a sim where there are not contacts,
I guess your car does not have physics wheels so it will be a body that most of the time will be floating.
if that's the case then yes the solver is exact 100% of the time,
this weekend I will add the last solver optimization that will make works in time proportional to the number of linked bodies in the contraction instead of the number of joints, ex an airplane, you type of vehicle with trailers. Many games and simulation do meat that criteria.

We're on git commit 776a1464a3512096a7ef734ed7bb709363491e5b, which should be 3.14. What do you mean by that it has all three solvers, are they selectable, or are all three still present in the code, but just one is actually used?

yes this is correct the three functions are
dgFloat32 CalculateJointForce
dgFloat32 CalculateJointForce_1_50
dgFloat32 CalculateJointForce_3_13

the last name tell you the sdk version when the function was implemented first.
remember these are no exactly identical because as I move the function form another version I have to make the necessary changes to have the function compiled. I also have to fix any bug that I find and I do no make those retroactive.

and yes the are in the code but that can no be selected, I do not like to make stuff that is retroactive until, what is knew is knew. these function at there for comparison. 'you can se break point and see that I fact you vehicle is solve in one pass.

you should at least experiment with latest version so that you do not fall too far behind. keep you correct version in case of any problem.
Physics library only get better when people test then, that are hundred of thing that can go wrong because it difficult to test every thing.
Newton get very little testing, the reason Is relatively stable is that I tend to rely on analytical algorithm rather that Geometrical ones, but even analytical algorithm you have to make compromises and that when the error can creep in.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 6:55 pm

we will probably update soon(ish)

any thoughts on the internal state aside from position, orientation & velocities?

And on getting/setting joint state? (if any)
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Tue Nov 14, 2017 7:58 pm

And on getting/setting joint state? (if any)
I do not know what that mean.

for the other question, at the movement when running with NewtonUpdate there is only one saving but that is going to changes soon.

if you look at function void dgWorld::RunStep ()
you will see that the loop that call to update the transform matrices is no part of the Step Dynamics loop,

the next reused is in function void dgWorldDynamicUpdate::ResolveClusterForces
but that's only for continues collision.
when an island is in CCD the update does not recalculate the new call broadphase, Island, sort new island, or recalculate new contacts unless they are needed at the time of impact, this is a sustancial saving when using CCD. This will be extended for when using substeps.
The only modification for the is that the Island need to be made for the full time step rather he for the sub time step. This isi how CCD work, the broaphase make island for the time step and then subsample at variable time step up to a max of 8 (I believes) so is should also work for sup stepping.
This will do for newton 3.15

The final saving and this is a big one happen when using Newton Asycroun Update.
when using this model the update signal a thread return and return immidialtlly to the caller.

if you are doing many step in a loop, the every update after the very first one will be blocked waiting for the previous update to finish the calculation and will not return back to the system.

say the first update take 3 ms, this mean the loop will take 3 ms * number of steps - 3 ms
so only 3 ms will be concurrent with eh client app.

this is how the newton demos was set up, it was running at 128 heated, but the fps was about 40 or 500 frame depending of the scene complexity.

when setting the engine to do one Update with sub steps, now teh solve will return immidially by the background thread will do in a loop doing and the update do the concurrent time will be 3 * ms

after setting the demo that way the fps is 10 time higher, basically the physics is free, as long as there simulation take less time that the other thread in the client app.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Tue Nov 14, 2017 8:31 pm

right, so most (if not all) other internal state has to do with CCD. That's something that at this time is not an issue for us.

Regarding joint state, the first question is do they have any? ie. like bodies have a position and orientation.

For our rotating state buffer we actually use three instances of the vehicle model class. This class contains instances of component classes, which eventually have a reference to a newton body. To keeps things simple these model instances share nothing, so we have something like:

state buffer 1 -> vehicle model instance 1 -> component instances -> newton body 1 in newton world 1
state buffer 2 -> vehicle model instance 2 -> component instances -> newton body 2 in newton world 2
state buffer 3 -> vehicle model instance 3 -> component instances -> newton body 3 in newton world 3

When rendering a frame one gets locked for reading, when doing a time step one gets locked for writing. Note that the read lock and the write lock will always point to different buffers and that the requests are for readable or writeable buffers. The callee does not know which buffer it gets. It is however guaranteed that a read lock always returns the most recent state.

This means that the state of the most recent time step update (say it's in #2) may not be in the same buffer that we got a write lock on for the next step (say it's #3). In our model this is easily handled by copying a state struct. From this state struct we need to set the current state to newton. For bodies we do this by using NewtonBodySetMatrix, NewtonBodySetVelocity, NewtonBodySetOmega (in that order).

Can we do that for joints as well? Do we need to? If they don't actually have state, there might be no need?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Tue Nov 14, 2017 8:46 pm

like I said the CCD will be use for the sup stepping, but that's some this will not happens in 3.14

states? the engine do not uses states, the body only have one position, rotation, one matrix, etc.
The engine does not uses double buffer in anyway, not sure how that impact your system.

Can we do that for joints as well? Do we need to? If they don't actually have state, there might be no need?


joint work the same way than bodies, they set kinematic relation between bodies and some have parameters that you can use to control them. like motor, spring and so on.

I support that is you were able to use bodies, and collision then you should be able to use joints the same way.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Wed Nov 15, 2017 6:40 am

ok, maybe I am not explaining very clearly

suppose the below scenario:

there are two bodies A (tractor) and B (trailer) joined by a joint (hitch). At time is T the state (position, orientation, velocities) of A is X1 and B is X2.

Now before we advance to time T + s (where s is timestep size) we set a new state (position, orientation, velocities) for bodies A and B, to Y1 and Y2. Where X1 != Y1 and X2 != Y2, they may even be kilometers apart. (Note that we set the body matrix using NewtonBodySetMatrix, not NewtonBodySetMatrixRecursive)

Will the joint after this 'jump' generate correct forces for the new states (position, orientation, velocities) of the bodies (at Y1 and Y2)? Or do joints have internal state, meaning the joint is (partly) still at or using X1 and X2?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Re: weird behavior using different sized time steps

Postby Julio Jerez » Wed Nov 15, 2017 9:19 am

no that will definitely won't work.
change the velocity of position at rune time is a no, no in newton. It will completely malfunction.
The engine is designed so that yo do the physic on each body as if they were individual bodies, and yo apply the external force.
then the solve will calculate the reaction force between the two jointed bodies.

for your Trailer, I believe you just do the physics in the Cap, and the physics Rig, the yo apply the net force to each body, and the joint will calculate the force at the joint.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: weird behavior using different sized time steps

Postby eabbink » Wed Nov 15, 2017 1:52 pm

I don't think I am quite following you..

Note that we are not changing anything in Newton during a NewtonUpdate. Only in between them. So a simplified cycle is:

A. set Newton Bodies to specific state (pos, orient, velocities) in world #2, the state is the result of the last NewtonUpdate done on world #1
B. calculate forces and torque on bodies world #2
C. NewtonUpdate world #2
D. set Newton Bodies to specific state (the result in world #2 of step C) in world #3
E. calculate forces and torque on bodies world #3
F. NewtonUpdate world #3
G. set Newton Bodies to specific state (the result in world #3 of step F) in world #1
H. calculate forces and torque on bodies world #1
F. NewtonUpdate world #1
repeat from A

We do this because we have three newton worlds, each with a set of bodies, for each of our state buffers. And we need to set the latest result from one world into another to advance one update.

For a single body this seems to work fine. (and I don't understand why it would malfunction, unless there is additional state aside from position/orientation/velocities). I have not yet tried to make it work with joints though. So my question basically boils down to:

do joints have state (information other than parameters that are the result of a previous time step and are needed in the current one), or are they stateless (only dependent for the new time step on their own parameters and the state of the connected bodies)?
eabbink
 
Posts: 21
Joined: Wed Jun 21, 2017 6:32 am

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 25 guests

cron