Development of organic leg for self balancing characters

[Julio Jerez]

I think get is pretty good now.

in file ...\applications\ndSandbox\demos\ndBipedTest_2.cpp

at the top there is a define
#define _TEST_ONE_FUTURE_STE_ACTION

that shows how the AI controller start training.
if you run the test, I will populate the replay buffer with random action, there you can see that since the action are has equal change to go forward or back. the max angel is never reached.

but the model is not balance, and simple topple to the floor.
This part is how the training start and how the action are generated.

them after the replay buffer has enough actions, what is does is that is try to get actions form the replay buffer, more than randomly.

so when happen is that the reward function, is the sum of the action to the future that the player survive the longer. since short action kill the model very soon, the reply buffer will have more longer sequences of good action than short sequences so it will optimize those.

this part is no in yet, but to emulate it if to uncomment the define. the controller will simple select the action with the large reward, and what is does is that it make the model oscillate around the equilibrium. each time making the angle grows larger.
this is a very remarkable result.
if we where using control theory, there are equation than can be use to determine is the system in stable. but for almost 20 years I tried that, and even when getting a model to balance, it is so complex that the slide change cause lot of code rewrite, compels logic and after a while is difficult to undertenant.

what the ML approach is expected to do, is that if should figure out that the action with the best reward is not always the nest one, it is possible to take few step in the wrong direction and them after some time in the futures make a turn this is what the Bellman equation, which is at the root of all machine learning, shows.
https://en.wikipedia.org/wiki/Bellman_equation

basically the neural net s a way to encode the bellman equation so that given a present state, the machine select there state with the the maximum expected reward.

an example would be say the model is really close to balance, the com is align with the zmp.
but to do that, in the previous step has to give a push to the hip. so now the hip has a large velocity.
if we evaluate the present state, the action with the large reward will be to move in the opposite direction, but of form previous experience that neural net now know that f is does that the model will over shout. so the evaluation form the net will come op with the value do nothing as the one with the the highest expected reward.
it is very difficult to do that with control logic.

anyway if you play with and without the define you can see the difference. but the train model only search one step, it is just a prove of concept for continuing to the rest of the QDN algorithm.

[JoeJ]

Just tried it. But don't expect i could make much sense out of it.
I might dare to look what's input and output of mysterious ML... maybe...

I see it's now doing some real motions, instead instantly changing the pose.
I now remember with some very early approach of mine i had such instant changes too.
Curiously this was very good at balancing on the moving platforms.
I was never able achieve this robust balancing later again, using better controllers causing smooth motion.
Not sure if i understand the true reasons. But i guess it's cheating using superman powers.

Currently working on those contact problems i have often mentioned.
It's not a failure of the simulation. The foot just tilts so slightly you could not see it visually, but the contacts move and the support polygon made from contacts becomes smaller, until it may be just a line eventually, if the model pushes all force against the edge of the foot.
That's all expected, and should be no problem at all. But small disturbances as the foot may get full contact later can be enough to make it fall. (I'm doing all this still with 2 bodies IP model, swinging left and right all the time).

It remains a difficult problem. Just snapping the contacts to the foot corners is no proper soultion and would cause side effects. I can not measure if the foot is titled. To do this i'd need to trace rays from all foot corners, which i do not want. So the contact support poly is all i have.
Now i could slow down if the com goes out of that poly, which should work to bring the foot back to full contact. But this would throttle two legged walking, because here's it's normal the com is out if that poly while one foot is in air.

I hope for you your AI is clever enough so you don't have to worry about such details

[Julio Jerez]

Just tried it. But don't expect i could make much sense out of it.
I might dare to look what's input and output of mysterious ML... maybe...

I see it's now doing some real motions, instead instantly changing the pose.
I now remember with some very early approach of mine i had such instant changes too.
Curiously this was very good at balancing on the moving platforms.
I was never able achieve this robust balancing later again, using better controllers causing smooth motion.
Not sure if i understand the true reasons. But i guess it's cheating using superman powers.

Currently working on those contact problems i have often mentioned.
It's not a failure of the simulation. The foot just tilts so slightly you could not see it visually, but the contacts move and the support polygon made from contacts becomes

I heard the pain. That's why I am moving away from the contacts, and joint torque and that sort if thing.

And yes understanding ml is difficult at first, and the internet is full of misinformation. Every body is an expert, and they just repeat the stuff some one else did.
I actually bought several books, that I have being reading for the best part of four month.
From the fundamentals to the more advanced book by Richrad Sutton.
At the end of tge day. A neural net is a flexible regression model, that can fix a hyper curve to a set of samples.

In that regard. The demo that trin the model will be complex, full of heuristic.
But once the traing happens, if sussefull.
What we get is a very simple model.
Basically a humanoid, with a brain that will take a measure of it state, (velocityy of com, foot in contact, player input, etc) feed it to the brain, and the brain will produce that action to take.
It will simply do that, because in training it saw, few states, that are close to the state it is, and what the brain dos is that it generates an interpolated action.

So in this case, if the model learn to Balance it body give few poses, the the run time, as long as the state are close to what it nows it should balance.

Anyway, I am getting ready to continue this weekend, and see what I get.

[Julio Jerez]

Regarding those angle issues, this sounds like you should use twist / swing decomposition instead.
....
Recently, when reworking the IK solver, i noticed for the first time the method is not really perfect either. The 90 degrees problem still exists, because if we use two angles to define a twist limit, the size of the limit narrows or widens a lot near 90 degrees of swing. Combined with simulation, such issues always remain a source of error, potentially adding energy to the system. But luckily we don't need so wide swing angles to model ragdolls, so the problem remains well acceptable in practice. Still the best method i know.
3D rotations remain a never ending source of trouble. :

on this Joe, I feel you have not given proper credit to the forward dynamics.
I actually set up a test so that you can see the functionality and how it is expected to act geometrically.

few days' back, when I read your pose, it game me something to think about,
and I when back to see if what you say has some concrete basics.
I can't really blame anyone because of the confusion, because even I had a hard time understanding it.
A problem that I saw is that when I made the IK joints, I did not follow any conversion to define the joint frames. since this is physics and it is far, far more powerful that a simple IK system.
Basically, I did not relate the position and the rotation matrices, that game a lot of flexibility but make far to use and prone to failure.
so I when back and adopted he same conversion that 3dsMax, and I believe all animation packages use to define bones.

first let us define what a Bone is.
A bone is a transformation matrix that has some limitations.
in the picture below, shows how a bone is define in max.

Untitled.png (9.14 KiB) Viewed 17205 times

here, you can see that the offset is limited to a vector in which the y and z componets are zero.
the is

posit = vector (d, 0, 0)

but doing this, if we apply that vector to a three Eulers rotations: Pitch, Yaw, Roll.
the pitch angle does does not affect the offset vector.
so if we apply the offset vector to the other two rotation in any order.
we get a vector of the for.

p [cos(yaw)cos(roll) cos(yaw)sin(roll) -sin(yaw)]

with this vector, you can see that only the given a vector position on the space of the chain root.
along with a swivel angle, it is possible to extract yaw and roll angle by doing

d = magnitude(p)
p = normalize(p)
pitch = swivel
yaw = -asin(p.z)
roll = atan2 (p.y, p.x)

I committed a test that show how those for parameter control an IK joint.
pitch is the swivel angle, and there the tree are x, y, z

If you sync, you can see it working, and hopefully makes more sense.

and as I said before, by limiting the matrix to certain values, meaning the offset y and z are zero, you lose dome freedom mainly when the yaw angle become +-90 degree,
but it is my judgement that that the benefit more than outwait that short coming.

[JoeJ]

Will look at the joint demo later to see if i have any complaint.

However, the problem i expected from your sayings is this, referring to your picture:
Imagine we bend the bone to the green y axis, and observe how it behaves on changing the pose.
Then do the same with the blue z.
If we get any difference in behavior, and i think this was the problem with your old cone limit joints, that's bad. And that's what quaternion twist / swing decomposition avoids gracefully.

With quat decomposition we have no 3 euler angles, we have just two. One around the red x (always in its current direction) for the twist, and the second from the default x direction to the current, rotated x direction (swing - and it does not matter in which direction we do this swing).
As a result, there are no poles or singularities where behavior changes (in usually unintended ways).
And there is no dependency on order of angles, except those two rotations which naturally express our expectations on skeleton joints.

But the advantage is not about a more intuitive model, it is about resulting better behavior in both physics and IK. Ofc. a tool like 3DsMax always needs to collapse any model down to 3 euler angles, so the user can edit numbers and their animation curves. The 'evil' euler angles are necessary for user interfaces, but internally we are free to use a better model.
For this reason i won't be able to say anything from moving euler sliders in your demo i guess. But i'll look at the code as well a bit...

[Julio Jerez]

umm, I think there is a confusion here.

the only joint that uses the cone limit is what is newton is the ndSpherical.
this use to be called ndBallAndSocket.

and does no uses euler angle, the matrix is factored into the swing angle that apply to one frame, and the quaternion angle that apply to the parent frame.

the quaternion angle does use a small angle approximation when the two axes are too close aligned.
this s because the cross product of two nearly aligned axis loses precision.


on the picture, the bone is defined alone the x axes only, this is the conversion use by all animations system that I know: 3dsmax, maya, blender.
the bone is defined to be use with IK joints. not for the other joints.

so it will have some rotation restriction, but as I said before, this restriction are all present in motion capture, or in hand rigged key frame animations is the animator decide to rig the skeleton with IK chain.

some IK solvers in those packages like Spline IK, CCD, and few others,
do not have rotation problems, but still use the same conversion,
but those solvers are not really suitable for physics for the reason that those solve are all based on over determined chains, they find a solution out of the infinity possible by some heuristic.

physics does the same, but the physics find the solution that conserve momentum, and those are unusually not the same.
for example, if you have an IK and you move one effector joint free. it will try to go to the specifies location. if you do the same with inverse dynamic, the joint will try to reach the limit but in doing that the root parent alone with any other body linked outside and inside the change will be affected.

that's the reason which the IK joint cannot by use in close loop. Instead it has to be use with the IK solver to calculate the body acceleration and use that as the joint motors.

but anyway, if you have time just try the demo, apply the parameter, you will see that is reproduce what an animator will see if he did that kind of rigging on 3dsMax or Blender.

[JoeJ]

I think our methods are equivalent, but i looked at the wrong place:

Code: Select all

void ndJointSpherical::SubmitAngularAxis(const ndMatrix& matrix0, const ndMatrix& matrix1, ndConstraintDescritor& desc)
{
//...

      const ndMatrix pitchMatrix(matrix1 * coneRotation * matrix0.Inverse());
      const ndFloat32 pitchAngle = -ndAtan2(pitchMatrix[1][2], pitchMatrix[1][1]);

      {
         ndQuaternion q0 = m_body0->GetRotation();
         ndQuaternion q1 = m_body1->GetRotation();
         ndFloat32 twistAngle = 2.0f * atan (
            ( ( ( (q0[3] * q1[0]) + (-q0[0] * q1[3]) ) + (-q0[1] * q1[2]) ) - (-q0[2] * q1[1]) ) /
            ( ( ( (q0[3] * q1[3]) - (-q0[0] * q1[0]) ) - (-q0[1] * q1[1]) ) - (-q0[2] * q1[2]) ) );

         ndAssert(fabs(pitchAngle - twistAngle) < 0.0001f); // see if we get the same angle


         // I do this:
         // ndVector avPin = matrix0.m_front + matrix1.m_front;
         // avPin *= (1.f / ndSqrt(avPin.DotProduct(avPin).GetScalar())); // bad Julio has no Normalize() :)
         // SubmitTwistAngle(avPin, twistAngle, desc);
      }

      SubmitTwistAngle(matrix0.m_front, pitchAngle, desc);
   }
}


[JoeJ]

Using the current demo, the limits are not called.
Probably because you disable limits with IK on, like i do as well?
And i don't know where you calculate the acceleration from IK, so idk where to look at.

No complaints from playing with the demo. My input is limited, but probably within the IK solver.

Oh, and i just realize: Above i have forgotten to apply the body orientation to joint orientation offsets to q0 and q1.

But i think it's the same results anyway.
If so, the only difference is the pin direction, which i set as the average of both bodies.
This works a bit better for me, but i'm not really sure about it.

[Julio Jerez]

ok, I think we agree, but just to clarify one last two point.

on this.

Using the current demo, the limits are not called.
Probably because you disable limits with IK on, like i do as well?
And i don't know where you calculate the acceleration from IK, so idk where to look at.

That's one of the benefits we get from the invers dynamics joint, by connection two or more joint to an effector joint, that will calculate the joint motors, you do not have to worry about limits, motors, friction.
They are all calculated parametrically form the IK equation I post above.
yes you lose some freedom, but the gains are far better.

this functionality is unique to newton and I started in 3.14.
It was not very developed back them, since there was not feedback between collision and solver phase. Now in 4.xx those problem are solved, and it is very practical.

if you look at the demo, at line 511 is where it calls the inverse dynamic solver to get those motors.
then, it places the motors accelerations on the linked joints just as if you were doing the motors per joint yourself. the code looks like this.

Code: Select all

m_invDynamicsSolver.SolverBegin(skeleton, &m_effectorsJoints[0], m_effectorsJoints.GetCount(), world, timestep);
m_invDynamicsSolver.Solve();
m_invDynamicsSolver.SolverEnd();


Oh, and i just realize: Above i have forgotten to apply the body orientation to joint orientation offsets to q0 and q1.

But i think it's the same results anyway.
If so, the only difference is the pin direction, which i set as the average of both bodies.
This works a bit better for me, but i'm not really sure about it.

the flexibility of the engine now that is cpp, is that you can just subclass your own joint from ndJointSpherical and simply overload that function, and it will behave the way you want just on that part.

[JoeJ]

I have concluded i can not solve the contact problem using the simple inverted pendulum model.
I think two feet are needed, and maybe that's also a reason to have two bodies per foot.
With such foot, we can lift the ankle and still maintain some toes contact while the weight is on the other foot. This way we can manage good contact independent from the movement of the upper body parts.

Thus i can't improve the IP controller any further now, and moving to the full ragdoll is the next step for me.
Will try the new features then and give feedback...

But first i have to get back to my terrain stuff.
I wanted to do both in parallel, but did not work. I ended up working just on physics alone.
So it will still take some time...

[JoeJ]

Ok, so now i have my ragdoll working in a basic way. I'm still far from the state of the old project, but i just thought about a very good question regarding adopting your IK stuff:

Let's think about the one armed robot you made, mounted to a static base.
And there is just one effector to control the gripper.

The user sets the effector to the target position, e.g. above some object to grab.

Then you know the target position, but what do you do about the unknown velocity at this position?

I can imagine your effector is a virtual body, so the user can set it's position AND it's velocity. So the target velocity isn't unknown, but the user is responsible to set it. (Mostly, like in the example, target velocity will be zero.)

Or, is the target velocity something like a variable to be optimized from your IK solver? And the user does not need to care? If so, what's your objective? Minimizing velocity? Or acceleration?

[Julio Jerez]

The effector itself is a super critical spring damper
PD.

I super critical spring damper, is a very nice way to control, strength and velocity at the same time.

The speed of the damper does in fact had to terminal speed
That is a function of both the spring const and the damper constant.

To get an idea of the math, check it out in Wikipedia.
The equation will provide a theoretical background for how it works, in practice it is difficult to set it precisely because the parameter involved are complex and variable at each position. So it is better to set by so trial tunning.

It is very quick and responsive, so it is not too had to set it after few test.

[JoeJ]

Makes sense. Thanks!

[Julio Jerez]

this is not carved on stone yet.
there are some benefit but that are some cons to using critically and under critically damp effector and motors.

here is a graphic of the output of a pd motor.

Untitled1.png (51.13 KiB) Viewed 14998 times

it show the tree type of outcome.
the first one is for when the damp is too small, or zero. obviously that one is no no, unless you are seeking a springing behaviors, which some time the case for some applications.

the secund and third are what we seek by adjusting the damp value, as you can see the graph is
position(angle) vs time.

which mean the time derivative will be the velocity.
as you can see when the target position changes, it takes some time for the effector to react and reach the target.
the time derivation is almost like a straight line, which mean is also move at a constant velocity for most of the trajectory, and as it get close is slower down the velocity.
This is alsmost like a magical behavior.

the const is that because of that behavior, it never quit reach the target.
another const is that the gap between the target and the effect is a function of the low.
the reason is that in the equation

f = k * s - C * v

K is a function of the Mass of the system which include the load and change base on the position,
but the is overcome but making the spring very relatively strong.

for newton 4 I decided to go with the kind of motor and no which the actuation which will have a graph like the read plot on the graphs below.

Untitled.png (54.96 KiB) Viewed 14998 times

you have seen those motors in newton 3.xx

here the effect do react the target, but there is that discontinuity when between switching form constant velocity and stop.

since you followed the engine development, you are familial width struggled to tune that over the years.
I found that for a very simple system with one or two motors, is no difficult to set, but as you start to put many of them, it becomes harder and harder to calibrate and you see those twitches that happens each time a discontinuity happens.

the point is that we can do both, but at this time, I only doing the PD method, and if I find that it will be need than maybe we can add it.

but, in the engine any application can make custom effector but subclassing from the joint and override the behavior.

[JoeJ]

There is an alternative method which i really like: Driving towards the target at given min / max acceleration.
I will add it to my IK solver soon, and i expect it produces nice natural motion between poses.

But i already use the method for my picking joint, and it gives very responsive and precise control without oscillation. Instead using mouse pick, i get a gizmo on click and then i can translate and rotate the gizmo, just like in a 3D modeling tool. And the bodies also feel like working with such tool. I'm very happy with it. No bouncing or oscillations, and lag is very subtle.

Idk, how's the method usually called, but it's nothing new and works like this:
Input is current pos and vel, target pos and vel, and maximum / minimum acceleration the motor can produce.
Output then is either min or max acceleration, depending on if we still accelerate or already have to break towards the target. Plus the time those accel. / deccel. phases will take until we flip or hit the target.
The movement we get is thus driven from constant accelerations, like a ball in free fall, which is why i say that's 'natural' motion.

In practice i need to switch to another method like damping if the time to target is smaller than few frames, e.g. 3. That's because simulation at discrete timesteps can't follow the analytical model precisely, obviously.

Because you get this time to target, the method is probably very useful in case we drive towards a distant target, e.g. some complex pose to grab some object to pick it up.
I'll also use given time to make sure all bones of a limb reach the target at the same time. E.g. it looks artificial if the hand is already rotated to the target after short time and then stops, but the ellbow is still rotating for a while.

If we only care about control form one frame to the next, the method probably isn't useful so much.
And it does not help with questions like 'what's my target velocity', plus we need to tweak those constant acceleration values as well.
Still worth to mention.

Btw, regarding parameter tweaking, there is some tedious research for us to do.
Besides figuring out natural joint limits, we'll also have to define max torque values.
I think 6 values for spherical and 2 for hinges should be good enough.
But i guess we won't find data and have to improvise...
Eventually it would be smart to do this early, because parameters as discussed here could be derived form that using just few global constant factors, i would hope.