Development of organic leg for self balancing characters

[Julio Jerez]

I am opening this thread for discussion of the the self balancing character.
the other thread was too old, plus I learned a few things.

the first one is that I underestimated the scope of the project by a great deal in each of my previous tries, so now I start with smaller scope approach.

I am still in the camp of the people that thinks the locomotion is a physics more a physics and control problem than it is an intelligence control problem, and I believe companies like Boston dynamics proven that. Boston dynamics robots are good control with good animatronic and very litle AI.

what It is different now for me, is that it is not an all or nothing problem, there most be some intelligent control, backed with strong physic and analytical control.

I have seen many videos and papers, and I see that many people think the other way, they think it is just an AI problem, in fact I'd seen people that think neutral net make up for the physics, that, I totally reject. Here is an example https://www.youtube.com/watch?v=JgvyzIkgxF0&t=826s

I will go later over the new changes, But in a nut shelve, what I have learned, is that a good physics solver combined with a robust controller is not enough to balance a contraction more complex than a box on a single or double pendulum.
The main reason is that a linear controller is based of a minimization of a quadratic function. Basically you build an optimization function, and you take the step that in the negative direction of the gradient of the function.
the problem is that with an articulated character made of a complex arrangement of bodies, joint and an environment, the energy function to optimized is not convex and has many local minimal.

so calculation the gradient and moving alone the negative of that gradient at some point, may move the entire character to a worse situation.
So to solve this, I am thinking to use a re-informent learning neutral net that calculate a trajectory.
basically calculate the state of the Character an arbitrary number of step in the future.

we can trained to say see if the energy goes down say 8, 12, 16 steps ahead.
the number can be tuned. but by doing this we can see if the character is in a better state has he continue doing that.

of course move the player ahead is not an options, but that what's where reinformeten learning with policies gradient algorithm are very good at.

at typical example with be something like:
say we have a walk animation, and we train a neural net to learn the animation.

at the end what we get is a DNN (deep neutral net) that give the inputs (joint angles) output the same joint angle.
That does not seen to do much, but now consider that we are playin this character on the same flat terrain that was use for the animation.
since the character only know how to walk, the result will be that he will paly the walk animation in a more expensive way.

but now consider that the terrain has some small perturbations, and we use a Collison system to detect the the feet locations and how far there are form the location in the animation.

animation system does this with Bad IK and blending. in the pass I had done this with Gaussian processes, basically this is find the close pose assuming each joint position is a gaussian distribution with mean centered at the walk cycles. the result are never satisfactory because gaussian interpolattion tend to smooth high frequencies. but that a different story.

in the new3 system we feet the pose at position t of the animation, but now the feet are offset by a small amount. and we read the results.

the new set of angle can be good or bad, but that whet Training can feet the problems, because by repeating the operation is possible to adjust the network weight so that the result get as best as it could. basically the net becomes a non lineal blender.

I have seen few paper that are doing similar thing, all claiming there are doing it better that the other, that, but for me is just an example.

[Julio Jerez]

The first step is that I was wrong thinking that a singular 6dok effector was enought to cover all cases.

I was very happy seen how good is was for mechanical robot with 6 dof arms, and a gripper.

But when trying g to use it with organic limb, I start to do what I always hated. Which add options and parameters to customize the joint.

Them I end up adding hidden links an bodies just to satisfy the 6dof requirements for the joint, when it should be the other way.
The joint has to satisfy the requirement of the articulation.

Two examples are the dog (spot) and the new dAiBot_1.
The leg only has 4 degrees of freedom, 3 translation and a swivel angle.

For a human the leg are identical but a human also has a foot. So I erroneously thought that the 6 for joint was the answer, but now I see that in reality the feet of a humanoid move independent of the leg, so in fact the leg is four dof.

This is not a solution to all problem, but is a very big relation, so I will convert the model to use this new joint.

Another type of joint is the five dof. This is good for quadruped and biped that wlaj on thier toe.
Basically their limbs has three link instead of two.
In that sence the iabot and dog are incorrect, but I will let it go like that as a learning process.
Them we make the 5dof joint.

I now added a new basic model, that I am going to experiment with. See how this approach goes.

[JoeJ]

This is good for quadruped and biped that wlaj on thier toe.
Basically their limbs has three link instead of two.

Actually all mammals have the same skeletal structure, including humans. (afaik)
Just humans have their ankle / heel mainly resting on ground.
But they also lift the heel up for the stance leg, when the leg is behind the body, shortly before they lift the whole foot up.
In this situation the equal skeletal structure with animals shows most obviously.

There are many simulations which use only one body for the foot, so toes and heel are rigid. I also did this, and interestingly it works pretty well.
So maybe we could do it, and it gives us as some optimization.

But i've actually changed to using two bodies for the foot. One reason is i can use the same IK solvers for humans and animals. I also assume for really natural results, two bodies are needed, and with more detailed models the difference will show up.

I have not tried any simulations yet. It surely is harder to solve, because now the mass of the toe body is even smaller, and we have one more joint to lift all the weight. But i guess the accurate joints will deal easily with this.

I have finished the work on full body IK, so i could start to simulate pretty easily from now on
But i'll still postpone this some time.
Actually i work on muscle simulations, just for visuals, to model the human body anatomically correctly. That's basically a childhood dream of mine, since i've failed to achieve this with drawing. I gave up my plans on a career as comics artist back then, because i've felt i'd never learn this.

The game industry is at the same state: No matter how many extra bones they use, it's impossible to get natural anatomy. They can scan Hollywood actors, but they can not display them without looking like abominations of nature, as soon as they bend an arm.

This needs an urgent fix, and i think it's possible at a very low cost.
So currently i look at anatomy programs to see where muscles and bones are and what they do.
Already learned a lot, e.g. the distance of the hip joints is actually larger than i have used in my older ragdoll models.
Maybe, after setting all this up, i may even memorize so i could eventually get my drawing skills right too... but actually i didn't draw anything for decades

[JoeJ]

two bodies are needed, and with more detailed models the difference will show up.

I remember i had an idea to avoid related visual problems. If we want to use just one body for the whole foot, i think we can. But we might need a second, more detailed skeleton for visuals, and some simple IK algorithms to calculate it's pose, modeling the heel lift.
I remember Havok had such skeleton mapping as a built in feature to drive detailed visual skeleton from more basic ragdolls.

We might think two skeletons are no longer needed now, but that's not really true. We still need at least non physical extra bones to model hands and fingers, jaw, eye orientations, etc.

All this keeps the single body foot an attractive idea, and for a start, it's always easier to work with simpler models.

What would you think about simulating hands and fingers too?
I would just assume that's unpractical / not worth it. But probably it becomes an option to progress further, once we have the basics working and adopted.

[Julio Jerez]

yes, I know most vertebrates have same morphology, but they use it differently.

if you look at horse, dog, elephants, the five bone that make the metatarsal toe bone are fused and use as part of the limb. So the move as part of the articulation of the leg limb
for primates those bones are the foot plant, and they move independent of the limb.

what I am saying is that, for the limb IK, I am not using just one joint for control the tip of the toe, as I though before, instead I will use one the control the tip of the Calf or the tip of the pre-toe bone.

this will produce a joint that has only four degrees of freedom, (three translation and a swivel angle)
I now have a sample, but still need a tweak for the elbow, joint to prevent form flipping.

that has always been a challenge, but I believe now that newton 4 can deal with more than 6 dof bilateral joints, my solution is to not to add a hard limit for the angle violation, but instead a strong spring damper penalty.

also for now, I am only focus on that humanoid type of limb (leg, calf)
if you look at the Boston spot model, they do not have foot articulation

[JoeJ]

what I am saying is that, for the limb IK, I am not using just one joint for control the tip of the toe, as I though before, instead I will use one the control the tip of the Calf or the tip of the pre-toe bone.

Ah, ok. I think i do something similar. My limb IK solver actually takes a target ankle / wrist joint position, plus (optionally) the target orientation of the hand / foot.

So if i set up a footstep for example, first i calculate the pose of only the two foot bodies at this place, including the heel roll if needed. This gives me target for the limb solver.

The rest of the limb is similar for arms and legs, so i use the same code for both. (Arms have one more body because of flexible shoulders vs. rigid pelvis.)

It's interesting to notice that those remaining few limb bones (upper and lower leg or arm) matter the most regarding the general IK problem.
To set up a footstep, or to grab an object, the foot and hand bodies are pretty trivial to calculate.
And the spine does not move so much in general.
If we have all this, calculating upper / lower limb bodies may seem trivial too at first.
But it's not, because of the open variable for the swivel angle.

Previously i've left the swivel angle to be set by the user.
But actually i control it by a set of weighted secondary objectives, e.g. maximize agility (avoid joint limits), minimize energy (hang the hinge downwards), or a given target position for the hinge itself (useful to implement crawling).

I can see now this idea gives me a very nice rig to design poses manually. It's much better than the IK Rig of Softimage XSI, which was a major inspiration for my initial ragdoll work.
Animators would like this.

[Julio Jerez]

what I am trying to say is that for Organic limb, I am removing the IK control out form the toe, and feet
I think that's a better way to control a rig than using one effector that control every thing

I am using only on effector limb, but it is no IK, IT is a close loop inverse dynamics.
the physics solve the whole this at once.

here is a picture of the two type of limb configuration

Untitled.png (3.94 KiB) Viewed 12388 times

the first one in the simpler, and can be use for most humanoid rigs.
here is has a
3 dof ball socket.
1 dof hinge
1 effector that control 4 dof.

them for a human for example we can add a feet by that loint can be its own OK, in the case of multiple toes of fingers, or a simple box or combination of spheres control by a double hinge PD joint.


on the right, we have a more complex limb because is has 5 dof.
3 dof ball socket.
1 dof hinge 1
1 dof hinge 2
1 effector that control 4 dof.

so the effector can't controlled, but
for creators like insects, dogs, dinosaurs, horses, etc.
the Metatarsus bone is in general control procedurally,
in other word, there is a script that control the second hinge angle,
for example for an insect the last joint is aligned to the up vector of the body insect.
or for a dino, it will be aligned to the same vector but at an offset angle.
That made the limb 4 dof.

will work on that problem until I get a decent model.

I made a demo that simple an simple Test AI Bot.
I will try to add a controller for it.
This model removes the problem of center of mass, and all that stuff so it will allow for getting knowledge of how to write the controller.

I am thing to use Newton Net training. they is a popular method called Policy Gradient Reinforcement Reward. I am reading about it so let us see. how that goes.


My claim is that a model like that is far simpler that the model people are using which is just a articulation of open loop connected joints. so the NN spend most the time learning how to calculate the motors torque or acceleration to simple no fall.

I do no think nature works that way, to me the locomotion part work in automatics pilot, and the brain only interact to provide goals of when something goes wrong like losing the balance.

I am try to mimic that but making the model a close loop. where all the limb are control but positioning the effector. that way the Neural Net will only has to focus on getting the motion

I am hypothesizing that that model should reduce the amount of training and the complexity of the nn

if you sync you will see the model,

is has 16 control knobs, 4 for each legs. each control is a variable form -1 to 1

so the task of a neutral net is to just figure out how to moves those knobs for say walking straight or follow a spline path.

so in essence the problem can be stated, give the model, and a spline path

the net, will be trained to maximizes the math the speed of the path. by only manipulation those 16 knobs.

[Julio Jerez]

I now added a normalized parameter mapper so that the robot control are knobs with range from -1.0 to 1.0

The cool part is that is very easy to set so that the movement is more or inside joint limits.
But that is still difficult, because the effector can some time moves outsize the work space of the ik chain.

The joint still has its limits because as the body move, the inertia makes the PDs some time violates the limits a little, but that's is what make it look so organic.

if you move the dog model, after a while, looks scary alive. and it is not really doing anything special.

I am very excited about these results using this joint that I will convert the other biped models, which are broken right now, then I will start adding a new module called ndDeepBrain

DeepBrain will be a deep neural network library to suit the needs of Newton. I'd seen PyTorch and Tensor flow, but that stuff is too huge, and is has all the bad red flags.
python, work on the cloud, difficult to get it to work in windows and too much of a black box for my test. so after looking around I decided that I will add a custom implementation in pure cpp, and will use CPU muticore and simd.

anyway, this will start after I get all the articulated model converted to this new system.

here is an example of what I am hopeful to get using deep learning with these robots.
https://www.youtube.com/watch?v=mji2ezCGS1c

in that demo, you see that the dog, it is two d model, so in that sense, it is easier, however the net has a harder time because the robot is an open loop, so the net has to make up for the physics calculation of joint motors. while our dog, this is done by the physic engine so the net will only make up for the kinematic.
To me, this has to translate for a much simpler neural net, that can converge to a optimal solution faster.

one of the dirty little secrets of the deep learning every one sweeps under the rug, is that the evaluation is quite expensive, those nets have thousands of neurons and spend ours, and our if not days training. but anyway, let us see what we get.

[JoeJ]

ndDeepBrain

Reading this just gave the impression that i'm really old and outdated.

still need to compile and look at the demo...

[Julio Jerez]

I am now approaching the more challenging part which the rig for a humanoid.
I am going to start easy. I have this model, that I found in one of those free 3d model site.

Untitled.png (77.38 KiB) Viewed 12362 times

it is not very good, I can't believe how hard is to find any decent models, most has licenses for using them, so much for open source.

anyway, I added this partial skeleton made manually with the bone tool.

opencl.png (41.09 KiB) Viewed 12362 times

it only has three chains, the hip spine, and the two legs.

these legs are an example of what I mentioned before that the foot are no part of the chain, but that still and bodies connected to the body by joints.

[JoeJ]

I (barely) remember this free tool: http://www.makehumancommunity.org/
They show no images on the site, but iirc it's an application similar to RPG character editors, and then you can export mesh and skeleton.

Still some work. Just capsules always worked well for me.
I prefer it even, because looking at the failures of skinning is more a distraction than a helpful visualization.

My Windows licence runs out in 4 days, i'll wait until the reinstall before i install Cmake and try the demo.
So you have some time left to implement backflips and a couple dancing rock'n'roll ;D

[Julio Jerez]

well I will be happy if I can get an ai controller that balance the humanoid.
I have been trying that for year, and each time come short.

I'd seeing that the policy gradient reward algorithm seems to be a successful and controlling a humanoid much better than a quadratic controller, so I will give a try.

I am now placing the model in the scene, and so far, this is how it looks like.

Untitled.png (140.73 KiB) Viewed 12341 times

there you can see the tow challenging parts.
1-the knee is almost straight, that has for the ik, because it has to operate on very close to the workspace of the chain.
2-the foot are now independent. so they need a procedural forward IK that orient the foot base of the position.

both those cases are taking away from the controller. as far as the controller know each leg has
four knobs that control: leg position (x, y, z) and swivel.

for this model instead of cartesian coordinate it will be polar coordinate,
the effector will control yaw, roll and leg length.

but anyway, let us see how far we can get this time.

[JoeJ]

Just tried to open CMake project directly in the newest Visual Studio, without installing and using CMake manually. Would be nice.
But failed on that. I guess it would work if i knew how, but i ended up installing CMake and using that.

I see a 4 legged spider and i can move the body using the sliders. Behaves nice and naturally.
Even if i pick it up with mouse and try to stress it, it behaves all well.

So far so good I don't have the most recent version, but i guess you don't have AI working yet.

It turned out drawing my muscles and skeleton bones is no fun, so I started a new side project where i can do some coding while waiting on my terrain simulations. I have not worked on any realtime stuff for years, and will begin with learning Newton 4. Probably i'll start work on walking ragdoll earlier than expected. Maybe we can have some competition of machine learning vs. old school human engineering...

[Julio Jerez]

few things.
cmake: I do not use the visual studio integrations. in the pass I always had bad experiences with those kind integration, I once tried visual studio generated make files, in the hope I could use for cross platform, and that had never worked, because what visual studio does is that it makes a make file that simply call a visual studio project file, and that only work in VS ide. so I do not take that chance anymore. I find the CMake GUI quite easy to use and so far, it works on all OS that I tested.

on the articulation.
I am breaking in three stages.
1-make the environment
2-Test them with some procedural controllers
3-try to make AI controller.

make the environments is just make few scenarios with different type of robots. I want them to be reliable and as accurate and realistic as possible. so far I have four, each with different configurations. there are three quadruped and one biped (the biped is still in the works)
the quadruped has different numbers of DOF, surprisingly enough, the Boston Dynamics Spot model is the simplest of all.
If you sync?, you will get that model and you can try it out for reliability and robustness.
I believe it has all the degrees of freedom of the real Boston Dynamics model.

Test environment, for this Dave Gravel and I are collaboration in making a simple controller that can move spot the dog around using procedural generated walk cycle and blends.
he has already start and you can see his results.
https://www.youtube.com/watch?v=7AnQGqOlKVA&t=10s

for me the purpose of this is to have a way to generate what in AI parlance is called the ground truth,
basically, what the AI use to compare results. I am not sold yet on the Reinforming learning what makes the AI learn to move around without guidance. So I will use ground truth in the form of animation to provide style.

the last part is to write the neutral network solver so that I can make smart controller.
that is a longer-term goal, since I have almost nothing and I do not plan on jumping the van wagon of PyTourch, Tensor Flow, and the others, I find them too cumbersome for what I am seeking.

on the competition, I rather have some cooperation. this is why I am setting the framework of these environments. but so far I have very little luck on anyone using, so my progress is slow.

[Lax]

Wow, I watched the video. The results are fantastic! Now just a jump function and backflip are missing