Development of organic leg for self balancing characters

by **Julio Jerez** » Thu Aug 18, 2022 1:36 pm

I now added the stochastic version of the optimizer. and this is how it looks like.

: Untitled.png (24.85 KiB) Viewed 10140 times

it is a little more higher variance. but that is expected, since stochastic simply mean polling the data.
so instead if optimizing the entire batch, it take a mini batch and move down hill to minimize that batch, then next time take another minibatch and do the same.
and as wee see with pool, the trend of the error is to go down, but from mini batch to mini bath, the error can goes up.

the benefic is that this make the optimization about 20 to 30 time faster. so it can compensate with more iterations. this optimization took optimizing Time 368.933615 (sec)
approximated 6.2 minute

while the full batch take over two our for half the iterations.
the cool part is that we support both

this is not small task, the training data, conation 60,000 samples, this is and 6 minute in a single core, is more that impressive, if you consider that the problem using tensor flow or PyTorch take hour.

anyway I am now will add the parallel trainer, which there but not tested, and the the save a load.

by **Julio Jerez** » Thu Aug 18, 2022 2:07 pm

the success rate on the training data is over 99%

that is remarkable result since some digit are so hard to detect that even a human will fail,

now I need to test it on the test dat which is 10000 digit not in the training set, see how that one does.

by **Bird** » Fri Aug 19, 2022 12:20 pm

Hey, that's very good. I occasionally play around with Dlib, which is a very mature machine learning library.

https://github.com/davisking/dlib
http://dlib.net/

Here's the results for the Minst dataset of handwritten digits
training num_right: 59978
training num_wrong: 22
training accuracy: 0.999633
testing num_right: 9919
testing num_wrong: 81
testing accuracy: 0.9919

Training and testing takes amount 1 minute on my machine. But he's using cudnn

by **Julio Jerez** » Fri Aug 19, 2022 1:42 pm

Ah that's a good reference point.

Yes my test case fail in 288 sample from 60000,
But if I let it go, I believe is can reach even better.
I just do not do enought, training passes.

Also I am no applying any pre conditioning of the training Data, many algorithm do normalization also down sample and all kind of tricks.

Here is the difference, my version is strict cpu
Last night I run my test again on my machine and to my suprice it took 99 seconds, I did it twice to make sure I did not commented out anything.

I am now writing the parallel version, in cpu so i expect that on my icore 7700, that time go under half a minute.

And if I get close to that, them I will consider a susscess and move to the implementing of the
Deep policy gradient algorithm for making ai controllers that can make arrangement of bodies and joints do cool stuff.

After I get the parallel solver I will ask you guys to test it, and see what results you.

On the susscess rate, it does not really matter which implementation archive better score, because it depends on the net configuration, layers, neurons per layer, what activation, kind of layers, and so on.

Basically the nimist is a stress test, to check the algorithm correctness, because Ann is a simply concept but the implementation is more complex than people in the internet let other people think.
They are all using pytotch or tensor flow, and there is a lot of work behind the scene.

But in any case, I am more surprised than I expected with these preliminary results, and I believe that also validate my intuition that there is a tremendous amount of misinformation about the Gpu an Ann. And the people who really undertand the problem and confusion, has not interest in clarifying it because it benefit them. Lot of self appointed experts saying gpu is the way to solve machine learning problems, and I am not convinced that is the total truth.

by **Julio Jerez** » Sun Aug 21, 2022 6:02 pm

ok I spend some time adding some of the basic functionality needed for this to be usefull.

so far these are ethe results I am getting:
from the very first vanilla test we got these results.

optimizing Time 619.069258 (sec)
training num_right: 59788
training num_wrong: 282
success rate on training data 99.646667%

testing Time 0.132002 (sec)
test num_right: 9663
test num_wrong: 337
success rate on training data 96.629997%

simply adding more iterations and reducing the learning rate, help a little on the training set, reduces the success rate on the training data, but does not really improve the error on the test data,
and that is bad. here are tow runs

optimizing Time 984.036465 (sec)
training num_right: 59808
training num_wrong: 192
success rate on training data 99.680000%

testing Time 0.139620 (sec)
test num_right: 9669
test num_wrong: 331
success rate on training data 96.690002%

small learn rate, does no help much
optimizing Time 993.155995 (sec)
training num_right: 59808
training num_wrong: 192
success rate on training data 99.680000%

testing Time 0.141015 (sec)
test num_right: 9672
test num_wrong: 328
success rate on training data 96.720001%

what explains this is that since neural net are regression machines, basically the fit a hyper surface over a set of sample data. the more complexity you add to the data the more it has the potential to over fit the data. this is it will be hugely successful predicting at data point form the training, but as soon as is tested with a sample that is even slightly different.
in the test we see that we can easily reach over 99% in training but only about 96.6% on the test.

96.6% seem good but it is actually bad for a model that that is supposed to predict the.
the image bellow explains the problem

: Untitled.png (92.99 KiB) Viewed 10099 times

the red dots, are the training sample, and the black dots are the test sample.
if we use a neural net that is very complex, (lots of layers and neuron) it will have the compacity to adjust the weights so that the error across the training point is very small. but is doing that the curve is very wavy, because it becomes a very high degree polynomium, that will be the blue line.

if on the other hand the network is not complex enough, them it can's adjust the way to meet the training sample, and that will be worse, because it will not be able to model the data.
an example of that would be trying to approximate a parabolic motion

with a straight line it will never work,
with a degree or higher for polynomial the cur will fit the sample, but if there is noise on the data, they it will be very wavy.
the correct model is a degree 2 polynomial.

but the trick is how to determine that degree of the polonomium not knowing the problem. and that is what the neural net is supposed to do.

the way do it is to just add enough complexities, to the net, and have figure out how to make the higher order terms be zero, or near zero.
there are few tricks to solve that.
1- using basics function, in teh case of Neural net woudl be gaussian curves,
2- using regularization.

I will try both methods, net week I will start with regularization
this article explains it: https://en.wikipedia.org/wiki/Regulariz ... athematics)

I will try ridge and maybe lasso which seem easy to add. and depending on how that goes.
them I can try basics functions.

right now the best result I am getting is but just adding stochastic descend with randomize inputs
plus retunring the net with the lower local minimal squares error

// returning the net with lowest mean square error
optimizing Time 3633.200419 (sec)
training num_right: 59822
training num_wrong: 178
success rate on training data 99.703331%

testing Time 0.141333 (sec)
test num_right: 9654
test num_wrong: 346
success rate on training data 96.540001%

// returning the net with lowest mean square error, plos randominzing the training sample each pass
optimizing Time 3097.118460 (sec)
training num_right: 59855
training num_wrong: 145
success rate on training data 99.758331%

testing Time 0.132634 (sec)
test num_right: 9672
test num_wrong: 328
success rate on training data 96.720001%

that 96.72 % has to go up to at least 99.0%

anyway. I thought, I was going to be ready to start trying deep policy gradient, but with a net to make the predictor with such error, will no work, the net has to improve.

by **Julio Jerez** » Fri Aug 26, 2022 10:05 am

after adding regularization options the rate improved a sustantially.

optimizing Time 688.682453 (sec)
training num_right: 59898 out of 60000
training num_wrong: 102 out of 60000
success rate on training data 99.830002%

testing Time 0.116078 (sec)
testing num_right: 9817 out of 10000
testing num_wrong: 183 out of 10000
success rate on test data 98.169998%

with a 98.17 success rate on the training data, is provably good enough that can be trusted to start using it for the Policy gradient
in particular I want to try A2C and A3C
https://paperswithcode.com/method/a3c

and see how that goes, but there are others

by **Julio Jerez** » Fri Aug 26, 2022 7:08 pm

I notice that take a lot of tries tuning hype parameters to get to converge once it reaches that low point. let it go for mini batch that are twice as large. I got to predict the training data to be about 99.8% and that took about 11 minutes with four cores. and that the best I had so far.

training data
num_right: 59903 out of 60000
num_wrong: 97 out of 60000
success rate 99.838333%
time 766.540034 (sec)

test data
num_right: 9821 out of 10000
num_wrong: 179 out of 10000
success rate 98.209999%
time 0.251424 (sec)

98.2% is not a bad result.

I am going to let it run for about 1.5 hours, it yielded this

training data
num_right: 59944 out of 60000
num_wrong: 56 out of 60000
success rate 99.906670%
time 5750.580978 (sec)

test data
num_right: 9825 out of 10000
num_wrong: 175 out of 10000
success rate 98.250000%

the error decrease in training, but not much on test, so probably not worth the extra time. but is good to know that the convergence continues decreasing.

Those results are in the ball part with state of the art of today's implementations of most sophisticated neural nets that preconditioning the data and extract structural info from it.
I say this is ready to test on deep deterministic policy gradients controllers.
we can always go back tweak it is we have to.

one thing I want to add is the momentum base gradient updates. that will help of the hyper parameters tweaking whit is turning out to be annoying.

by **Julio Jerez** » Sun Sep 18, 2022 1:53 pm

hey Dave, Joe and anyone else, check the latest commit.
this is very early stage. but I let this guy express how I feel
https://www.youtube.com/watch?v=XkBPP5c63q0

Using the Q learning principles, I got the model to find the balance at multiple points.
If you sync you will see the model find the balance by moving the com to the zero-moment point. of course, is bends the upper body but that is expected because the brain is not aware of the upper body orientation.
There is still lot of work to do.

1-I need to find a way to reset the body and clear internal joint states, this cause the weird jump each time is complete a trial.
2- find the DQN predictor using the neural net instead of reading the q value from the end. so far, this uses the Bellman equation with a Q value of one and only one step in the future. essentially just take the action with the largest q value, which happens to be the step that moves the com to the zmp.

once the DQN predictor is added, I expect to be able to find the balance and also maintain the spine upright.

but after that, I need to switch to the Policy Gradient methods that work with continue spaces. the DQN only works with discrete spaces so each time you add a news action, with possible options at eact state, the number of outputs increases exponentially.

rigtht now there are only one variable that can take three values: move forward, do nothing or move backward.
so that's 3^1
adding spine orientation will be one more variable and that's
3^2,
them adding the ability to moves in the plane add two more variable variable and that's 3^4, as you can see this does not scale well for large multidimension actions problems. but the reason to implemented it is because it is practical for small scale problems, like some stuff in games, and also because if the bases of more advanced algorithm like DDPGN

by **JoeJ** » Sun Sep 18, 2022 3:23 pm

Haha, congrats! :mrgreen:

In cases it becomes blue, does that mean it has found a solution?
Or is it just pausing and preparing for the next attempt?

This is my first contact with ML. I do not dare to look at the code... :lol:

by **Dave Gravel** » Sun Sep 18, 2022 3:38 pm

Really nice.
It look very good and it is funny to watch.
It look to behave good without the sleep mode too.
It is a bit less jumping without the sleep mode.

Nice work

by **Julio Jerez** » Sun Sep 18, 2022 7:19 pm

ah very nice to test without out sleep.

it actually shows the limitation of DQN, since it can only handle discrete action. after is fond a balance,
the slight offset from a balance point makes it take one step and overshot on the other direction,

then is change the direction but now it fand a balance father away with the body bended.
It is kind of funny, to see that is does find a trajectory of balance points.

that is in fact a very good find, which explains that balancing a biped humanoid is not that difficult of a problem. It is a very good surprise, because I was expecting a balance on a very narrow space around the support point.

next week I will spend more time completing this, some bug to fix and see if we can a more interesting control before moving to continuum action space.

by **Julio Jerez** » Sun Sep 18, 2022 7:25 pm

JoeJ wrote:In cases it becomes blue, does that mean it has found a solution?
Or is it just pausing and preparing for the next attempt?

yes, when becomes blue is found a point for the CG that is so close to the zmp projection, that the solver next step produces an acceleration so small than fall below the sleep threshold, and since newton 4 do not have frame count for sleeping, object goes to sleep in one tick.
it is quite remarkable, that manage to bring it to sleep. even when the correction pushed are fix.

discrete actions control is not going to cut in the end, but as prove of concept it is a good learning exercise.

by **Julio Jerez** » Mon Sep 19, 2022 1:56 pm

ok, I now fixed some of the problem with the model.
The first one was that the IK, do have some gimbal lock limitation. in the image below.

: 20220801_140603.png (319.69 KiB) Viewed 9868 times

you can see that as the controller tries to shift the com to math the zmp,
it does that by moving the leg forward, but in doing that, the torso bend backwards.

if this was FK, that would no be a problem, by using IK, teh angles are extracted form the position which is a vector of teh form

p = cos(alpha) cos(beta) sin(beta) sin(alpha) cos(beta)

here
x = cos(alpha) cos(beta);
y = sin(beta)
z = sin(alpha) cos(beta)

when beta is close to 90 degrees sin of beta is zero which is fine, but as beta become negative

them cos (-beta) = cos(beta) so the x and z position are the same,
but since the leg is not allowed to bend backward, them is rotates 180 degree.

numerically this is right by geometrically is not, and cause the solve generate huge joint forces

in the image you can see the the legs start to flip backward as the torso and the leg form and pass the 90 degree angle limit.
after adding the condition of terminate state when the angle reach +-80 degrees, them the mode just restart, but now the repro is complete deterministic.
the IK shortcoming is one of the price with most paid for the flexibility.

anyway, as you get more and more acquittance with the way a neural net word, you find out that adding control login is more general and simple.

what I realized that last night, I solve by adding state logic.
but today, I just saw that all I had to do is to incorporate that in the function that value the state. but checking that is the angle is too big, them the stare is final, making is a bad trajectory.

I can see how neural net can fix problem even with bad physic. they can exploit they weakens by going around them rather that going analytically finding the bug and try to fix it.

by **Julio Jerez** » Mon Sep 19, 2022 4:02 pm

with the last weak, now consider the com speed and velocity.
when now can see that the controller really try to balance the model by going is both directions.
and not just moving in one direction until it dies.

This is because it only can predict one step in the future. but since it can take discreet steps. is does no know how to predict two or more steps away.
that's what the DQN will do.

but there will be not logic, the neural net will just take the speed and relative position as input and magically will say do this.
I think I now have the model where I want.

by **JoeJ** » Mon Sep 19, 2022 4:43 pm

Regarding those angle issues, this sounds like you should use twist / swing decomposition instead.
I use it for ragdoll joints, and it's stable and behaving well even if the pins go close to 180 degrees apart. Also using this for IK.
We talked about this years ago already. I noticed your cone limit joints had some expected issues due to their geometrical method, similar to gimbal lock. Using the quaternion twist instead resolved those issues for me. You could look up my old joint which was part of Newton demos for some time.
Some years back the topic came up at gamedev.net, and Dirk Gregorius descriped what he is doing the same way. (An old friend of yours, afaik ;D ) Actually i guess pretty everybody is doing it this way.

Recently, when reworking the IK solver, i noticed for the first time the method is not really perfect either. The 90 degrees problem still exists, because if we use two angles to define a twist limit, the size of the limit narrows or widens a lot near 90 degrees of swing. Combined with simulation, such issues always remain a source of error, potentially adding energy to the system. But luckily we don't need so wide swing angles to model ragdolls, so the problem remains well acceptable in practice. Still the best method i know.

3D rotations remain a never ending source of trouble. :/

Development of organic leg for self balancing characters

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self-balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Re: Development of organic leg for self balancing character

Who is online