A SERIOUS SURGICAL simulation!

[Charles]

I have been working on a surgical physics simulation as described in the above vedio for months. This is called interventional surgery.The guidewire is nested in the catheter, and the surgeon manipulates the proximal end of the guidewire and catheter to move and twist them around the vessel, with the goal of getting the head of the guide wire to the lesion and then treating it. (You can not see the vessel in the vedio becasue it is invisible on purpose). This kind of simulation of long slender stiff rod with collision is a difficult problem, there are so many papers about it. I have tried Bullet/PhysicX/Obi Plugin of Unity/SOFA/ODE/SimBody, none of them worked until i found the Newton Dynamics!

There are several important physics features of the surgery:
1.The model of tools: The catheter and guidewire are easy to be bended and have strong torsion resistance and can not be stretched. I create many spheres and connected them with joints to simulate the them.
2.The playerinput: The surgeon can only pull/push/twist the proximal of catheter and guidewire.
3.The collision detection and handling: There are two kinds of coliision at first glance. Collision occurs between the guidewire/catheter and blood vessel, or between the guidewire and catheter.
(1)In the first case: The blood vessel is deformed periodically, which means we can create the exact mesh collider at any time. So that the blood can be seen as a rigidbody rather than a softbody.
(2)In the second case: The nest of the catheter and the guidewire should not be treated as a collision problem otherwise it will be so complicated and time consuming! As an alternative, for example, we can force the nodes of guidewire to move along the catheter whenever they are inside the catheter, and use real physics to controll the nodes when are outside the end of the catheter. The inside/outside can be determined by move distance which i think is not a difficult problem.

So far, I have been using the Unity plugin of Newton Dynamics. And I have implemented one guidewire move and twist in a complex blood vessel with no defromed(The deform is just an animation actually).It works quite well!

But there also several problems, it would be appreciated if you can help me:
1.The Unity plugin is not very perfect for now and it seems to have not being maintained since a long time ago. There are several features needed to compelete the project:
(1) Store and load the mesh collider info at running time.
(2) Dynamically enable and disable joints.
(3) Move the base sphere of the entire guidewire totally by user, in another word, the physics behavior of the base spere will not be affected by other newtonbodies.
(4)to be continued.
2. Whar do you think of the whole thing? Is it possible to be implemented with Newton Dynamics?

[Dave Gravel]

Yes sure newton can simulate this pretty easy:
I have already help and work with someone to make the exact same project in the past.
The project is around 13 years old now, I can't talk much about it because I have signed non-disclosure agreement papers.
The only thing that I can say about, It using pure newton lib with old GLScene 3d engine.
Unity engine is not really popular at this old time.
The project started with ode engine at the begin and because of slow simulation with multiple joints and capsule shape collision problems, It using newton dynamics engine as replacement.

https://patents.google.com/patent/EP2467842A1

I can say for sure, Newton can simulate it very good already from far in times.
Sorry about my english.

[Julio Jerez]

I see, I will try to update the unity plugin, no sure how much work will that be since Unity had changed so much after we wrote the plugin but I try.

Your story is not different than the story of each an every one of the people using the engine here.
They try Newton out of desperation because they bought on the narrative put out by the self appointed physics engine appraisals who had made a career of writing papers telling people what engines are good and which aren't.
There is so much misinformation out there that is difficult to keep track of.

Take for example this popular robot simulator that integrated newton as the forth solution.
http://forum.coppeliarobotics.com/viewt ... ton#p30185

The user get better results with newton, but he assumes that he is doing something so wrong that the wrongness makes Newton works as he expected. He assumes he and newton are both wrong and somehow the wrongness cancel out to gives the correct results. Here is a quote from him.

So far I had impression, that Vortex engine is more accurate than Newton, for example after trying out motionPlanningAndGraspingDemo.ttt, so I am struggling to set up the simulation in proper way, so that the Vortex engine works properly with fluent motion. I am wondering if I am doing something wrong so that (as I assume) better engine does not work as it should. I would rather use Vortex if it is more precise, maybe Newton avoids some important parameters here which are just not set properly...

He thinks that some how I am skipping some law of physics that makes his simulation work correct.
Somehow these people thinks I am such magician that I can make such hacks to make rigid body mechanics appear correct but deep down they know I am cheating or doing something wrong.
Never they stop to think that maybe I am just using a correct implementation of reduced coordinates systems mechanics.
That forum is full of people with the same problems: dismembering joints, jittering, mass ratios problems, simulation not given the expected results, that when the try the Newton plugin these problem are solved.
So they ask, why it works with newton and not with the the others because they are afraid they are doing something wrong and Newton is Wrong, but never that maybe it is the other engines who are base on brand new law of physics that might be wrong.

The answer seems to be, of course the users are going to think that way when the admininatrors of the application decided to issue this water mark massage each time a user select Newton as solver

warning: the Newton Dynamics plugin is still in a BETA stage, and you might experience strange/inaccurate behavior, and crashes!

The Newton engine is open source and has been for more than a decade, but they still warn the user from using, why is that? Well they are sponsored by commercial engines, so it is on their best interest that people use those engines, so they have Newton there because without it 9 out of 10 simulation do not work, but that 1 in 10, they hope buy a commercial license.
This is the kind of corruption that goes on behind this.

This has been my disappointment with some of the dishonesty in academia and their hitjob papers.

[Charles]

Yes sure newton can simulate this pretty easy:
I have already help and work with someone to make the exact same project in the past.
The project is around 13 years old now, I can't talk much about it because I have signed non-disclosure agreement papers.
The only thing that I can say about, It using pure newton lib with old GLScene 3d engine.
Unity engine is not really popular at this old time.
The project started with ode engine at the begin and because of slow simulation with multiple joints and capsule shape collision problems, It using newton dynamics engine as replacement.

https://patents.google.com/patent/EP2467842A1

I can say for sure, Newton can simulate it very good already from far in times.
Sorry about my english.

What you said is exciting! Thanks very much!

I have just viewed the link provided by you, it seems like a patent about an endoscope simulator. And i just want to be sure when you mentioned "make the exact same project in the past" , do you mean the same project as the youtube vedio describes the cardiovascular intervention or the link descreibes the endoscope simulator? i just want to be sure about that though the above two are very samiliar to each other when comes to physics simulation.

[Charles]

I see, I will try to update the unity plugin, no sure how much work will that be since Unity had changed so much after we wrote the plugin but I try.

Your story is not different than the story of each an every one of the people using the engine here.
They try Newton out of desperation because they bought on the narrative put out by the self appointed physics engine appraisals who had made a career of writing papers telling people what engines are good and which aren't.
There is so much misinformation out there that is difficult to keep track of.

Take for example this popular robot simulator that integrated newton as the forth solution.
http://forum.coppeliarobotics.com/viewt ... ton#p30185

The user get better results with newton, but he assumes that he is doing something so wrong that the wrongness makes Newton works he expect but he assume is wrong. her is a quote from him.

So far I had impression, that Vortex engine is more accurate than Newton, for example after trying out motionPlanningAndGraspingDemo.ttt, so I am struggling to set up the simulation in proper way, so that the Vortex engine works properly with fluent motion. I am wondering if I am doing something wrong so that (as I assume) better engine does not work as it should. I would rather use Vortex if it is more precise, maybe Newton avoids some important parameters here which are just not set properly...

He thinks that some how I am skipping some law of physics that makes his simulation work correct.
Somehow these people thinks I am such magician that I can make such hacks to make rigid body mechanics appear correct but deep down they knwo is wrong.
Never they stop to think that maybe I am just using a correct implementation of reduced coordinates systems mechanics.
That forum is full of people with the same problems, dismembering joints, jittering, mass ratios problems, simulation not given the expected results.
When the try the Newton plugin these problem are solved.
So the ask why it works with newton and not with the the others because they are afraid they are doing something wrong.

The answer seems to be, of course the users are going to think that way when the admininatrors of that application decided to issue this water mark massage each time a user select Newton as solver

warning: the Newton Dynamics plugin is still in a BETA stage, and you might experience strange/inaccurate behavior, and crashes!

The Newton engine is open source and has been for more than a decade, but they still warn the user from using and why is that. well they are sponsored by commercial engines, so it is on their best interest that people use those engines, so they have Newton there because without it 9 out of 10 simulation do not work.
but that one in 10, they hope but a commercial license.
This is the kind of corruption that goes on behind this.

this has been my disappointment with the academia and their hitjob papers.

I think most people are not familiar with underlying technology of physics engine, so that they tend not to try a new engine until they are sure the new engine can give them what they want. An easy way for these people is just listening to the mojority or the "experts". And as you said, some "experts" will mislead people for commercial purpose. I had tried a few engines myself and read many papers before I came here, but I think most people will not do it this way.

In my opinion, the best way to demonstrate the power of an engine is the showcase. The Newton Dynamics does not have as many showcases especially convicing showcases as some other engines.

Anyway, proof by facts, the Newton Dynamics can do things that many other engines can not at least in my case(stiff long slender rod). I will keep developping my project using Newton Dynamics and Unity(as a render engine). It would be appreciated if you can help me with that. I have been using Unity a lot but almost not using c++ and swig (but learning now ). And i think it will also be a convicing showcase for Newton Dynamics!

[Charles]

The unity plugin i have been using is https://github.com/svenberra/NewtonUnityPlugin.
The NewtonDynamics version is the commit of time 2019.6.30 at github.

And i found a problem in the Unity Plugin. Let's call it Q1.

Q1: As i said before, i'm using joints of sphere nodes to simulate a rod and i set the mass of the first node to 0 such that it won't be affected by physics system.

The question is when i set the postion/rotation of the first node to simulate the player to manipulate one end of the rod, the first node jitters even if there is only itself in the scece.

Below are the vedios and screen captures describing this problem:
https://drive.google.com/file/d/1aL0yLc ... sp=sharing
https://drive.google.com/file/d/11XBwEx ... sp=sharing
https://drive.google.com/file/d/1PQrMI9 ... sp=sharing

[Julio Jerez]

This week I will try to update the pluggin to the correct version of the engine.

Regard to the jitter, my guess is the you set the position directly, and that's one of the thing that is different when using a library like Newton. Teleporting is a no no.

There are other ways to do what you want.
1- get rid of the static body, and replace it with a dynamic body just like the others in the links.
2- connect the body you want to control to the world with a kinematic joint.

Now you can manipulate the kinematic joint with a matrix, and the joint will translate position and rotation changes to the necessary accelration and angular acceleration that will be used by the general solver to calculate the force and torque need to make that move.

When you move a body, you are adding discrepancy to the corrent state of the arrange of bodies and joints, and the only way to fix that is by penalties, but in Newton penalties are intended to correct small integration errors not as a physics feature, so you will get large vibrations.

I am not sure if the kinematic joint is exposed in Unity but after I updated the plugin I will check that it is.

[Dave Gravel]

do you mean the same project as the youtube vedio describes the cardiovascular intervention or the link descreibes the endoscope simulator? i just want to be sure about that though the above two are very samiliar to each other when comes to physics simulation.

For my part I have work and help about bronchoscope project.
I can't say if he have done something about cardiovascular intervention.
The rod system is pretty much the same and look very similar to the video.
The only big difference that I can see is about the human body mesh and the rendering.
I have help to fix problems with ode and the general simulation, But in final it running too much slow with long jointed rods and multiple collisions inside the human body part mesh. I have help to convert the ode physics to newton dynamics physics and it have fix a lot problems and the slow simulation problem too. I can't say more about it, I don't have work more on this project after but I know that Paul Van Dinther have done a lot more with it.
Maybe you can try to contact Paul for more information he have a user in newton forum but I don't remember his nickname here.
It make a very long time that I don't have see him online or talk with him.

The current newton dynamics version running faster with multiple joints and big contact count, I'm pretty sure that the simulated rod can only run and look better now.

the first node jitters

I remember to have get the same problem too.
I remember to have use zero mass too in the past.
It have become fixed with some work around, But with the new kinematic joint like Julio saying it can surely give really nice result without jitter.

[Charles]

This week I will try to update the plugging to the correct version of the engine.

Regard to the jitter, my guess is the you set the position directly, and that's one of the thing that is different when using a library like Newton.
Teleporting is a no no.
There are other ways to do what you want.
1 get rid of the static body, and replace it with a dynamic body just like the others in the link.
2 connect the body you want to control to the world with a kinematic joint.
Now you can manipulate the kinematic joint with a matrix, and the joint will translate position changes to the nessesary force and torque that will be used by the general solver.
When you move a body, you are adding discrepancy to the corrent state of the arrange of bodies and joints, and the only way to fix that is by panalties that are intended to correct small integration errors, not large errors, so you will get large vibrations.

I am not sure if the kinematic joint is exposed in Unity by after I update the demo I will check that it is.

Thanks for your quick reply and detail ! The Unity plugin of current version does not have the kinematic joint so far as i know. It's great that you can provide the newest version of the Unity plugin next week! While waiting for the Unity plugin, I'm going to dig the c++ source code of Newton Dynamics, hoping i can modify the Unity plugin myself soon casue there may be many more interfaces needed to be exposed to Unity when developing my surgical simulation project!

By the way, I am curious that how you make a profit through an open source physics engine? Further to say, what keeps you doing this for so long a time? Just curious. .

[Julio Jerez]

Ok I just sync to the plugin and try to build it.
afte commenting out few lines and adding dAssert (0)
it compiled without any issues. that's not to say I will run.
I have no test it yet, but I have to leave for work now.

if you sync you will get a version that does compile. you can try that out.
then later is very eassy to extend the plugin funtionality but it is a tedious process.
To add a funtionality is a three steps process.

1- you add a c sharp interface in unity for the feature you want to add.
say adding the kinematic joint.
2- you write a c++/header class that glue the Unity interface and the class in newton that support that feature
3-you add the header file to the swig script. file newton.i

then you compile the plugin and swig will generate the wrapper that turn the class into a managed c++ objects that can be called from c chaps.
file newton_wrap.cxx

if everything goes right then you just exposed a new feature.

After we test that the current features are working, we can add the interface for kinematic joint and you can see how the process is done so that you can add what you need as your own time.

on this

By the way, I am curious that how you make a profit through an open source physics engine? Further to say, what keeps you doing this for so long a time? Just curious. .

I have never made a penny from Newton, I do this because I love classical mechanic and since I am a mechanical engineer I do it to try new things.
my main priority is a balance between correctness and performance. but I will not hesitate to throw perforce of the board if is compromises correctness so much that make it unacceptable.
I would accept an algorithm that asymptotically approach correctness if it in fact does.

[Julio Jerez]

How you found out about this plugin, because as far as I know it has been ignored and dismissed offhand without even a test by almost every one over the Unity community.
This do not want anything that may afront their physics solution whatever that is now.

[Charles]

if you sync you will get a version that does compile. you can try that out.
then later is very eassy to extend the plugin funtionality but it is a tedious process.
To add a funtionality is a three steps process.

Thanks a lot for your timely help! I will donwload the new plugin and try to add new features immediately! I think the process of intergrating unmanaged c++ code into managed c# code will happens a lot because i and many others are using Unity and physics engines are often written in c++, so i think i must get very familiar with it even it is tedious.

[Charles]

How you found out about this plugin, because as far as I know it has been ignored and dismissed offhand without even a test by almost every one over the Unity community.
This do not want anything that may afront their physics solution whatever that is now.

Well, i browse lots of papers and try quite a few existing tools before i actually start this project. The process takes me nearly a month or even more and finally it will save my time just as it currently does(i have found Newton Dynamics! ) . In a word, this project is very hard if i start from scratch with all the math and physics so that i determine to use existing tools at last.

The exact process how i found the Unity plugin is more or less like this:
1.I search many physics related keywords mainly on Unity Assert Store/Goole/Github. And i got Newton Dynamics(c++) after some time.
2.So i hope it would be nice if there is a Unity plugin ready-made, then i go to the Unity Assert Store to search for it but got nothing.
3.Then i goto Github to search the plugin and there it is! (But it has been paused for so long a time )

I preferred to search the Unity Assert Store firstly since plugins in that place work out of the box. It could be easier to be accepted by Unity developers if Newton Dynamics is provided in that place.

[JernejL]

This is amazing stuff, it shows how game programming - physics and everything regularly touches real world - there is a reason why tesla and spacex hire a lot of people from game development area.

This simulator is made to teach doctors and to save lives, and uses same programming library that others use to make games - this is amazing!

[Charles]

This is amazing stuff, it shows how game programming - physics and everything regularly touches real world - there is a reason why tesla and spacex hire a lot of people from game development area.

This simulator is made to teach doctors and to save lives, and uses same programming library that others use to make games - this is amazing!

Yes! This simulator is exactly like what you said! I'm on it!