Getting piles to settle

[Bird]

Hi Julio,

Is there anything in Newton4 that can help with getting large piles of bodies to settle down? I'm rendering with a path tracer and the top level BVH acceleration structure has to be rebuilt every tick if there is motion in the scene. Right now, I check the each body's sleep state during each tick and when all bodies are sleeping, then I can stop rebuilding the BVH. This works great in a lot of scene but when there's piles of bodies there always seems to be some small intermittent motion within the pile.

Here's an example of what I'm talking about
https://youtu.be/FKb8j9i_oks

[Julio Jerez]

Are those piles of convex hulls?

I have not tried stuff like that yet.
There are few tricks in 3.14 and lower to do that, but the problem in that 3.14 has every varian of the solver tweaks since newton 1.0
For newton 4.0 I simply took the vanilla version of the solver
and added that with out any thick.

But yes I know I have to check that issue at some point.
I will check after I add the rest of the collision shapes and the serialization.

this solver is different from previous in that it is not an island solver, instead I solves for all active bodies and their immediate neighbors.
But that does not seems to be enough as it keeps doing lot of churning when bodies are in precarious unstable state.

Instead I think it needs to do two rings. active, immediate, and next immediate.
The other part is that the default drag in newton 4 is zero.
As opposed to in 3.14 and lower. So energy is never drawn out from system, this help to stabilized thing in these cases.
I need to see which of these policies to adopt, but at the same time I want to minimize the coaching on behave of the user.

Anyway, the answer is yes that will be fix, but after we get the rest of the shapes in.

[Bird]

Yes, those a convex hulls. Thanks for looking into.

Newton4 is really coming along nicely. I find it's much easier to use and understand and doesn't require much user code to get up and running!

[Julio Jerez]

Newton4 is really coming along nicely. I find it's much easier to use and understand and doesn't require much user code to get up and running!

yes you get the goal. that is one of the primary goal of this iteration of the engine.
this is not that much different that the original, since the original was always C++ underneath.
but it has been almost 18 years, and at the time new languages were emerging and had problems interfacing with c++, so I decided to make a C wrapper.
The problem was that as the time pass, much functionality was implemented in the C wrapper, and we end up with an amargaman of c and C ++ making it hard for many users to navigate.

this time around the primary interface will be C++, then I will add the C wrapper, that will expose the needed functionality on demand. Then people can extend it for their needs.

the cool thing is that the CPP core engine is very powerful, and exposes everything.

so the primary goal is essy to use interface.

the engine will also have a file format similar to what Gazebo uses.

this format is not serialization, instead will be a legitimate xml format the will do only the physics part. the graphic will be saved as url references to some external format that can be fbx, obj, PLY etc.

this way the users can import thier own complex models. for example the excavator demo or a vehicle. It takes lot of work to make and almost no one understand it, even I have a hard time navigating thought it, so it become s a curiosity.
I hope these options help with the users.

I now complete the fbx import so after this it will be easy to share scenes for debugging.

[Bird]

this format is not serialization, instead will be a legitimate xml format the will do only the physics part

A lot of active projects I see these days are using json instead of xml. I use this json library and find it to be really well done, easy to use and well supported.
https://github.com/nlohmann/json

I now complete the fbx import so after this it will be easy to share scenes for debugging.

I see you are using OpenFBX which seems like a nice project but there don't appear to be many FBX exporters for us to use to share our scenes for debugging. I found this one but it hasn't been updated in 4 years.
https://github.com/jskorepa/fbx

[Julio Jerez]

Jhon.
I will considered. The one thing I do not like about xml is how wastfull of memory they are. Plus they require to load the object twice. Is possible to parse in place, but this is a feature of xml almost no one uses. Tinyxml2 promise to fix the extra memory but I have not try yet.

If json fixes that, plus is more human readable, yes I go with json.

On the fbx, yes this openfbx is quite nice. I have to fix some bug in the library. It appears they either used the method that the Blender folk used to reverse engineer the format, or the other way around.
I say this because file with deep hierarchies exported by the original fbx library from aurodesk, load incorrectly in blender
And when I load the excavator demo with open fbx, it looks identical to how they look in blender.
I found the bug in the part where they calculate the transform offset of submeshes.
So it only works correctly if the all of the gizmos transform are frozen. But that actually ruin any authoring that any modeler did to a rig.
After I fixed that, now seem to load the file as they are exported from max or from blender.
There are some other missing features like material paramers, but those can be added if you compared to the original fbx headers.

The reason I do not use the original fbx libraries from auto desk is that they have a restrictive license that forbid distribution of the libraries but allows distribution of the files. Unless you get permission.

I wonder how engines like unity and unreal handle it, but I think this is the reason blender uses his own parcel, but has some bad bugs that make fbx generated in blender be incompatible with max.
I see those king of issue with collar as well, where there collada specification are so flexible that collar file expert by max can't be loaded in sxi and they use the same library. I dispice collada specifications.

The other alternative is to use assimp, but it is too huge of an over kill for me. Since I only need a lightweight importer of a general enough formal for the demos.

[Julio Jerez]

al right bird, we nwo can export and import scenes.

I very simple still but I believe I can enough so that you can export you scene and then shared for testing.

to export a scene you call void Save(const char* const path) const;
on the newton object.

I end up using xml, I knwo the JSON are much nicer, but I found it another distraction, I did not even try tynixml2 since when I try building is gave ton of duplicate macros error.

anyway teh export is set that you can embed it in to make that was already tiny xml but using
void Save(nd::TiXmlElement* const rootNode, const char* const assetPath) const;

it also exports large meshes files as binaries, I should probably do the same for convex hulls but for now they are part of the xml file.

It does not export use data, but since it can export user bodies, teh use can add whatever they want.
with teh demos I will probably expert references to the fbx. but for now to see the import scene you need to select the solid or wire frame debug option from the menu.

now after I make export the players, they I can go one with more engine functionality.

[Bird]

I can't seem to get this latest version to compile. Yesterday I had no problem using my same build system. Any ideas what might cause this. The NewtonSDK github project compiles and runs fine. I've checked that I have all the proper defines
_NEWTON_STATIC_LIB", "_CUSTOM_JOINTS_STATIC_LIB", "_DVEHICLE_STATIC_LIB","_CNEWTON_STATIC_LIB"
"_WIN_64_VER",
"_D_SINGLE_LIBRARY",

Error message is in Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include\sal.h

Error C2144 syntax error: 'int' should be preceded by ';' (compiling source file ..\..\..\newton4\source\dNewton\ndWorld.cpp) Newton4SDK C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include\sal.h 2361

Code: Select all

#ifdef  __cplusplus // [
#ifndef __nothrow // [
# define __nothrow __declspec(nothrow)
#endif // ]

extern "C" {  // This line causes the error

#else // ][
#ifndef __nothrow // [
# define __nothrow
#endif // ]
#endif  /* __cplusplus */ // ]


[Julio Jerez]

umm, I just build using vs2019 and was fine, but you did that.
I do not know what that could be, the new functions added where Load and Save.

try comment out the body implementation and see if you narrow to what part cause that error.
The the change since yesterday is that the new function take an extra argument.
I really do not like when visual studio gives those and the internal compiler errors.

do no forget to click Install so that the new header are copy to the install folder. Maybe that's part of the problem.

[Julio Jerez]

I still use VS 2015, but as a note, when I buidl with VS 2019 I got few warning in the engine code that I fixed. so try sync again.

[Bird]

No luck with the latest github. Googling suggests that it might be caused by a syntax error somewhere but I checked and I'm using the same files that build fine in the NewtonSDK distribution. Grrr
https://social.msdn.microsoft.com/Forums/en-US/8a545a3c-4faf-418e-a470-7fe713c567ac/salh-error-c2144-syntax-error-int-should-be-preceded-by-?forum=vcmfcatl

I checked again on a version from yesterday and there's no problem. I'll try to track it down tomorrow.

[Julio Jerez]

Thats frustrating, the thing is that in general for syntact errors the compiler usually tell the line number of the file that last originated the error.

Anyway on the bright side in testing the file export, I discover the pop problem that you first showed me. An I were able to make a repro.
I think I know what it is, so I will try to fix it.

[Julio Jerez]

Can you tell me the last version the compiled;
There must be a header file missing.
We should be able to find it be looking at the history of changes.

[Bird]

I made a repo for Newton4 using my build system. Could you try it and see if you hit the same error on
vs2019 and vs2015. I don't use vs2015 here but I set up a bat file for it

https://github.com/Hurleyworks/Newton4StaticLibrary

To generate a solution of your choice, just double click on one of the generateVS20*.bat file which will create a build folder for the compiler you chose.

The last one I saved here that compiled didn't have any xml code added yet.

[Bird]

I did just find an older version that works that includes dTinyxml. I see if I can find the difference when I get a chance later today.