newton on gpu or discrete HPC

A place to discuss everything related to Newton Dynamics.

Moderators: Sascha Willems, walaber

newton on gpu or discrete HPC

Postby Julio Jerez » Mon Sep 24, 2018 12:09 pm

I downloaded intel open cl again in case I made a mistake last time but it does not work.
I keep getting this error "fail to find device"
when I launch visual studio.
If I ignore it, then the code builder menu is not grayout but does nothing.

I went to the intel forums and what I found was apauling, it seems
one in two people have problem getting opencl to work since 2011 or so and morons from intel keep telling to install drivers and some nonsensical answers.
man Intel really is in chambles, they not seem to get anything right.

anyway, I was thinking how to add support in some way for high performance computing and it seems each solution available comes with lots of strings attached. Them it hit me, we have Microsoft AMP.
that use direct compute, I think, but is hidden from the interface. Now that we have plugin support, maybe we should try again see how that goes.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Mon Sep 24, 2018 2:54 pm

to begin this is the test scene which is a pyramid of 100 x 100. (aprox 5000 bodies)
profiler.png
profiler.png (15.81 KiB) Viewed 2209 times


is running on 6 core ceon, and as you can see the solve is about 5 times slower ta the collision/broadphase

There are few thing that can be optimize still an I will do that because that part can't be don on GPU, but of we can remove those point and make the
say twice as fast that this could run in real time. let us see.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby JoeJ » Tue Sep 25, 2018 9:25 am

I wanted to see if there is an AMD CL driver, because it worked for me on Intel as well, but i found this:
https://community.amd.com/thread/230440

Seems AMD has abandoned CL for CPUs.

However, maybe you could use something like this: https://github.com/alexhultman/libvc to bypass the complexity of Vulkan and focus just on the shaders to evaluate performance quickly. (have not tried myself)

To compile a GLSL shader to Spir-V, you need glslangValidator.exe from Vulkan SDK: http://www.lunarg.com

and a bat file like this: glslangValidator -V my_shader.glsl -o my_shader.spv


This is just a suggestion to do some quick testing outside of your repo. (I doubt AMP is that useful.)


Something similar should exist for DX12 i guess... if anyone knows?
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Tue Sep 25, 2018 11:50 am

I am going to give it another try to Microsoft AMP.
last time I try, it was very disappointing.
but maybe is better now.
basically is a template class that launch compute shades.

we could probably do the same, and not rely on the mercy of Intel, Amd, apple, nvidia and Microsoft.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Tue Sep 25, 2018 2:14 pm

Joe remind me of this again?
why can't we just make two dlls NewtonDx and NewtonGl

we start with NewtonDx by copy and pasted the AVX or default and we morph it to hand written ComputeShader for the lower to the higher level?
why is that not a possible solution and forget about all these extra proprietary languages that seem to be solutions in search of a problem.

I committed the testable you use a base lien comparison, and I took a picture if the 100 x 100 stack
profiler.png
profiler.png (17.78 KiB) Viewed 2183 times


the difference is that this one is running on my home pc with four cores, but the iteration count is set to 12 instead of 4 like in the top diagram.
remarkably the setup bring the stack to rest rather quickly so that, so for us that goal standard.

as you can see the solve is about 50 time or more slower and the whole thing runs between
1 and 2 fps
if we can make it 10 time faster with a good GPU, I consider it a success.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby JoeJ » Wed Sep 26, 2018 3:03 am

Julio Jerez wrote:Joe remind me of this again?
why can't we just make two dlls NewtonDx and NewtonGl


Yes. Seems we never get a portable compute API that survives, and with async compute we don't even want this anymore(!) So adressing each API would be the only practical choice.

I would choose DX12 because HLSL has wide support. This grants you peak performance, and you will be up to date with updates. For example, NV talks about 'work generation shaders' (yeah! :D ), which they have for raytracing but it is not yet exposed to any API. DX12 will be first. DX12 has raytracing - DX11 has not and Vulkan not yet.

DX11 users can run your HLSL shaders with little effort.

Vulkan users would take your DX HLSL shaders, compile them to Spir-V and run them inside Vulkan Engine. This should work without a need to port the shaders to GLSL i guess. (Or at least it will work in the future. VK wants to support HLSL. They work on this since the beginning.)

OpenGL users might be able to use Spir-V too? I don't know, a bit unlikely, but there were at least plans for this.


Any user that cares about performance will have to spend work to run your stuff async with graphics. E.g. running Newton shaders while rendering depth prepass and shadow maps. This means we need some Interface like calling Newton->DX12PlugIn->GenerateSolverCommandList (int queue);

Likely there will be a number of command lists for different tasks like collision detection, various solver stages etc. Users may want to distribute them to multiple queues in parallel or to just one queue serially and they will have to care for synchronization themselves. They have to know about correct order and dependencies. The interface will be more complex than we like it to be, as i expect.

Users with unsupported APIs have to replicate that functionality and will likely contribute their API implementation to your open source.


Which means for the start you can pick whatever API you want and ignore the rest, do some experiments until it starts to make sense. And after that we could start talking about Interfaces etc.

You could also create your own shading language and convert it to GLSL and HLSL. But i assume that never becomes necessary.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Wed Sep 26, 2018 7:48 am

JoeJ wrote:I would choose DX12 because HLSL has wide support.


that's what thought, the problem I am having with dx12 is that it use to be that directx had a platform sdk that you download and came with many demos.
now that does no exist, instead in VS 2013 ther is an app wizard that make a basic hideous universal windows app that I do not understand at all.

how do people get start with DX12, I her is so diffrent than dx11 and dx10 that people really nee to start for scratch.
There is a lot more information on vulkan. but I want to start with cx12
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Wed Sep 26, 2018 11:24 am

when I build a test dx12 app in window 2015 I keep getting these moronic errors despite installing all teh nonces they as me to install on VS 2015.

1>DEP0100 : Please ensure that target device has developer mode enabled. Could not obtain a developer license due to error 800704C7.========== Deploy: 0 succeeded, 1 failed, 0 skipped ==========


I start to think these shenanigan Microsoft is doing with is just a sham to fire people to go to MS store, liek apply and Stem are doing now.

the work is going to hell with the all these huge media companies taking every one for fool.
The price we are paying for all the free stuff is just too high.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Wed Sep 26, 2018 2:13 pm

ok I started the project, basically I simply copied the avx solve and renamed.
the is does not do anything in GPU yet, the idea is to morph it to a GPU little by little.
you said that your cpu does not supports avx, tell me when you run can you load any of the GPU solvers?
if not maybe we nee to make a basic one the use SSE so that is use as there reference to copy from.
any way I di no added to the solution instead I am using CMake you make a tailor solution, I star to like this Cmake stuff a lot.
later I see if I can start adding the Directx 12 boiler plate set up, and see how that goes.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby JoeJ » Wed Sep 26, 2018 2:43 pm

I asked someone with DX12 experience.

Many examples here, maybe compiles better than wizard template:
https://github.com/Microsoft/DirectX-Gr ... MiniEngine

or this:
https://gpuopen.com/gaming-product/hell ... dk-sample/

Would be good to find something compute only. Setting up swapchain alone is 100k lines of code - at least :) But for compute you would not need that.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: newton on gpu or discrete HPC

Postby JoeJ » Wed Sep 26, 2018 2:54 pm

...at least nBody sample with compute shader, but graphics too:

https://github.com/Microsoft/DirectX-Gr ... es/Desktop


I cant find a compute only sample. But you could strip down the nBody sample.

Expect even more anger and frustration, much more! :mrgreen:


Edit: Do not remove the debug layer. It really helps a lot.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Wed Sep 26, 2018 3:28 pm

https://github.com/Microsoft/DirectX-Graphics-Samples
ah that's better, lots of solutions there, starting with Hello World.
that's what I was looking for.

I will start there, and yes I hope I do not have to set rendering stuff,
but I have not read stated explicitly this is possible anywhere other than the AMP wrappers.
But I am shying away from AMP, because it seems to be very poorly done.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby JoeJ » Wed Sep 26, 2018 4:26 pm

Compute without graphics really should work, think of using iGPU for compute for example. I know in Vulkan it works, certainly DX12 is the same.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: newton on gpu or discrete HPC

Postby Julio Jerez » Wed Sep 26, 2018 7:15 pm

well following the sample program I added the call to crate the hardware device and seems to work so far.

Joe a few questions.
you said the avx plugin did was no rechniozed in you system, or was it the AVX2.
can you tell me is you can run any of the plugin at all.
can you run CMake?

the reason is that I added this plugin to the cMake this way we can tailor the build to what we are testing.

tonight I will make a SSE plugin in so that we can use as the base lien to compare all others.
but I will not added to the build, instead it will be selectable using CMake,
please when you get time see if you can run Cmake.
Julio Jerez
Moderator
Moderator
 
Posts: 11048
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: newton on gpu or discrete HPC

Postby JoeJ » Thu Sep 27, 2018 2:44 am

After updating CMake, i select all 4 plugins and building projects succeeds without issues.
I open NewtonSDK.sln... 22 projects show up uncluding all 4 plug ins
building ALL_BUILD...

oh, my favorite:
1>------ Build started: Project: ZERO_CHECK, Configuration: Debug Win32 ------
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Cpp.Platform.targets(57,5): error MSB8020: The build tools for v141 (Platform Toolset = 'v141') cannot be found. To build using the v141 build tools, please install v141 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".

Maybe i have chosen VS2017 in CMake instead 2015. So i open sln in 2017... looking good... yeah, everything build successfully.

I run sandbox, but it does not display any info or options on what plug in it is using. :?:
My CPU has only SSE, no AVX. But the sandbox runs fine.

Oh, one problem: Only 32bit build is available, no 64bit. :!:
EDIT: CMake does not show a config option for 32/64 bit.




CMake gui output:
Code: Select all
Selecting Windows SDK version 10.0.16299.0 to target Windows 10.0.17134.
The C compiler identification is MSVC 19.12.25835.0
The CXX compiler identification is MSVC 19.12.25835.0
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x86/cl.exe
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x86/cl.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x86/cl.exe
Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.12.25827/bin/Hostx86/x86/cl.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Detecting CXX compile features
Detecting CXX compile features - done
build visual studio projects
link to static runtime libraries
build 32 bit
CMAKE_CXX_FLAGS_DEBUG is /MTd /Zi /W4 /Od
CMAKE_CXX_FLAGS_RELEASE is /MT /arch:SSE2 /Zi /W4 /Ox /Oi /Ot /Ob2 /fp:fast
dMath
dgCore
dScene
dNewton
dVehicle
newton
dgPhysics
dTimeTracker
dAnimation
dContainers
dCustomJoints
tinyxml
glfw
imgui
dgNewtonSse4.2
dgNewtonAvx
dgNewtonAvx2
dgNewtonDx12
demosSandbox
DebugRelease
Configuring done
Generating done
Last edited by JoeJ on Thu Sep 27, 2018 2:52 am, edited 1 time in total.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 2 guests