what pofiler people use?

A place to discuss everything related to Newton Dynamics.

Moderators: Sascha Willems, walaber

what pofiler people use?

Postby Julio Jerez » Mon Oct 13, 2014 12:34 pm

I am planning to remove all the profiling code form the engine, and use some kind of instrumented profiler that is not part of the engine.

A cool profiler is Telemetry by Rad tool. However they do not list the price of the license.
has any one use it or know of an equivalent?

I do not know of any that works like Telemetry, Telemetry I believe is inspired from how Console profilers work, which are frame capture profiles.

I would hate to have to write a fame capture profiler tool.
I do not really like profile like Vtune or AMD, they are good for finding really bad functions, but not for optimizing overall subsystems frame by frame.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby manny » Mon Oct 13, 2014 3:20 pm

Microsoft ships AWESOME tools for console development. however these are only for licensed developers :(
Apple also ships super tools together with xcode for both desktop and mobile development.
Unfortunately I can't recommend anything for windows based desktop development.
http://www.instaLOD.io - InstaLOD - State of the art 3D optimization
manny
Site Admin
Site Admin
 
Posts: 131
Joined: Tue Feb 11, 2014 6:49 pm

Re: what pofiler people use?

Postby Julio Jerez » Mon Jun 01, 2015 12:54 pm

well I send an email to Rad for a Telemetry quote, they didn't even bother to reply.

I guess I am going to have to write an open source Frame capture profiler them.
one the capture frame by frame and use either a file output or a server.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby Pierre » Tue Jun 16, 2015 5:04 pm

I hear that VerySleepy is great for a quick analysis. It will tell you what is slow, but not why it is slow.

V-Tune will tell you why.
Pierre
 
Posts: 30
Joined: Tue Jun 16, 2015 4:40 am

Re: what pofiler people use?

Postby JoeJ » Sat Jul 04, 2015 12:12 pm

AMD CodeXL supports CPU profiling, but i'm not sure this means generic code and Intel, not just OpenCL kernels.

http://developer.amd.com/tools-and-sdks/opencl-zone/codexl/codexl-benefits-detail/

I liked it a lot for OpenCL.
User avatar
JoeJ
 
Posts: 1147
Joined: Tue Dec 21, 2010 6:18 pm

Re: what pofiler people use?

Postby Julio Jerez » Sat Jul 04, 2015 1:00 pm

had anyone seen Telemetry?
I can't believe there isn't an open source equivalent.
The closest thing I found is Visual studio concurrency visualizer.

I asked Rag tools for a quote and after several called they finally replied, but unfortunately the price is ridiculous, $7000.00 a year for a single license and you are not allowed to distribute code, not even the integration. Those people are out of touch with reality.
It takes no more than a week maybe two to write a profile just like telemetry, the only problem is that it diverts my attention.
Maybe some day I write a frame capture thread profile.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby lperkins2 » Mon Jul 13, 2015 6:29 pm

Have you considered gprof, and compiling with gcc's -pg ? So far it's done everything I want, and is fairly simple to use.
lperkins2
 
Posts: 37
Joined: Fri Jul 03, 2015 4:16 am

Re: what pofiler people use?

Postby Julio Jerez » Wed Mar 09, 2016 8:15 pm

well in the end after being sick and tire of trying to figure out how to profile the engine, I decided to write my own instrumented profiles.
I turned out that this is quite trivial app, that only contain one file on the server size.

I called the people at Rag to as for a quote for Telemetry, but the ask for almost 8 grand's a year for a license and non of the code can be published, so that's not an option.

I am writing one that can emulate the same basic functionality, and I am not sure if I should make a stand alone app so that people can use it for profiling anything.

Maybe people want to contribute o n writing the Client, to give an idea of how the client should present that data here is how Telemetry looks like: http://www.radgametools.com/telemetry.htm

we do not need all the whistle and belts that they have, but the implementation already track Thread, what is what I am interested in the most.

I will star the client using wxWidget, but maybe the client can be a C sharp app,
It could be written on any language, since it only needs to present that dat frame by frame.

I am not planning to make it that connect to the serve at runtime. After many years I found out that all those thongs are quiet useless, off line is better because and the end of eth date what you want is a history of several runs to see the progress on some algorithm.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby godlike » Thu Mar 10, 2016 9:10 am

I had the same problem with you. Couldn't find a decent tool to profile my multithreaded engine. What I did was something super simple and I have to say that it worked quite well. So here is how it works:

Inside my code I start and end trace events:

Code: Select all
ANKI_TRACE_START_EVENT(SCENE_VISIBILITY_TESTS);
// Execute code...
doTests();
ANKI_TRACE_STOP_EVENT(SCENE_VISIBILITY_TESTS);

void doTests()
{
    ...
    ANKI_TRACE_START_EVENT(SCENE_VISIBILITY_COMBINE_RESULTS);
    ...
    ANKI_TRACE_STOP_EVENT(SCENE_VISIBILITY_COMBINE_RESULTS);
}


At the end of my frame (for your case the end of NewtonUpdate() and NewtonUpdateAsync()) I call

Code: Select all
ANKI_TRACE_STOP_FRAME();


Quite simple. The ANKI_TRACE_START_EVENT and ANKI_TRACE_STOP_EVENT are thread safe and ANKI_TRACE_STOP_FRAME will have to happen at a sync point (where there are no threads working).

ANKI_TRACE_STOP_FRAME gathers all the event information and appends them in a .json file. This json file is what google is using for tracing inside android and for chromioum browser. So I use chrome to view the trace file, zoom, measure the time etc.

Zoomed out:
Image

Zoomed in:
Image

And here is a very small example json:
http://pastebin.com/V0rgVYUR
User avatar
godlike
 
Posts: 58
Joined: Sun Mar 16, 2014 3:48 am

Re: what pofiler people use?

Postby Julio Jerez » Thu Mar 10, 2016 2:25 pm

ANKI_TRACE_STOP_FRAME gathers all the event information and appends them in a .json file

is this .json some public format that I can export to?
Microsoft has one app named concurrency visualized, I though of using that but it is a very crude app.
and as everything MS does is proprietary, so in the long run that will not work.
I for now will do windows, by there is nothing particular that makes it window only.


I almost have the server code written, is a dll and is use the same way you shows,
I still have much work to do but this is how is look like now.
Code: Select all
#define dTimeTrackerCreateThread(name)                                    \
   dTimeTracker::GetInstance()->CreateThread (name);

#define dTimeTrackerTrackTime(name)                                       \
   static dCRCTYPE __crcName__ = dTimeTracker::GetInstance()->RegisterName (name);   \
   dTimeTracker::dTimeTrackerEntry ___trackerEntry___(__crcName__);   

#define dTimeTrackerSync()                                             \
{                                                                                   \
      dTimeTrackerTrackTime("timeTracker")                              \
      dTimeTracker::GetInstance()->SavaData ();                           \
}


basically those macros will save the time information per frame per thread, and save then to a file.
My plan is to make so that is profile a finite number of frame. say 100 or 200, it could be configurable.
Now I need the document viewer which I will use C# for simplicity.

I did some test already and it works, but then I removed the macros placement because I want to add it to a new configuration.
If you sync the code will find a new third party project called timeTracker that implement the sever code.
I do no think it will be too hard to get this going , and I believe it will be nicer that I first thought.

I have to do that because there are some areas of the engine that are optimized, and it would be hard to make faster, the bigger problem is the load balance in multithreaded mode.
Basically I nee per frame localized profiler data, that something that sampling profiler like vtune, Very Sleepy or AMD do not do.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby godlike » Thu Mar 10, 2016 2:56 pm

Julio Jerez wrote:is this .json some public format that I can export to?


Yes. Here is the "spec" https://docs.google.com/document/d/1CvA ... SU/preview

Also my implementation is here:
https://github.com/godlikepanos/anki-3d ... re/Trace.h
https://github.com/godlikepanos/anki-3d ... /Trace.cpp
User avatar
godlike
 
Posts: 58
Joined: Sun Mar 16, 2014 3:48 am

Re: what pofiler people use?

Postby Julio Jerez » Thu Mar 10, 2016 3:19 pm

Oh wow, I did not know there was and standardized format for that, this save me a lot of time,
I can export the dat to that format and use what toll to see the result,
thank you very much. :mrgreen:
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby Julio Jerez » Thu Mar 10, 2016 3:45 pm

one more question, I do not see any application that I can run to visualize that data?
I look a google chromium and I see a bunch of video f people show lot of web deep stuff.
what app are you using to view the data the profiler outputs?

I found this:
http://www.chromium.org/developers/how- ... iling-tool

is that the tool? because it seems to be just for profiling Google Chrome.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby Julio Jerez » Thu Mar 10, 2016 3:58 pm

I found this article:http://www.gamasutra.com/view/news/176420/Indepth_Using_Chrometracing_to_view_your_inline_profiling_data.php
seem to have all the info I need. here is the key paragraph. This is awesome. :mrgreen:
Viewing external profile data files
One of the more unsung features of the tool is that it allows you to load external log files, which is the coolest part of the tool for non-web and non-mobile developers. By modifying your game's existing inline-profiling code to output a JSON object compatible with the chrome://tracing tab, you can use it as your visualization tool for your game's profiling purposes.
Julio Jerez
Moderator
Moderator
 
Posts: 11043
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: what pofiler people use?

Postby godlike » Thu Mar 10, 2016 6:10 pm

Julio Jerez wrote:one more question, I do not see any application that I can run to visualize that data?
I look a google chromium and I see a bunch of video f people show lot of web deep stuff.
what app are you using to view the data the profiler outputs?

I found this:
http://www.chromium.org/developers/how- ... iling-tool

is that the tool? because it seems to be just for profiling Google Chrome.


You just open chrome and enter about:tracing and that will load a "special" page. Pretty cool :)
User avatar
godlike
 
Posts: 58
Joined: Sun Mar 16, 2014 3:48 am

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 9 guests