Introducing Newton Scripting language

A place to discuss everything related to Newton Dynamics.

Moderators: Sascha Willems, walaber

Introducing Newton Scripting language

Postby Julio Jerez » Thu Dec 13, 2012 6:48 pm

for moe time I have being working on the next step fo rteh engine whis is the Scripting and AI.
this week I put some more work on teh script system which is a object orieted laguage derive from Java and csharp
I would going with Lua but teh lack of OOP in Lua is a turn off for me. oen fo eth main reason fo rthsi scrip laguan si that it will have buidl in support for emdibng with oteh laguages.
basically It will use the Java native funtion support for that.
It is also an optimizing compiler, I still have long way to go but here is ha oputput of a intemediate laguge code genmerated for a naive quick sort

thsi si a calss example
Code: Select all
// a very naive qsort for integers
public class qsort
{
   // very primitive qsort
   public void sort (int[] a, int m, int n)
   {
      if (n <= m) {
         return ;
      }

      int i = m - 1;
      int j = n;
      int v = a[n];
      int x;
      
      while (1) {
         do
            i = i + 1;
         while (a[i] < v);
         
         do
            j = j - 1;
         while (a[j] > v);

         if (i >= j) {
            break;
         }
         
         x = a[i];
         a[i] = a[j];
         a[j] = x;
      }
      
      x = a[i];
      a[i] = a[n];
      a[n] = x;
//      sort (a, m, j);
//      sort (a, i + 1, n);
   }
}


this is the intemediate laguage generated

Code: Select all
[code]optimized version
function sort
   argument a
   argument m
   argument n
   local scope0_i
   local scope0_j
   local scope0_v
   local scope0_x
   if (n <= m) goto exit
   scope0_i = m - 1
   scope0_j = n
   t0 = n * 4
   scope0_v = a[t0]
label3:
   scope0_i = scope0_i + 1
   t1 = scope0_i * 4
   t2 = a[t1]
   if (t2 < scope0_v) goto label3
label7:
   scope0_j = scope0_j - 1
   t3 = scope0_j * 4
   t4 = a[t3]
   if (t4 > scope0_v) goto label7
   if (scope0_i >= scope0_j) goto label2
   t5 = scope0_i * 4
   scope0_x = a[t5]
   t6 = scope0_j * 4
   t7 = a[t6]
   a[t5] = t7
   a[t6] = scope0_x
   goto label3
label2:
   t8 = scope0_i * 4
   scope0_x = a[t8]
   t9 = n * 4
   t10 = a[t9]
   a[t8] = t10
   a[t9] = scope0_x
exit:
   ret[/code]


this code is almos as good as what Visual studio prouce ins release build, but it use a lot more registers.
my next step if to use and dynamics Registare allocation, to regide the number or local variables, and this will make a good canditate for Just in time compiler.
I have big plan for this project
I anyone can comment I like to hear soem opinions about this project.

This script languge is a standalone project.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Thu Dec 13, 2012 8:24 pm

for a reference to compare here is what visual studion 2008 in release mode generate for the same sorce file

Code: Select all
PUBLIC   sort
   cmp   ebx, ecx
   jle   SHORT $LN11@sort
   push   ebp
   push   esi
; Line 15
   mov   esi, DWORD PTR [eax+ebx*4]
   push   edi
   dec   ecx
   mov   edx, ebx
   npad   3
$LL7@sort:
   inc   ecx
   cmp   DWORD PTR [eax+ecx*4], esi
   jl   SHORT $LL7@sort
$LL4@sort:
   dec   edx
   cmp   DWORD PTR [eax+edx*4], esi
   jg   SHORT $LL4@sort
   cmp   ecx, edx
   jge   SHORT $LN17@sort
   mov   ebp, DWORD PTR [eax+edx*4]
   mov   edi, DWORD PTR [eax+ecx*4]
   mov   DWORD PTR [eax+ecx*4], ebp
   mov   DWORD PTR [eax+edx*4], edi
   jmp   SHORT $LL7@sort
$LN17@sort:
   mov   esi, DWORD PTR [eax+ebx*4]
   mov   edx, DWORD PTR [eax+ecx*4]
   pop   edi
   mov   DWORD PTR [eax+ecx*4], esi
   pop   esi
   mov   DWORD PTR [eax+ebx*4], edx
   pop   ebp
$LN11@sort:
   ret   0



as you can see visual studion et away with an adressing mode of the x8c cpu here

Code: Select all
$LL7@sort:
   inc   ecx
   cmp   DWORD PTR [eax+ecx*4], esi
   jl   SHORT $LL7@sort
$LL4@sort:
   dec   edx
   cmp   DWORD PTR [eax+edx*4], esi
   jg   SHORT $LL4@sort


my script code need to generate the addresing mode wih two instrution, because ity is designe for general porpuse CPU
Code: Select all
label3:
   scope0_i = scope0_i + 1
   t1 = scope0_i * 4
   t2 = a[t1]
   if (t2 < scope0_v) goto label3
label7:
   scope0_j = scope0_j - 1
   t3 = scope0_j * 4
   t4 = a[t3]
   if (t4 > scope0_v) goto label7



it is my contention that the script code is faster that teh x86 code, so I expect the script to run with in a fraction o feth speed of any native code on teh same machine, evenm running under teh virtual machine intepreter. :D
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby JoeJ » Fri Dec 14, 2012 4:07 am

So the intermediate language is what you compile from your script, and JNI compiles that to native code.
For the last step is Java required to be installed, or is it enough to link to some lib to enable that?
Do you plan to integrate some luxury stuff like std::vector, map, list..., like in c#, where that's part of the language?

And what plans do you have for AI? Pathfinding / Hide / Attack / Run away / Crowd behaviour...?
I'd just like to get a raw impression, so i can leave that stuff at the bottom of my todo list, where it currently is :)
User avatar
JoeJ
 
Posts: 1453
Joined: Tue Dec 21, 2010 6:18 pm

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Fri Dec 14, 2012 7:43 am

no java is not required, this is model afte jave cchapr and c++ by taking what is common across teh languages and using of a script system, but it is not any of those languages.

ultimatlly as I study more an more those languages, I can see why Java Grammer is better for a scripting language than csharp and c++, therfoe I am adopting more form jave that form the other.

since the schipt will be a subset of Java, any script source code will run as a java program under a Java virtual machine without modifications, that can be use to test and debug the script.
also a java source migh be able to run as as script provided it does no use exotic funtilality like expection handling which I do not see usful for a script system
but I re iterate thsi is stand alone scripting language, it will no nedd anything to run on any computer.

The compiler conver teh script to a intermadiate code. teh intemediate code have teh sytac of a simple C programe where each intruction is of the form

x = op (y)
or
x = y op w

this is eassy for the optimization by the optimization face but is not required.
the virtual machine runs a special version of this code, a special version is one where of tha intemerica variavle are mape to a limite number of register

for example say we have a x86 cpu, we tell the scrip compiel to generate the code with 6 registers.
but if we have a machine with 32 registers, we then use say 24 registers. and so on
the result is that then more registe the fewer registe spill and more eficienet code.

in late face the code can be tralate instrution by instruction to Just in time compiler but that is and option that is far in the future, or I can leave for a third party to do it.

for map, list and oeth funtion form ths stl, java hav eteh copspt of Generic (the equvalen of templates in c++)
vertion 2.0 o fteh script languge will soprt Generic as well, therfore it iwll support map and list and vector but no as native feature and but as standard library
version one will come with small standard library hat will include strings and input.

for AI, a while back I started the newton AI, which at its core have a hierchical state machine, the script system will be integrated to that system by using the native interface feature of the script
and yes all of the function like path finding will be part of the system.


in teh next few day I will make a newton demo demo tha use the script syetm for controlling some NPC players.
if foudn that making the demo all in CPP is too time consuming and they can no be too complex other wise teh CPP code get in teh way of people undertadind the system
this problem can be all avoid by using a script system that is integrate to teh engine and resolve lost in tralation form codeing style, compel c++, and teh like

basically this demom will be a bank system that use the engine functions to make control players all with the scripting system
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Fri Dec 14, 2012 12:48 pm

now I have that in generate the function intemediate function with all the local variable at the begining.
Thsi mak et eassier for the byte code generation.
Instruction local can be ignored or can be a registe depend of the target mnachine or target virtual machine.
in this case the optimize version of a quick sort use 17 register, but I have no writter teh rgist allocation function which should reduce that to may 5 or 6
this is how the intemediate code looks now.
Code: Select all
optimized version
function sort
   argument a
   argument m
   argument n
   local $2
   local $3
   local $4
   local $5
   local $6
   local $7
   local $8
   local $9
   local $10
   local $11
   local $12
   local $13
   local $14
   local $15
   local $16
   if (n <= m) goto exit
   $2 = m - 1
   $3 = n
   $4 = n * 4
   $5 = a[$4]
label3:
   $2 = $2 + 1
   $7 = $2 * 4
   $8 = a[$7]
   if ($8 < $5) goto label3
label7:
   $3 = $3 - 1
   $9 = $3 * 4
   $10 = a[$9]
   if ($10 > $5) goto label7
   if ($2 >= $3) goto label2
   $11 = $2 * 4
   $6 = a[$11]
   $12 = $3 * 4
   $13 = a[$12]
   a[$11] = $13
   a[$12] = $6
   goto label3
label2:
   $14 = $2 * 4
   $6 = a[$14]
   $15 = n * 4
   $16 = a[$15]
   a[$14] = $16
   a[$15] = $6
exit:
   ret



I can see a problem wit the laguage already, for example instrution $16 = a[$15]
is inconsistent because teh intrution set fo rteh virtual machine is risk like therefore, "a" whsi is a local variable is inconsistent
this can be fis by make the param stamenet issue param and then move, therefore it will preload all arguments into local variable and it will work with local variabel only.
the the rigister allocation will decide which register to spill out of the register set when the number of register is larger that the maximun allowed.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby JoeJ » Sun Dec 16, 2012 3:57 pm

Sounds good!
I have the idea to put a gui on top of a script language.
Somehow like Google Blocks, but hopefully more suited for the non-progarmming, creative player.
For that i'd need to be able to handle endless loops, crashes, bad menory access etc...
I can handle endless loop with an additional counter, and the other cases should never happen in a script language, true?
User avatar
JoeJ
 
Posts: 1453
Joined: Tue Dec 21, 2010 6:18 pm

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Tue Dec 18, 2012 9:21 am

once you have a poweful compulir, the posibilitie are enless, the good thong abpout this script is that it can be extended by teh user.

say for exampel you are making a system wher you wna to add GUI funtionality as a native part of your system
all you have to do is this
-open the script language grammar
-find the statement block

Code: Select all
Statement
   : EmptyStatement                                    {$$ = $1;}
   | Block                                             {$$ = $1;}
   | ReturnStatement                                     {$$ = $1;}   
   | ExpressionStatement                                  {$$ = $1;}
   | ConditionalStatement                                 {$$ = $1;}
   | IterationStatement                                 {$$ = $1;}
   | JumpStatement                                       {$$ = $1;}      
   | SwitchStatement                                    {$$ = $1;}
   ;


edit the stament block, by addin a new rule to the list, for example


Code: Select all
Statement
   : EmptyStatement                                    {$$ = $1;}
   | Block                                             {$$ = $1;}
   | ReturnStatement                                     {$$ = $1;}   
   | ExpressionStatement                                  {$$ = $1;}
   | ConditionalStatement                                 {$$ = $1;}
   | IterationStatement                                 {$$ = $1;}
   | JumpStatement                                       {$$ = $1;}      
   | SwitchStatement                                    {$$ = $1;}
                | MyNamiveGUI                          {$$ = $1;}
   ;


then in anothe line you define teh grammer rule for that statement
then you cna eieth add a new class ot a that will tokenize the new statement to a machibe code,
for voila you nwo have native support to you new statement

what this does i sthat we can have custom native buiding for end use need, say fo rexample you wan to make your classes have a game entrity as memeber, you cna added as part opf teh script, not binding required
but that is not all thera re lot opf flexibility
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Tue Dec 18, 2012 9:28 am

now I am writing teh register allocation, for test I am using this simple function


Code: Select all
   public void sort__ (int b, int c)
   {
      int a = 0;
      do {
         b = a + 1;
         c = c + b;
         a = b * 2;
      } while (a < 10);
      return c;
   }


The compile front end tranlate it to this

Code: Select all
function _sort__
   argument _b   ; passed in temporary t0
   argument _c   ; passed in temporary t1
   t2 = 0
   local scope0_a
   scope0_a = t2
label1:
   t3 = 1
   t4 = scope0_a + t3
   t0 = t4
   t5 = t1 + t0
   t1 = t5
   t6 = 2
   t7 = t0 * t6
   scope0_a = t7
   t8 = 10
   t9 = scope0_a < t8
   if (t9 != 0) goto label1
   t10 = t1
   goto exit
exit:
   ret t10


as you can see this is not effiecnt at all, but it is usful for teh debugger whiuc will be later,
the optimzation face is translate the code this

Code: Select all
before register allocation
function _sort__
   argument _b   ; passed in temprary t0
   argument _c   ; passed in temprary t1
   local scope0_a
   scope0_a = 0
label1:
   t0 = scope0_a + 1
   t1 = t1 + t0
   scope0_a = t0 * 2
   if (scope0_a < 10) goto label1
   ret t1


much more effiecent, but still use lot of generaic tree argument stastement,
the register allocation now translate the same code to this

Code: Select all
function _sort__
   argument _b   ; passed in temprary t0
   argument _c   ; passed in temprary t1
   reg1 = 0
label1:
   reg1 = reg1 + 1
   reg0 = reg0 + reg1
   reg1 = reg1 * 2
   if (reg1 < 10) goto label1
   ret reg0
 


it only use tow registers, and each struction is tow argement instead fo three argumnet, thsi mean thsi cide can be translater stragght without any modification to an intell precesor if we want to
btw thsi code optimized at teh same level that Visual stidio 2010 in realise mode.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Stucuk » Wed Dec 19, 2012 5:14 am

Will you be able to register functions? Like LUA's lua_register() command? So that the script can fire off functions in your apps code.
User avatar
Stucuk
 
Posts: 801
Joined: Sat Mar 12, 2005 3:54 pm
Location: Scotland

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Wed Dec 19, 2012 7:27 am

Oh this is going to be much cooler, say you wnat to bind eh script to you propject

you make a you class like this

Code: Select all
public class myclass
{
        // function call to bind to host project
         public native int SomeSpecialFunction( int x, int y);
         public native int OtherSpecialFunction( int x, class otherobjects);
       
       // funtion on teh script
       public int SomeOtherFunction( int x, class otherobjects)
      {
           // I can do stuff here, for example
          int x =  SomeSpecialFunction (morevalue, someMember);
      }
   
     int morevalue;
      class someMember;
}


bascially you add the keyword "native" to funtions you want to call or that you want the applciation to call you by, the script compile generate this header file
Code: Select all
#include <ScriptBinding.h>
int myclass_SomeSpecialFunction_ (ScritptVitrualMachine* const virtualMachine,   ScriptObject* const me,  int x, int y);
int myclass_OtherSpecialFunction_ (ScritptVitrualMachine* const virtualMachine,   ScriptObject* const me,  int x, ScriptObject* const otherobjects);

you use that header file to implement you function to interface with that class

ScriptBinding.h will be the SDK for interfacing with the virtual machine.
you can call function of a call, you can find classes, and do whateever is needed to do a two way interaction with the script.

to call any function on of a script class you can use the virtual machine, even for calling a native function.
or you cna call directpy any native function form you applycation bypassing the virtual machine.

this is the plan
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Stucuk » Wed Dec 19, 2012 7:35 am

So its a C++ only thing then? Not cooler if it is. LUA's register method is language independent.
User avatar
Stucuk
 
Posts: 801
Joined: Sat Mar 12, 2005 3:54 pm
Location: Scotland

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Wed Dec 19, 2012 7:39 am

not this is not c++, this is languge independed

the script system in object orierented, but it can inteface to any language by making script class function that are impelmnetd by the host application. not registraion is needed you just decalre a funtion in teh class
and the script compiler do the bind work for you by creating thsi header C file in
Code: Select all
#include <ScriptBinding.h>
int myclass_SomeSpecialFunction_ (ScritptVitrualMachine* const virtualMachine,   ScriptObject* const me,  int x, int y);
int myclass_OtherSpecialFunction_ (ScritptVitrualMachine* const virtualMachine,   ScriptObject* const me,  int x, ScriptObject* const otherobjects);


the header can even be made of oteh laguages as well
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Wed Dec 19, 2012 11:51 am

to give you an idea how powerfull the compile is, here is a function to calculate the fibinacii value of a number
Code: Select all
   public int Calculate (int n)
   {
      if (n == 1)
         return 1;
      return Calculate (n - 1) + Calculate (n - 2);
   }



this is what Visual studion generate in release mode

Code: Select all
?Calculate@@YAHH@Z PROC               ; Calculate, COMDAT
   push   esi
   push   edi
   mov   edi, DWORD PTR _n$[esp+4]
   lea   eax, DWORD PTR [edi-1]
   cmp   eax, 1
   jne   SHORT $LN14@Calculate
   mov   esi, eax
   jmp   SHORT $LN9@Calculate
$LN14@Calculate:
   push   eax
   call   ?Calculate@@YAHH@Z         ; Calculate
   add   esp, 4
   mov   esi, eax
$LN9@Calculate:
   lea   eax, DWORD PTR [edi-2]
   cmp   eax, 1
   jne   SHORT $LN21@Calculate
   pop   edi
   add   eax, esi
   pop   esi
   ret   0
$LN21@Calculate:
   push   eax
   call   ?Calculate@@YAHH@Z         ; Calculate
   add   esp, 4
   pop   edi
   add   eax, esi
   pop   esi
   ret   0


This is wha the script compile generates,
Code: Select all
function _Calculate
   argument _n   ; passed in temporary t1
   reg0 = 0
   if (reg1 != 1) goto label0
   reg0 = 1
   goto exit
label0:
   push reg1
   reg1 = reg1 - 1
   call _Calculate
   reg2 = reg0
   pop reg1
   push reg1
   reg1 = reg1 - 2
   call _Calculate
   pop reg1
   reg0 = reg2 + reg0
exit:
   ret reg0


It use the same intrution sematic if an x86 cpy, and it use teh same number of register.
this mean the script code will run faste that a Visual teh same code compiled with visual studio in realse mode, if the script was converted to Just in time compiler instruction by isntrutions.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Re: Intruducing the Newton Scripting language.

Postby Stucuk » Thu Dec 20, 2012 2:59 am

Without a function's pointer being passed i don't see how the Script System can call a function in the application. If a class is passed then it would only be language independent if its an Interface(At least i know Delphi and C++ Interfaces are compatible).
User avatar
Stucuk
 
Posts: 801
Joined: Sat Mar 12, 2005 3:54 pm
Location: Scotland

Re: Intruducing the Newton Scripting language.

Postby Julio Jerez » Thu Dec 20, 2012 7:24 am

That's the beauty of the script an the reason why I am making it.
I revisited many script languages that are in existance corrently and in my opinion the way the integrate with languged is too complex,
it is the case for LUA and Phyton.

Seeing there is lot to be desired on these languages, I decided to make an Object Oriented script languge, that is similar to real Language like Java, and csharp but that do not require all
the infrastructure that you need for those language.
This script language is object oriented, but is not C++, the binging is achieved using the same classes methods and it is a build language feature.

I am close to have a demo, maybe them it will be clear to you what I mean. but beleive me ther is no need for funtion pointer, global variable, memory allocation.
yet thsi will be strong typed, much eassy that LUA an Phyton.

In the case of language liek Pascal, ther will be no C binding needed, teh script will generate the glue code in Pascal direcly, you just need to link that out header to you project, and the binding done.
Julio Jerez
Moderator
Moderator
 
Posts: 12249
Joined: Sun Sep 14, 2003 2:18 pm
Location: Los Angeles

Next

Return to General Discussion

Who is online

Users browsing this forum: No registered users and 9 guests

cron