Introducing Newton Scripting language

[Julio Jerez]

Her are few thing that will never made it people has you actitude.

-When Microsoft made Visual studio, there were already several compiler with ID (Borland, Watcom, Metrowerk)
and now with have xcode. I am guessing that tyeh Linux workld follow you confromality and the say
we do no need an ID there are already too many

-Before Google with have Yahoo, and peopel still make new browse that do the exact same thing.
-Before Safary with had FireFox
-Before FireFox with had Internet Explorer
-Before Internet explorer with had NetScape,
-Before Direct3d we already had OpenGL
-Before Linux, there was already Unix and windows.
-When I made Newton, there were already Havok and Math Engine and Vortex,
and after I made Newton there are many people hwo made newer engines.
C# would not be made because we already had Java. and even after that some one made Mono has zero difference with C#.
The List goes on and on, but I believe those are good examples.

I am making a Scripting Language that is based on Java, you do not have to used it, but that is not going to stop me.

BTW this will not run on Java virtual Machine, the java virtual Machine is not partially Very Good.
this scripting language will compiler Java source code, but it will not Generate Java Byte code.
This ascription Language is and optimizing compiler that will generate code competitive that a C++ optimizing compiler.

also the code will be ready for Just in time compilation in one pass on any CPU, but it will have its own Virtual Machine.

[JoeJ]

Is scripting ready to use?
I'd like to do live coding, f. ex. tweak ragdoll controller code at runtime.
Stupid trial and error guys like me tend to waste a lot of time recompiling code.

[Julio Jerez]

no yet. I will go another round as soon as I get the soft body to stable state.

[Julio Jerez]

I put some work on the scripting language, I am now using LLVM for the high level intemedia optimization and here ius the result of a simple function

Code: Select all

// small c function
int Fibonacci (int n)
{
    return Fibonacci (n - 1) + Fibonacci (n - 2);
}

// my script compile generate not agresive optimizations at all
function int _Fibonacci::_Fibonacci::int
entryPoint:
   argument int reg1
   int reg2 = reg1
label_0:
   int reg1 = reg2 - 1
   int reg0 = call _Fibonacci::_Fibonacci::int
        int reg1 = reg2 - 2
        int reg2 = reg0
   int reg0 = call _Fibonacci::_Fibonacci::int
   int reg0 = reg0 + reg2
   ret int reg0

// Visual studio Best shoot
?Fibonacci@@YAHH@Z PROC               ; Fibonacci, COMDAT
; Line 7
   push   ebp
   mov   ebp, esp
   push   esi
; Line 8
   mov   esi, DWORD PTR _n$[ebp]
   lea   eax, DWORD PTR [esi-2]
   push   edi
   push   eax
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci
   dec   esi
   push   esi
   mov   edi, eax
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci
   add   esp, 8
   add   eax, edi
   pop   edi
   pop   esi
; Line 9
   pop   ebp
   ret   0
?Fibonacci@@YAHH@Z ENDP               ; Fibonacci

This is some serious optimization going on there, and this is no even using aggressive optimizations like tail recursion elimination or instruction reordering.
I seriously start to believe my function will outperform Visual studio even when running under the interpreter.

[Julio Jerez]

now this is monstrously scary

Code: Select all

int Fibonacci (int n)
{
    if (n == 0)
       return 0;
    if (n == 1)
       return 1;
    return Fibonacci (n - 1) + Fibonacci (n - 2);
}


Code: Select all

// visual studio 2010 64 bit, no stack frame ox optimizations
?Fibonacci@@YAHH@Z PROC                              ; Fibonacci, COMDAT
; Line 7
$LN25:
       push   rbx
       sub    rsp, 32                                 ; 00000020H
       mov    ebx, ecx
; Line 10
       cmp    ecx, 1
       jne    SHORT $LN1@Fibonacci
; Line 11
       mov    eax, ecx
; Line 13
       add    rsp, 32                                 ; 00000020H
       pop    rbx
       ret    0
$LN1@Fibonacci:
       mov    QWORD PTR [rsp+48], rsi
       mov    QWORD PTR [rsp+56], rdi
; Line 12
       xor    edi, edi
       dec    ecx
       jne    SHORT $LN16@Fibonacci
       mov    esi, edi
       jmp    SHORT $LN11@Fibonacci
$LN16@Fibonacci:
       call   ?Fibonacci@@YAHH@Z               ; Fibonacci
       mov    esi, eax
$LN11@Fibonacci:
       lea    ecx, DWORD PTR [rbx-2]
       test   ecx, ecx
       je     SHORT $LN19@Fibonacci
       call   ?Fibonacci@@YAHH@Z               ; Fibonacci
       mov    edi, eax
$LN19@Fibonacci:
       lea    eax, DWORD PTR [rdi+rsi]
       mov    rdi, QWORD PTR [rsp+56]
       mov    rsi, QWORD PTR [rsp+48]
; Line 13
       add    rsp, 32                                 ; 00000020H
       pop    rbx
       ret    0

Code: Select all

// newton script, basics optimizations
function int _Fibonacci::_Fibonacci::int
        push reg16
        push reg17
        int reg17 = reg0
        int reg0 = reg17 == 0
        if (reg0) goto label_1 else goto label_2
label_1:
        int reg0 = 0
        goto label_5
label_2:
        int reg0 = reg17 == 1
        if (reg0) goto label_3 else goto label_4
label_3:
        int reg0 = 1
        goto label_5
label_4:
        int reg0 = reg17 - 1
        int reg0 = call _Fibonacci::_Fibonacci::int
        int reg16 = reg0
        int reg0 = reg17 - 2
        int reg0 = call _Fibonacci::_Fibonacci::int
        int reg0 = reg16 + reg0
label_5:
        pop reg17
        pop reg16
        ret int reg0

[/code]

[JoeJ]

Here's what i get from vs 2013 (32bit, don't know hat optimizations i have on)

Code: Select all

;   COMDAT ?Fibonacci@@YAHH@Z
_TEXT   SEGMENT
?Fibonacci@@YAHH@Z PROC               ; Fibonacci, COMDAT
; _n$ = ecx

; 599  : {

   push   esi
   mov   esi, ecx

; 602  :     if (n == 1)

   cmp   esi, 1
   jne   SHORT $LN1@Fibonacci

; 603  :        return 1;

   mov   eax, ecx
   pop   esi

; 605  : }

   ret   0
$LN1@Fibonacci:

; 604  :     return Fibonacci (n - 1) + Fibonacci (n - 2);

   lea   ecx, DWORD PTR [esi-1]
   push   edi

; 600  :     if (n == 0)

   test   ecx, ecx
   jne   SHORT $LN16@Fibonacci

; 601  :        return 0;

   xor   edi, edi
   jmp   SHORT $LN11@Fibonacci
$LN16@Fibonacci:
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci
   mov   edi, eax
$LN11@Fibonacci:

; 604  :     return Fibonacci (n - 1) + Fibonacci (n - 2);

   lea   ecx, DWORD PTR [esi-2]

; 600  :     if (n == 0)

   test   ecx, ecx
   jne   SHORT $LN24@Fibonacci

; 601  :        return 0;

   xor   eax, eax

; 604  :     return Fibonacci (n - 1) + Fibonacci (n - 2);

   mov   eax, edi
   pop   edi
   pop   esi

; 605  : }

   ret   0
$LN24@Fibonacci:
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci

; 604  :     return Fibonacci (n - 1) + Fibonacci (n - 2);

   add   eax, edi
   pop   edi
   pop   esi

; 605  : }

   ret   0
?Fibonacci@@YAHH@Z ENDP               ; Fibonacci
_TEXT   ENDS


Better to read without the c source:

Code: Select all

; Function compile flags: /Ogtp
;   COMDAT ?Fibonacci@@YAHH@Z
_TEXT   SEGMENT
?Fibonacci@@YAHH@Z PROC               ; Fibonacci, COMDAT
; _n$ = ecx
; File c:\dev\pengii\pengii\engine\gitest.cpp
; Line 599
   push   esi
   mov   esi, ecx
; Line 602
   cmp   esi, 1
   jne   SHORT $LN1@Fibonacci
; Line 603
   mov   eax, ecx
   pop   esi
; Line 605
   ret   0
$LN1@Fibonacci:
; Line 604
   lea   ecx, DWORD PTR [esi-1]
   push   edi
; Line 600
   test   ecx, ecx
   jne   SHORT $LN16@Fibonacci
; Line 601
   xor   edi, edi
   jmp   SHORT $LN11@Fibonacci
$LN16@Fibonacci:
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci
   mov   edi, eax
$LN11@Fibonacci:
; Line 604
   lea   ecx, DWORD PTR [esi-2]
; Line 600
   test   ecx, ecx
   jne   SHORT $LN24@Fibonacci
; Line 601
   xor   eax, eax
; Line 604
   mov   eax, edi
   pop   edi
   pop   esi
; Line 605
   ret   0
$LN24@Fibonacci:
   call   ?Fibonacci@@YAHH@Z         ; Fibonacci
; Line 604
   add   eax, edi
   pop   edi
   pop   esi
; Line 605
   ret   0
?Fibonacci@@YAHH@Z ENDP               ; Fibonacci
_TEXT   ENDS


[Julio Jerez]

is scary isn't it. I start think that it may be possible that the script run in interpreted mode at a speed comparable to what Visual studio can produce,
and if compiled it may beat Visual studio.

This weekend I will have the first example running on the virtual machine.

[JoeJ]

Somehow i hope you're wrong
But if not you should think of an own name and standalone project for the script language - people may think it's related to physics and not paractical for their needs.

I'd like to see what gcc produces...

[Julio Jerez]

Oh I would no pretend to be faster than visual studio or GCC, that will be crazy.
I do pretend to be in the ball park of byte code language like Java, C# or MONO
there is a website that has some bench mark comparison between languages for different algorithm
http://benchmarksgame.alioth.debian.org ... test=nbody

you can see that Java, C#, Haskel, Ada, and some other are within 2 to 3 time slower that C++ or Fortran.

Then you can see that all the scrip languages, LUA, Python, Perl, Ruby are on the order of hundred of times slower than C/C++

then you have scripting language like JavaScrip and Lisp which are on the order of 4 to 5 time slower,
My expect scrip as a minimum has to be better than those last tow when running on byte code,
and better than Java and Mono when running in binary machine code.

[Julio Jerez]

I finally got around to work some more on the scrip system. and now I can generate some cod etha can be exacted by the virtual machine.

here is a fully working fibnacci test compiel by the script with epilog and prolog

Code: Select all

function int _Fibonacci::_Fibonacci::int (int _n)
   enter 3, 0
   int r17 = int r1
   int r0 = int r17 == constInt 0
   ifnot (int r0) goto label_2
   exit 3, 0
   ret constInt 0
label_2:
   int r0 = int r17 == constInt 1
   ifnot (int r0) goto label_4
   exit 3, 0
   ret constInt 1
label_4:
   int r1 = int r17 - constInt 1
   int r0 = call _Fibonacci::_Fibonacci::int (int r1)
   int r16 = int r0
   int r1 = int r17 - constInt 2
   int r0 = call _Fibonacci::_Fibonacci::int (int r1)
   int r0 = int r16 + int r0
   exit 3, 0
   ret int r0


The code is almost identical to what Visual studio can generates.
Now I can write the code emitter and try the virtual machine

[Julio Jerez]

hey guys, I was working on a the script compiler a little and I decided that instead of writting my own I rather use and existing script like Lua.
I found that Lua code generator is quite bad, even LuaJit is horrendous, I can do far, far better that that. In fact I can do almost better that Microsoft Visual studio, and I only have few optimizing transform passes.
Speacking of visual studio, do you guy remember when I complained how medicry Visual studio code generator was? I found this:
https://blogs.msdn.microsoft.com/vcblog/2016/05/04/new-code-optimizer/

finally they admitted that they were using a 50 years old technology to produce code for what they said 22 century state of the art, it took Microsofts 40 years to realize they need a better back end. This is what they said:

The main motivation for a new optimizer framework was the desire to have more aggressive optimizations, such as ones that take advantage of more compile-time information and modern compiler developments. The design of some of the older optimization passes made it difficult to implement more advanced transformations and to make improvements at a faster pace. As the new framework was intended to be the basis of many future optimization efforts, a core design objective was to make it easier to implement, test and measure new optimizations.

-The old expression optimizer has a small set of known transformations and a limited view of the function
– this prevents discovering all the expressions that could be optimized.
Many small optimizations based on identifying patterns
– known as peephole optimizations – are either missing or implemented only for certain target architectures.
Vector code – either from intrinsics or generated by the auto-vectorizer – can be optimized better.

in other words we any one who though was using optimized code by getting a new Visual station was taken for a fool.
This is why I was thinning to write my own intrinsic language converter to compete newton, because when I see the code generate by visual studio, I want to cry. I can assure anyone I can do far better that that.

This is what large corporations like Nvidia, Intel and Microsoft do to their costumers, Nvidia been the worse liar of the bunch.
If you say anything, what you get the common fallacy of an army of fan boys siding with the popular option just because is backed by a company.
The part that is more disgusting is that when these people adopt these method they act as if the invented it.

The moral of this is that is that there is absolutely no reason to get visual studio 2017.
if the pattern is the same as in the pass, it will take Visual studio two or three generations to get a clean production product.
I almost guarantee that VS 2017 will be slower than VS 2015 and orders and full of bugs that will not be fixed until maybe 2020 with VS 2019 or VS 2020.

[zak]

The first version of new SSA optimizer is already present as default in Visual C++ 2015 update 3.
It can be disabled with "/d2SSAOptimizer-" compiler switch.
SSA optimizer is default in Visual C++ 2017 too, but why not use VC2017 for this?
We can use VC2017 ( VC2019-2020 ecc. ) with SSA optimizer disabled, but can be interesting to enable it from time to time to test performance, and utilize it when we think it is completely reliable.

[Julio Jerez]

oh do not get me wrong, Single static assignment (SSA) is the most advanced compiler technique for writing optimizing compiler.
This was technique was invented by Mark N. Wegman, a PHD researched at IBM in the 1989.
This was a published since the early 90's and it had been use in optimizing compiler like Wacom, that run circles around visual studio for year an years.
SSA make many algorithm simpler, but is also open to the opportunity for new transformation passes that are not possible with out it. The Net is full of PHD papers that are all twist on some standard pass that can get some specialized transformation of the code.
So if you are making a compiler of any kind, even a script compiler, you better use SSA

That is not the problem I was referring too, I was referring to the fact that for almost 20 years Microsoft has been lien claiming that each new version of visual studio was better that the previous and that was simple a lie. All they did was at best bug fixes, adding features to the editor, and adding support for new CPU features but they never changed the code generator.

I have been buying visual studio professional since VS 6 (in 1998)
and I rarely see a better code generator from a newer version. In fact I have seen plenty severed of regrations.
right now I have VS2010, 2013 an 2015 in my system and I do not see any difference among then.
In fact I use 2013 because I despise what the done to the editor in 2015, when the auto complete does not let you work, since it insist is reformatting the code the way they want not the way I want.

so on the SSA stuff, sure it sore be better, but if I am going to pay 600 for a new version I do no was to be a Beta, (in the case for MS maybe even alpha) tester.

So far I know SSA is not a patented technology, so why did the wait 20 years to move to it?
I am skipping vs 2017 just like IU did with VS 2012 and the reason is that
they have an optimized that is highly tweaked, using a trick called peephole optimization.

one of the things SSA is good for is for general CPU impendent optimizations, peephole is not a generic optimization, in fact peehole is considered a hack among software engineers.

but is you read the blog all they are talking about is porting their pile a peephole trick to the new SSA based code generator, that are not talking about the other general optimization.
so I do not expect any of these version to be any better that the previous for at least five years.

[Julio Jerez]

here is an example of what I say, form stack overflow

accepted
Yes, it's a bug. Specifically, it's a bug in the new SSA optimizer introduced in VS2015 Update 3. The undocumented command line option -d2SSAOptimizer- tells the compiler backend to use the old optimizer instead, which causes the bug to not manifest.
FYI, you can minimize your repro to:

Code: Select all

int main()
{
    volatile int someVar = 1;

    const int indexOffset = someVar ? 0 : 1;

    for (int i = 1 - indexOffset; i < 2 - indexOffset; ++i)
    {
        return 0;
    }
    return 1;
}

which will help the compiler developers localize the problem more quickly.

Addition from Codeguard (I decided that Casey's answer should be THE answer): I have received reply from Microsoft (Gratian Lup, author of blog post Introducing a new, advanced Visual C++ code optimizer):

Yes, this is indeed a bug in the SSA Optimizer itself - usually most bugs reported as being in the new optimizer are in other parts, sometimes exposed now after 20 years.
It's in a small opt. that tries to remove a comparison looking like (a - Const1) CMP (a - Const2), if there is no overflow. The issue is that your code has (1 - indexOffset) CMP (2 - indexOffset) and subtraction is not commutative, of course - but the optimizer code disregards that and handles (1 - indexOffset) as if it's (indexOffset - 1).
A fix for this issue will be released in the next larger update for VS2017. Until then, disabling the SSA Optimizer would be a decent workaround. Disabling optimizations for only this function may be a better approach if it doesn't slow down things too much. This can be done with #pragma optimize("", off): https://msdn.microsoft.com/en-us/library/chh3fb0k.aspx

if you have a compiler bug is somethinglike:

Code: Select all

const int indexOffset = someVar ? 0 : 1;

be ready for the next 5 years it will be a bumpy ride. It tell you.

[zak]

be ready for the next 5 years it will be a bumpy ride. It tell you.

so on the SSA stuff, sure it sore be better, but if I am going to pay 600 for a new version I do no was to be a Beta, (in the case for MS maybe even alpha) tester.

Oh, that's right.
I was referring to the Community Edition.
Community Edition has all the features of Professional including optimizations, in fact Community Edition is Professional edition (except CodeLens)
The only difference is the license that is very permissive also for commercial products (up to 250 PCs or a maximum of one million sales)
That is, it is good to develop Newton and for most of us who use Newton (I would like to have revenue for more than a million, I'd like to pay for the professional license)
As for the VS2015 intrusion as it comes to code, ide is highly customizable in this regard, with so many options (maybe too many) such as indentation, tabs, and more. Once set, you should type the code according to your style.

Ultimately using the Community Edition you can not spend money without losing any function, you can take advantage of new versions of VS, you can test opimizations with "/d2SSAOptimizer-" compiler switch and provide VS2017 (VS2019-VS2020 etc.) projects that many Newton users use.
You could also continue to use VS2013 to write code, and use VS2017 or later to compare optimizations and deliver projects.