NewtonCollisionSerialize and cross-platform development.

[urkle]

I am working on the Linux and Mac ports of the upcoming Amnesia and we are using NewtonCollisionSerialize to serialize the newton data into cache files that will be distributed with the game.. However I am finding that the data generated by newton to the callback is not the same between architectures.. (ie.. PPC and x86).. I will be working on a linux 64bit build shortly and fear the issue will also exist between 32bit and 64bit.. From a quick look at the data it *seems* to be just 32bit data in the native Endianess. So I could swap it on PPC and "hope" it works. but if the data is size-wise different between 32 and 64-bit then it's no good..

Our goal is to include the cached data in the game install so the users would not have to wait for everything to cache on game install. Any help/thoughts on this subject will be very welcome.

[Julio Jerez]

From a quick look at the data it *seems* to be just 32bit data in the native Endianess

this is right, all data is 32 bit atoms in the native Endianess of the system.
there data is the same for 32 and 64 bits.
you can read a serialize file produced in a 32 bit system, in a 64 bit system without problems.

the indianess is a problem but for big indian like I macs you can you can just do an indian swapping every 4 bytes in the call back and it will be fine.
sicne teh number of byte on each call to teh call back is a mutipel of 4 bytes always.

[urkle]

I actually tried doing an endian swap in the call back on every 4 bytes and it didn't actually work. Some of the data did get swapped to look correctly, however other did not. and I put a logger to log how much data in each call back was provided and there are different amounts provided per architecture and it looks like the blocks are provided in different orders.

I did this by running a universal binary as native x86 and then using "arch -ppc".

Now we are using Newton 2.08 as newer apparently caused issues.

[Julio Jerez]

it is happing on all serialized shapes?
can you serialize one shape on a PPC, and post it so that I can load it in the PC and see what is wrong?

what issues you have with the teh newer version of Newton?

to do this you may have to upgrade to the latest version that was avalible for Mac, I think is 2.18
there should not be any errors in teh upgrade.

[urkle]

I'm testing further right now.. What I found was that evertime I ran the cache generation script the objects got put in different orders in the output file.. (the one I am testing has 11 objects being serialized). Now when comparing two of the SAME object they do in fact seem to be the same (aside from the byte order) So I added back in my swapping and for the few that I tested (where the they were for the same objects) they seem to be exactly the same.

There are a few that have different sizes between the PPC and x86 generation of the serialized data. by around 100 or so bytes.

I'll have to get with the lead developer as to what the issue was. What he documented in the svn log was "09 had straaange joint problems" so we reverted back to 08. As we are prepping for a production release in a month.. I'm not sure switching versions now is going to happen.. But it looks like the swapping may work. And if the 64bit also serializes as 32bit then I should be "all good" on linux too.

[urkle]

I've attached an extraction of one of the newton serialized objects that is different sizes between ppc and x86.

And looking at the objects.. ONLY the once material ends up different sizes. the rest are exactly the same. odd. And even ends up different on repeat runs on the same architecture..


Well.. the next test is to try the x86 built data and have the ppc load it and see if it goes "boom"

[Julio Jerez]

a serilized data can be diffrent in a PCC, PPC, Linux, or Max86
it all depend on teh code teh CPU generates. The importan thong is that they are compatible.

also if you can read the PPC on the PC, I beleive the oppssite is fixed.

I will try this tomorrow morning, it is late now, and I am tires.

is this a PPC generated file?
Oh never mind it has both.

what shape and I supposed to see in those files?

[Julio Jerez]

Ok I tried to open the tow file, and I when over the code,
There is a compatibility problem between 2.08 and 2.24 so the data cannot be read correctly
I did read the make file, and the x86 file in the PC in both 32 and 64 bit data, and there is not alignment bug.
This is what I did , I made the Mac Build for 2.24 and I will make the Linux too,
You will have to upgrade to 2.24 and re save your serialize data, the upgrade should be painless.
The doing it again and see if it work,
Just to make sure this is how you need to write you serialize/deserialize callbacks

Code: Select all

// for litle indians,
static void SerializeFile (void* serializeHandle, const void* buffer, int size)
{
   _ASSERTE ((size % 4) == 0);
   fwrite (buffer, size, 1, (FILE*) serializeHandle);
}

static void DeSerializeFile (void* serializeHandle, void* buffer, int size)
{
   _ASSERTE ((size % 4) == 0);
   fread (buffer, size, 1, (FILE*) serializeHandle);
}


// for big indian serialization
static void SerializeFile (void* serializeHandle, const void* buffer, int size)
{
   _ASSERTE ((size % 4) == 0);
   const int* ptr = (int*)buffer;
   for (int i = 0; i < int (size/sizeof (int)); i ++) {
      int data = SWAP_INT32(ptr[i]);
      fwrite (&data, sizeof (int), 1, (FILE*) serializeHandle);
   }
}

static void DeSerializeFile (void* serializeHandle, void* buffer, int size)
{
   _ASSERTE ((size % 4) == 0);
   fread (buffer, size, 1, (FILE*) serializeHandle);
   SWAP_FLOAT32_ARRAY (buffer, size);
}

This will make file that are independent of the platform, therefore you can read then in Linux, Mac or PC.
I am a believer that data should be independent of the Platform.

Here are the archive for pc and Mac 2.24

http://www.newtondynamics.com/downloads/NewtonWin-2.24.rar
http://www.newtondynamics.com/downloads/NewtonMac-2.24.zip

Unfortunately I am not in the position to produce file for testing because I am in the process of changing the demo to use the Native GUI of the OS in every platform.
After I decided to use WxWidget and Collada, the SDK grew by about 80 megabytes compresses, and the demo in teh SDK are too complex with all for the average Newton User.

As result must people are no updating to the latest version and stop at the last 2.08 which does not use so many this party external libraries.
I am changing all that so that the only third party libraries are libraries that are part of the SDK, like tinysml, and glew, bu every thong else will be native and minimal.
For that I am making a newton file format, and an editor, and because of that some demos do no work.
However the newto libraries do work.
Please upgrade to 2.24 and generate the file again, and see if it works.

[Carli]

Maybe you could split the SDK from the demos.

You could make simple make files for the demos as an extra download platform independend and the SDK just with the DLL (and maybe some language specific headers.)

[urkle]

Julio, thanks for you quick replies. I have tested the 2.08 doing the byte swapping and it failed to work across x86 and PPC. And since we are REALLY close to game release (end of the month), the lead developer does not feel comfortable switching a very well tested version (in our game) at this point in time. So I have implemented a workaround (tagging the arch in the cache files to force rebuild/ignoring). However, once the release is out and we have more time available we will begin looking into the latest Newton 2 in hopes that it does indeed provide the platform agnostic cache we so desire.

Another issue I found when testing this cross-arch cache loading is that there is no way to tell the NewtonSerialize callback that "uh.. I got no more data.. Bail please". Is that something that can be added in in a future SDK release? As what happens now is the PPC arch will loop endlessly in newton's serialize reader trying to read and since the callback can not say "hey I've passed the end of the file" it just keeps going..

[Julio Jerez]

yes of coruse chnage the phsics is can be a big problem at that stage.

Another issue I found when testing this cross-arch cache loading is that there is no way to tell the NewtonSerialize

It does, if you open the file in and ascci editor you will see that teh lat workld is end\0
maybe it is a bug in eth PPC arch.

well you you have time then we can check the serialized data. it would be nice to make sure, that funtionality world.
tell me did you write yuor serialize funtion for PC and PPC as I posted? I will verify myself next time I am in the PC by savin a fiel in the ppc, and loading on teh pc.

I will make teh call back save in lithe indian in teh demos, so tah teh bin fiel isn platform independet.