As I'm proceeding in the OgreNewt4 wrapper and I’ve run into a reproducible assert in the ray casting code against a ndShapeStatic_bvh.
Rays: basic vertical rays from player controller (downwards ray to the ground, plus some forward rays)
Integration: OgreNewt4 wrapper calling ndWorld::RayCast(...) with a closest-hit callback
In debug, I get this assertion inside ndShapeStatic_bvh::RayCast:
- Code: Select all
ndFloat32 ndShapeStatic_bvh::RayCast(ndRayCastNotify& callback,
const ndVector& localP0,
const ndVector& localP1,
ndFloat32 maxT,
const ndBody* const body,
ndContactPoint& contactOut) const
{
ndBvhRay ray(localP0, localP1);
ray.m_t = ndFloat32(1.0f);
ray.m_me = this;
ray.m_myBody = ((ndBody*)body)->GetAsBodyKinematic();
ray.m_callback = &callback;
ndFloat32 t = ndFloat32 (1.2f);
ForAllSectorsRayHit(ray, maxT, RayHit, &ray);
if (ray.m_t < maxT)
{
t = ray.m_t;
ndAssert(ray.m_normal.m_w == ndFloat32(0.0f)); // <- asserts here
ndAssert(ray.m_normal.DotProduct(ray.m_normal).GetScalar() > ndFloat32(0.0f));
contactOut.m_normal = ray.m_normal.Normalize();
contactOut.m_shapeId0 = ray.m_id;
contactOut.m_shapeId1 = ray.m_id;
}
return t;
}
After ForAllSectorsRayHit, the ray.m_normal looks like this in the debugger:
- Code: Select all
ray.m_normal = {
x ≈ 8.5e-12,
y ≈ 9.1e-43,
z ≈ 8.5e-12,
w ≈ 9.1e-43
}
So xyz are finite and small, but w is non-zero garbage (looks like an uninitialized stack float). Because of the debug assert that m_w must be exactly 0, this hits every time the ray actually hits the mesh.
If I comment out the assert, the raycast behaves correctly (the hit normal passed into my callback is fine in xyz, and gameplay works), so this is a debug-only cleanliness issue.
How I call the raycast
From my OgreNewt4 wrapper:
- Code: Select all
void Raycast::go(const World* world,
const Ogre::Vector3& startpt,
const Ogre::Vector3& endpt,
int /*threadIndex*/)
{
mLastBody = nullptr;
mBodyAdded = false;
mStart = startpt;
mEnd = endpt;
mCallback.m_param = ndFloat32(1.0f);
mCallback.m_contact = ndContactPoint();
ndWorld* const ndworld = world->getNewtonWorld();
const ndVector p0(startpt.x, startpt.y, startpt.z, ndFloat32(1.0f));
const ndVector p1(endpt.x, endpt.y, endpt.z, ndFloat32(1.0f));
ndworld->RayCast(mCallback, p0, p1);
}
Its a common scenario in my NOWA-Engine and nothing has changed and in OgreNewt3 (ND3) it did work.
And in ndScene::RayCast Newton masks the w-component anyway:
- Code: Select all
bool ndScene::RayCast(ndRayCastNotify& callback,
const ndVector& globalOrigin,
const ndVector& globalDest) const
{
const ndVector p0(globalOrigin & ndVector::m_triplexMask);
const ndVector p1(globalDest & ndVector::m_triplexMask);
...
}
So from my side, xyz are valid and w is don’t-care; you already zero it inside ndScene::RayCast.
The player ray in this particular case is something like:
- Code: Select all
start: (-2.5917597, 0.29416525, -8.9903698)
end : (-2.5917597, -499.70584, -8.9903698)
The scene has a ndShapeStatic_bvh built from a triangle soup; degenerate triangles are already filtered out on my side (I skip faces with any edge² < 1e-12 and slightly extrude almost-flat faces), so this assert is not related to the triangle soup degeneracy I fixed earlier.
Analysis
Looking at the behavior:
ray.m_t is less than maxT so the ray did hit something.
ray.m_normal has valid xyz, but m_w is never explicitly set to 0 anywhere in the callback path.
The only place that writes ray.m_normal is the internal callback passed to ForAllSectorsRayHit, which is ndShapeStatic_bvh::RayHit(...).
From that, it looks like RayHit sets the normal’s xyz but leaves w uninitialized, and then ndShapeStatic_bvh::RayCast asserts that w is exactly 0. That’s what I’m observing in the debugger: x,z ≈ 8.5e-12, but w is tiny non-zero stack junk like 9.136e-43.
In release builds this doesn’t matter, since no code uses w and everything works fine.
Proposed minimal fix
In ndShapeStatic_bvh::RayHit(...) (wherever you finalize the hit and assign ray.m_normal), you can force w = 0 before storing it:
- Code: Select all
ndFloat32 ndShapeStatic_bvh::RayHit(
void* context,
const ndFloat32* vertex,
ndInt32 strideInBytes,
const ndInt32* indexArray,
ndInt32 indexCount)
{
ndBvhRay& ray = *(ndBvhRay*)context;
// ... compute face normal as ndVector normal (xyz valid, w arbitrary) ...
if (t < ray.m_t)
{
ray.m_t = t;
// Ensure w = 0 to satisfy RayCast debug assert
normal = normal & ndVector::m_triplexMask; // (1,1,1,0)
ray.m_normal = normal;
...
}
return t;
}
or equivalently:
- Code: Select all
ray.m_normal = normal & ndVector::m_triplexMask;
And in another place the same issue:
- Code: Select all
ndFloat32 ndShapeStatic_bvh::RayCast(ndRayCastNotify& callback, const ndVector& localP0, const ndVector& localP1, ndFloat32 maxT, const ndBody* const body, ndContactPoint& contactOut) const
{
ndBvhRay ray(localP0, localP1);
ray.m_t = ndFloat32(1.0f);
ray.m_me = this;
ray.m_myBody = ((ndBody*)body)->GetAsBodyKinematic();
ray.m_callback = &callback;
ndFloat32 t = ndFloat32 (1.2f);
ForAllSectorsRayHit(ray, maxT, RayHit, &ray);
if (ray.m_t < maxT)
{
t = ray.m_t;
ndAssert(ray.m_normal.m_w == ndFloat32(0.0f));
ndAssert(ray.m_normal.DotProduct(ray.m_normal).GetScalar() > ndFloat32(0.0f));
contactOut.m_normal = ray.m_normal.Normalize();
contactOut.m_shapeId0 = ray.m_id;
contactOut.m_shapeId1 = ray.m_id;
}
return t;
}
Same solution:
after the:
- Code: Select all
t = ray.m_t;
ray.m_normal = ray.m_normal & ndVector::m_triplexMask;
This keeps xyz exactly as computed, but guarantees ray.m_normal.m_w == 0.0f, so the assert in ndShapeStatic_bvh::RayCast becomes valid and never fires.
Why this is safe
My code only ever reads the xyz components of the normal (I convert it to an Ogre::Vector3), so modifying w has no behavioral impact.
Your ndScene::RayCast already uses m_triplexMask on positions, so treating w as “must be 0” is consistent with the rest of the codebase.
The bug only manifests in debug builds (due to the assert); release already works, which supports the idea that only w is not being initialized.
Best Regards
Lax
