I think i have currently the same problem another user had years ago when working on a shufflepuck / bowling kind of game.
I made a very basic character controller. Capsule with up vector constraint, driven by external force.
Now the capsule does not move precisely along the input of external force direction. Instead it's direction has a somewhat random offset (about 30 degrees at most). It behaves like the floor would have anisotropic friction properties (it hasn't), which tilts the movement direction.
This is how i imagine to fix it:
Afaik, on contact generation, Newton calculates a primary direction of friction orthogonal to the contact normal. Mostly aligned to relative velocity. It also calculates a secondary direction perpendicular to this.
Now i want to rotate these directions so they align with the players reference space of movement, which is the camera in my case.
After that, i hope movement follows the desired input direction accurately.
Previously i did such things in some callback, e.g. changing contact velocity to make conveyor belts.
Now i guess that's a matter of subclassing somethinjg. Maybe ndContactNotify? But i'm not sure...