Why did you choose BEPUutilities over XNA's math? Is there a performance difference?
They should be basically identical, even down to the assembly level in many cases (there are not many ways to code Vector3.Add). The one exception is on WP7 Mango. XNA math has an advantage there due to XNA-only SIMD/ARM-NEON optimizations.
BEPUutilities was chosen because it will run anywhere. For example, it's possible to dump the BEPUphysics assemblies on linux and things just work (provided a compatible version of Mono is installed).
[BEPUutilities is used instead of other open source managed math libraries like OpenTK's since I had been incrementally building the math library over time for other purposes. By the time I decided to move to a fully dependency free version, my own library was already ready.]
My graphics/the rest of my engine uses XNA's math class, is there a reasonable way to mesh BEPUutilities in with this? For example I simply swapped out the previous version of Bepu with your new one, I got error
One way is to use something like the MathConverter class. This is what the BEPUphysicsDemos do.
Another common option is to put
implicit casts in the BEPUutilities math types which transparently convert between the XNA and BEPUutilities types. Those casts aren't in there by default since they would make the BEPUutilities library depend on XNA. If you're only targeting XNA, though, there's no problem with tossing them in.
There is always the option of replacing all the BEPUutilities references with XNA references, too. It's not hard, but it is tedious enough to discourage me from doing it anymore
Does BEPUutilities have undefined/quirky behavior, would it be reasonable to just use BEPUutilities for my graphics/outside Bepu?
It behaves pretty much exactly like XNA. If I remember correctly, the largest difference would be the absence of the BoundingFrustum struct.
I personally use BEPUutilities math for everything in my projects. The core of SharpDX has no dependencies on SharpDX math that I'm aware of, so it is pretty easy. Vertex buffers and everything else are just built from BEPUutilities types.
XNA (and perhaps the higher-level SharpDX toolkit, I haven't used it) can get in the way a bit more. For example, whenever you interact with SpriteBatch or BasicEffects, it will expect XNA types and a conversion will be necessary.