BEPUphysics

Grab a profiler and measure, measure, measure- guessing without some guiding data is, at best, a waste of time.

Despite that, I'll make one guess- if the space is truly getting filled with tons and tons of static objects, the broad phase could be taking longer. The broad phase doesn't know that it doesn't have to test separate static objects against each other until it traverses the acceleration structure and evaluates collision rules. For many thousands of static objects, this can eat up some time. If this is the cause and the objects must be in the space, bundling the static objects into fewer, larger objects would be good if possible. The StaticGroup is good for combining lots of static objects (like StaticMeshes, InstancedMeshes, Terrains, and kinematic entity collidables) together. If the objects don't need to be in the space (e.g. they are extremely far away), then removing them would be a good idea. In order for this guess to be correct, profiling results must show a disproportionate amount of time spent in the broad phase update (this is a necessary, but not individually sufficient, condition).

I knew this had to be my fault. I was passing in some uninitialized vertices, so there were masses of them at 0, 0, 0 causing the broadphase to slow down a lot.

Now that that's fixed, I was able to fly around for a bit with the new code and the stuttering has decreased a lot. I'm still getting some lag spikes from the GC but I still have a bit more work to do in reducing the garbage created. So, you were right about it being a GC issue. Thanks for all your help with this.