So how can I fix this? Is there a demo I can look at to see what I should be assigning the filters to?
The CarDemo (through the SimpleCar), RagdollDemo, and TankDemo (through the Tank) all make use of SubgroupCollisionFilter.
The idea is that a body can belong to a group, and within the context of that group, there is a 'subgroup membership' mask and 'collides with' mask. If two bodies are in the same group and the 'membership mask' of one body overlaps with the 'collides with' mask of the other body, then they'll collide. They'll also collide if they are in different groups.
It's not intended to be a fully general solution to all collision filtering needs- the demos just use it to stop car wheels from colliding with the car body, or connected body parts in a ragdoll from colliding with each other and such. It's assumed that the user will create whatever custom filtering is needed for their own application, if any.
If you don't need that kind of functionality, you can just remove the filters and the relevant part of the callback that uses them.
That's really strange...so there's no way tornadoship should or could be occurring?
Under normal circumstances, it shouldn't happen, right.
If a constraint configuration is sufficiently difficult that the solver can't reach a satisfactory solution, surplus momentum can appear and
can cause things like tornadoship, but I couldn't get such behavior to happen in the repro with any reasonable configuration. That makes sense- usually those sorts of convergence failures require a significantly more complex (or ill-conditioned) constraint system, or extremely restricted solver time (very few iterations, very long timestep duration).
What would be a timestep duration of ideal length?
Infinitesimal!
Alas, computers aren't infinitely fast.
There's no ideal for all simulation, other than 'the shortest you can afford'. A rule of thumb is that you shouldn't give a constraint a frequency higher than about half your update rate. In other words, at 40hz, the frequency passed to SpringSettings shouldn't be higher than 20. Sometimes it's okay to go higher, especially if there are no feedback loops, and sometimes you have to go lower (very strong feedback loops). The frequency of a constraint is how fast it will try to oscillate in isolation, so frequencies that are high relative to the update rate can become impossible to represent by the integrator and things have a tendency to become unstable. (academic sidenote: implicit integrators can handle higher stiffnesses without exploding, although they do so by effectively damping away all the unrepresentable motion.)
If I were to use substepping, how would that fit into the whole rewinding part? Would there be extra stuff I'd need to set up & watch out for, or would it work as is?
Almost no changes- if you wanted to use the brute force 'call timestep more often, with shorter timestep duration', you can just replace the single call with multiple and you're done. Similarly, for substepping, you just call Create with the appropriate parameters and you're done. Don't have to step other game logic more frequently or anything.
The only subtlety is that any custom logic you want to run on a per-physics-step or per-substep basis would need to be scooted around accordingly. The timesteppers expose callbacks for that. For stuff like buoyancy, you can almost certainly ignore this and just apply it at the external tick frequency.
This really concerns me...the only (intentional) physics differences between the main project's and the repro's ships is that the main project's ship isn't a box, isn't affected by gravity, and has my buoyancy code on it (which handles gravity).
As always, if you can reproduce the behavior in a minimal repro, I can take a look and probably diagnose it immediately.