"Rewinding" Simulations

Discuss any questions about BEPUphysics or problems encountered.
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

"Rewinding" Simulations

Post by tomweiland »

So I'm getting into the more complex parts of client-server setups now, and one thing I need to sort out is lag compensation.

Most multiplayer games will "roll back" their simulation to the point in time when the input was sent by the player and simulate from there, thereby making corrections to the previously simulated game state. In order to do this, the physics need to be resimulated too.

The new Unity Physics is designed with multiplayer use cases in mind, meaning it's relatively simple to rewind and resimulate.

I suspect this may be more difficult with Bepu, however Unity Physics is still very much in the early stages, so I'm not sure how viable it is to use currently.

This brings me to my question: is it possible to "roll back" Bepu (v2)?

I was thinking about it and it occurred to me that since I have to store the state of the game anyways, I could directly set positions/rotations/velocities of bodies at the point in time I need to go back to...would this be a "good" way to approach the problem?
I remember reading something about an option to make Bepu deterministic, so I'd have to look into that again, but is this whole concept achievable with Bepu or are there too many issues with setting something like this up?

In theory, as long as I properly set all bodies that can move, I should be able to restore the simulation to a previous state, right?

Of course there's still the issue of not being able to use Bepu in Unity, so I'm not entirely sure how I'll handle client prediction without access to the same physics...but that's a separate question.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

Simply caching historical pose/velocity should work fine. It's true that internal cached state will mean that the simulation won't produce bitwise identical results, but since the resimulation 1) is different due to user input anyway and 2) lasts a fraction of a second, determinism shouldn't be a practical concern.

(The Deterministic property also only guarantees local determinism given identical input sequences, so a resimulation that doesn't start from the same internal state won't produce the same results. It is possible in theory to extract and reapply the internal state to get around this, but the value in this use case is effectively zero.)
Of course there's still the issue of not being able to use Bepu in Unity, so I'm not entirely sure how I'll handle client prediction without access to the same physics...but that's a separate question.
Depending on the target platform, you could go supergrosstown and have a separate local process running bepu on top of CoreCLR on the client computer, communicating with near zero latency to the unity process :P

Honestly, I'm not sure if that would be more or less gross than some of the alternatives...
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

Norbo wrote: Wed Oct 30, 2019 8:47 pm Simply caching historical pose/velocity should work fine. It's true that internal cached state will mean that the simulation won't produce bitwise identical results, but since the resimulation 1) is different due to user input anyway and 2) lasts a fraction of a second, determinism shouldn't be a practical concern.
Awesome! However, not all user input will be different every time it's resimulated. In fact I think it's likely that usually only a few players' inputs will have changed by the time a resimulation is necessary, meaning it'd be best if the other user inputs which don't change produce the same results. My guess is you're probably correct about determinism not being key, but it'd still be nice.
Norbo wrote: Wed Oct 30, 2019 8:47 pm (The Deterministic property also only guarantees local determinism given identical input sequences, so a resimulation that doesn't start from the same internal state won't produce the same results. It is possible in theory to extract and reapply the internal state to get around this, but the value in this use case is effectively zero.)
So what you're saying is that given the same poses/velocities, there could still be differences in the internal state?
Norbo wrote: Wed Oct 30, 2019 8:47 pm Depending on the target platform, you could go supergrosstown and have a separate local process running bepu on top of CoreCLR on the client computer, communicating with near zero latency to the unity process :P

Honestly, I'm not sure if that would be more or less gross than some of the alternatives...
Sounds complicated :P
Ideally I'd like to keep my options regarding platforms open...are there any limitations to this approach?
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

So what you're saying is that given the same poses/velocities, there could still be differences in the internal state?
Yup. The engine incrementally updates a bunch of different datastructures in the broad phase, narrow phase, solver, bodies sets, and so on. Any difference can propagate to noticeable divergence given enough time and opportunity for chaos. Fortunately, 250ms isn't much time.
Ideally I'd like to keep my options regarding platforms open...are there any limitations to this approach?
If you're allowed to launch and communicate with an arbitrary process and the platform is supported by CoreCLR or another runtime that can handle bepuphysics v2, it'll work. So linux/mac/windows should be fine, IOS/android probably not.
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

So I've implemented states for my server now which get stored for half a second, allowing me to rewind time.

However, I'm having issues with applying forces—specifically my buoyancy code is causing issues, although I suspect I messed up somewhere and that's on my end.

However, when I turned it off and tried to move my player around, it behaved very similarly with the position shooting up to either (NaN, +/-Infinity, NaN) or (+/-Infinity, +/-Infinity, +/-Infinity). I'm using the built in character controller and it's constraints-based movement.

Positions/velocity should be getting set properly...but is there anything I need to watch out for with constraints when rewinding to a previous state?
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

Constraints do preserve impulses from the previous frame to warm start the next solve, but the expected failure mode there is just slightly worse accuracy. Calling Simulation.Solver.ScaleAccumulatedImpulses(0) would wipe out the cached guesses. I don't expect it to be related, but it's worth trying to see if anything changes.

Another option would be to significantly increase the Simulation.Solver.IterationCount- since the old impulses are just an initial guess, more iterations should compensate for a bad guess. Again, this shouldn't explain a full NaNsplosion, but it's something to rule out.

Collision detection runs before the solver, so any changes in constraints caused by the teleportation should be caught. The character would be updated too- though there isn't much state to worry about in the character constraints anyway.

As always, a reproduction in the demos would be helpful to narrow things down.
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

Norbo wrote: Tue Nov 05, 2019 10:26 pm Constraints do preserve impulses from the previous frame to warm start the next solve, but the expected failure mode there is just slightly worse accuracy. Calling Simulation.Solver.ScaleAccumulatedImpulses(0) would wipe out the cached guesses. I don't expect it to be related, but it's worth trying to see if anything changes.
I tried this and at first I thought it fixed it, but that was only initially.
Norbo wrote: Tue Nov 05, 2019 10:26 pm Another option would be to significantly increase the Simulation.Solver.IterationCount- since the old impulses are just an initial guess, more iterations should compensate for a bad guess. Again, this shouldn't explain a full NaNsplosion, but it's something to rule out.
How much is "significantly"? I set it to 1024 and that has stopped my objects flying up and down, however the client is still receiving positions of (NaN, +/-Infinity, NaN), and objects still "jitter" around. Also are there performance implications of changing this value dramatically?
Norbo wrote: Tue Nov 05, 2019 10:26 pm As always, a reproduction in the demos would be helpful to narrow things down.
If I get really stuck, then I might come back to this...it's not something that would be easy to reproduce since I'd have to replicate a lot of my server setup.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

How much is "significantly"? I set it to 1024
1024 qualifies as a significant increase :) The default is 8, and many simulations can get by with 2.
Also are there performance implications of changing this value dramatically?
Yes- the cost of the solver is proportional to the number of iterations. Setting it to an extreme value is only useful as a diagnostic.
that has stopped my objects flying up and down, however the client is still receiving positions of (NaN, +/-Infinity, NaN), and objects still "jitter" around.
It doesn't sound like an accumulated impulse problem then- the extreme iteration count may have allowed the solver to partially compensate for some nasty inputs, but the other behavior implies something else is going on.

Shot in the dark- is the timestep duration passed Simulation.Timestep kept constant?
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

Norbo wrote: Tue Nov 05, 2019 11:51 pm Shot in the dark- is the timestep duration passed Simulation.Timestep kept constant?
Yes, whether I'm resimulating a tick or generating a new one, I always pass in the same timestep (0.025, aka 40 ticks per second).
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

I just realized that my code's execution order is like this for each tick:
  1. Check if there are unprocessed inputs, if so, set the state to the time of the input and resimulate up to the present
  2. Physics timestep
  3. Apply impulses for flotation & set input for character constraints
Wouldn't this mean that if impulses are applied, and then the state is reset to a different time, those impulses would be applied "in the past" because they aren't reset along with stuff like pose/velocity? I suspect this is a major contributor to the problem, considering that my flotation code is making stuff fly infinitely high up into the sky.

Is there a way to clear all impulses that haven't been used yet, or should I just change the execution order so the physics timestep happens between impulses being applied and time potentially being rewound (which would result in no left over impulses, right)?
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

Applying an impulse immediately modifies the velocity, so there's no extra state to clear.
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

Norbo wrote: Wed Nov 06, 2019 9:44 pm Applying an impulse immediately modifies the velocity, so there's no extra state to clear.
Hmm I see.

So what about the fact that I'm using a constant timestep? Is that good or potentially problematic?
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

Constant timesteps are good for stability. Just wanted to rule out something like accidentally taking a negative timestep or other potentially catastrophic issues.

Unfortunately I can't think of any other likely causes. Might be time to start pruning stuff out bit by bit, working toward a stripped reproduction case (or finding more hints along the way). Ideally the issue could be reproduced without any kind of client/server architecture- just a local set of simulation interactions which cause a failure.
tomweiland
Posts: 99
Joined: Wed May 08, 2019 12:17 am

Re: "Rewinding" Simulations

Post by tomweiland »

So I have no idea if this will help, but I've found something very strange.

I added a basic cube with my flotation code to the simulation, so it has none of the other player-related stuff, and integrated it with my state system. What I don't understand is that cube falls to the ground (meaning it's not spawning inside anything else) and sits there. When it's underwater, buoyancy impulses are applied, and the velocity gets bigger and bigger, but it doesn't move.

Surely a velocity of 350,000m/s should be able move a 0.1kg cube upwards against the measly effect of gravity (which I've set to 9.8m/s/s)...
I'm not sure if it's related to the actual issues I'm having, but this is certainly very strange.

PS: during the duration of me writing this post, the velocity has increased to 500,000m/s...

EDIT: I tried it a few more times, and this result doesn't seem to be consistent after all. Sometimes it doesn't move but the velocity grows, sometimes the Y components of both velocity and position shoot to NaN almost instantly, and other times the Y components of both velocity and position get really large (like in the several hundred million range).

This is a bit of a noob question, but does it make a difference whether I set the position/velocity like this:

Code: Select all

gameObject.collider.Pose.Position = value.pose.Position;
gameObject.collider.Pose.Orientation = value.pose.Orientation;

gameObject.collider.Velocity.Linear = value.velocity.Linear;
gameObject.collider.Velocity.Angular = value.velocity.Angular;
or like this:

Code: Select all

ref RigidPose _pose = ref gameObject.collider.Pose;
_pose.Position = value.pose.Position;
_pose.Orientation = value.pose.Orientation;

ref BodyVelocity _velocity = ref gameObject.collider.Velocity;
_velocity.Linear = value.velocity.Linear;
_velocity.Angular = value.velocity.Angular;
I haven't worked with structs too much, so I'm not entirely sure if there's a difference in this case.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: "Rewinding" Simulations

Post by Norbo »

Sounds like the object is asleep- many of the BodyReference's functions are intentionally barebones and do not wake the body up automatically. The XML docs for each should mention whether they do or don't. All the direct reference properties and apply impulse functions do not wake the body up. You can set the BodyReference.Awake to true to force the body awake.
Post Reply