"Rewinding" Simulations
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
"Rewinding" Simulations
So I'm getting into the more complex parts of client-server setups now, and one thing I need to sort out is lag compensation.
Most multiplayer games will "roll back" their simulation to the point in time when the input was sent by the player and simulate from there, thereby making corrections to the previously simulated game state. In order to do this, the physics need to be resimulated too.
The new Unity Physics is designed with multiplayer use cases in mind, meaning it's relatively simple to rewind and resimulate.
I suspect this may be more difficult with Bepu, however Unity Physics is still very much in the early stages, so I'm not sure how viable it is to use currently.
This brings me to my question: is it possible to "roll back" Bepu (v2)?
I was thinking about it and it occurred to me that since I have to store the state of the game anyways, I could directly set positions/rotations/velocities of bodies at the point in time I need to go back to...would this be a "good" way to approach the problem?
I remember reading something about an option to make Bepu deterministic, so I'd have to look into that again, but is this whole concept achievable with Bepu or are there too many issues with setting something like this up?
In theory, as long as I properly set all bodies that can move, I should be able to restore the simulation to a previous state, right?
Of course there's still the issue of not being able to use Bepu in Unity, so I'm not entirely sure how I'll handle client prediction without access to the same physics...but that's a separate question.
Most multiplayer games will "roll back" their simulation to the point in time when the input was sent by the player and simulate from there, thereby making corrections to the previously simulated game state. In order to do this, the physics need to be resimulated too.
The new Unity Physics is designed with multiplayer use cases in mind, meaning it's relatively simple to rewind and resimulate.
I suspect this may be more difficult with Bepu, however Unity Physics is still very much in the early stages, so I'm not sure how viable it is to use currently.
This brings me to my question: is it possible to "roll back" Bepu (v2)?
I was thinking about it and it occurred to me that since I have to store the state of the game anyways, I could directly set positions/rotations/velocities of bodies at the point in time I need to go back to...would this be a "good" way to approach the problem?
I remember reading something about an option to make Bepu deterministic, so I'd have to look into that again, but is this whole concept achievable with Bepu or are there too many issues with setting something like this up?
In theory, as long as I properly set all bodies that can move, I should be able to restore the simulation to a previous state, right?
Of course there's still the issue of not being able to use Bepu in Unity, so I'm not entirely sure how I'll handle client prediction without access to the same physics...but that's a separate question.
Re: "Rewinding" Simulations
Simply caching historical pose/velocity should work fine. It's true that internal cached state will mean that the simulation won't produce bitwise identical results, but since the resimulation 1) is different due to user input anyway and 2) lasts a fraction of a second, determinism shouldn't be a practical concern.
(The Deterministic property also only guarantees local determinism given identical input sequences, so a resimulation that doesn't start from the same internal state won't produce the same results. It is possible in theory to extract and reapply the internal state to get around this, but the value in this use case is effectively zero.)
![Razz :P](./images/smilies/icon_razz.gif)
Honestly, I'm not sure if that would be more or less gross than some of the alternatives...
(The Deterministic property also only guarantees local determinism given identical input sequences, so a resimulation that doesn't start from the same internal state won't produce the same results. It is possible in theory to extract and reapply the internal state to get around this, but the value in this use case is effectively zero.)
Depending on the target platform, you could go supergrosstown and have a separate local process running bepu on top of CoreCLR on the client computer, communicating with near zero latency to the unity processOf course there's still the issue of not being able to use Bepu in Unity, so I'm not entirely sure how I'll handle client prediction without access to the same physics...but that's a separate question.
![Razz :P](./images/smilies/icon_razz.gif)
Honestly, I'm not sure if that would be more or less gross than some of the alternatives...
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
Awesome! However, not all user input will be different every time it's resimulated. In fact I think it's likely that usually only a few players' inputs will have changed by the time a resimulation is necessary, meaning it'd be best if the other user inputs which don't change produce the same results. My guess is you're probably correct about determinism not being key, but it'd still be nice.Norbo wrote: ↑Wed Oct 30, 2019 8:47 pm Simply caching historical pose/velocity should work fine. It's true that internal cached state will mean that the simulation won't produce bitwise identical results, but since the resimulation 1) is different due to user input anyway and 2) lasts a fraction of a second, determinism shouldn't be a practical concern.
So what you're saying is that given the same poses/velocities, there could still be differences in the internal state?Norbo wrote: ↑Wed Oct 30, 2019 8:47 pm (The Deterministic property also only guarantees local determinism given identical input sequences, so a resimulation that doesn't start from the same internal state won't produce the same results. It is possible in theory to extract and reapply the internal state to get around this, but the value in this use case is effectively zero.)
Sounds complicatedNorbo wrote: ↑Wed Oct 30, 2019 8:47 pm Depending on the target platform, you could go supergrosstown and have a separate local process running bepu on top of CoreCLR on the client computer, communicating with near zero latency to the unity process
Honestly, I'm not sure if that would be more or less gross than some of the alternatives...
![Razz :P](./images/smilies/icon_razz.gif)
Ideally I'd like to keep my options regarding platforms open...are there any limitations to this approach?
Re: "Rewinding" Simulations
Yup. The engine incrementally updates a bunch of different datastructures in the broad phase, narrow phase, solver, bodies sets, and so on. Any difference can propagate to noticeable divergence given enough time and opportunity for chaos. Fortunately, 250ms isn't much time.So what you're saying is that given the same poses/velocities, there could still be differences in the internal state?
If you're allowed to launch and communicate with an arbitrary process and the platform is supported by CoreCLR or another runtime that can handle bepuphysics v2, it'll work. So linux/mac/windows should be fine, IOS/android probably not.Ideally I'd like to keep my options regarding platforms open...are there any limitations to this approach?
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
So I've implemented states for my server now which get stored for half a second, allowing me to rewind time.
However, I'm having issues with applying forces—specifically my buoyancy code is causing issues, although I suspect I messed up somewhere and that's on my end.
However, when I turned it off and tried to move my player around, it behaved very similarly with the position shooting up to either (NaN, +/-Infinity, NaN) or (+/-Infinity, +/-Infinity, +/-Infinity). I'm using the built in character controller and it's constraints-based movement.
Positions/velocity should be getting set properly...but is there anything I need to watch out for with constraints when rewinding to a previous state?
However, I'm having issues with applying forces—specifically my buoyancy code is causing issues, although I suspect I messed up somewhere and that's on my end.
However, when I turned it off and tried to move my player around, it behaved very similarly with the position shooting up to either (NaN, +/-Infinity, NaN) or (+/-Infinity, +/-Infinity, +/-Infinity). I'm using the built in character controller and it's constraints-based movement.
Positions/velocity should be getting set properly...but is there anything I need to watch out for with constraints when rewinding to a previous state?
Re: "Rewinding" Simulations
Constraints do preserve impulses from the previous frame to warm start the next solve, but the expected failure mode there is just slightly worse accuracy. Calling Simulation.Solver.ScaleAccumulatedImpulses(0) would wipe out the cached guesses. I don't expect it to be related, but it's worth trying to see if anything changes.
Another option would be to significantly increase the Simulation.Solver.IterationCount- since the old impulses are just an initial guess, more iterations should compensate for a bad guess. Again, this shouldn't explain a full NaNsplosion, but it's something to rule out.
Collision detection runs before the solver, so any changes in constraints caused by the teleportation should be caught. The character would be updated too- though there isn't much state to worry about in the character constraints anyway.
As always, a reproduction in the demos would be helpful to narrow things down.
Another option would be to significantly increase the Simulation.Solver.IterationCount- since the old impulses are just an initial guess, more iterations should compensate for a bad guess. Again, this shouldn't explain a full NaNsplosion, but it's something to rule out.
Collision detection runs before the solver, so any changes in constraints caused by the teleportation should be caught. The character would be updated too- though there isn't much state to worry about in the character constraints anyway.
As always, a reproduction in the demos would be helpful to narrow things down.
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
I tried this and at first I thought it fixed it, but that was only initially.Norbo wrote: ↑Tue Nov 05, 2019 10:26 pm Constraints do preserve impulses from the previous frame to warm start the next solve, but the expected failure mode there is just slightly worse accuracy. Calling Simulation.Solver.ScaleAccumulatedImpulses(0) would wipe out the cached guesses. I don't expect it to be related, but it's worth trying to see if anything changes.
How much is "significantly"? I set it to 1024 and that has stopped my objects flying up and down, however the client is still receiving positions of (NaN, +/-Infinity, NaN), and objects still "jitter" around. Also are there performance implications of changing this value dramatically?Norbo wrote: ↑Tue Nov 05, 2019 10:26 pm Another option would be to significantly increase the Simulation.Solver.IterationCount- since the old impulses are just an initial guess, more iterations should compensate for a bad guess. Again, this shouldn't explain a full NaNsplosion, but it's something to rule out.
If I get really stuck, then I might come back to this...it's not something that would be easy to reproduce since I'd have to replicate a lot of my server setup.
Re: "Rewinding" Simulations
1024 qualifies as a significant increaseHow much is "significantly"? I set it to 1024
![Smile :)](./images/smilies/icon_smile.gif)
Yes- the cost of the solver is proportional to the number of iterations. Setting it to an extreme value is only useful as a diagnostic.Also are there performance implications of changing this value dramatically?
It doesn't sound like an accumulated impulse problem then- the extreme iteration count may have allowed the solver to partially compensate for some nasty inputs, but the other behavior implies something else is going on.that has stopped my objects flying up and down, however the client is still receiving positions of (NaN, +/-Infinity, NaN), and objects still "jitter" around.
Shot in the dark- is the timestep duration passed Simulation.Timestep kept constant?
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
I just realized that my code's execution order is like this for each tick:
Is there a way to clear all impulses that haven't been used yet, or should I just change the execution order so the physics timestep happens between impulses being applied and time potentially being rewound (which would result in no left over impulses, right)?
- Check if there are unprocessed inputs, if so, set the state to the time of the input and resimulate up to the present
- Physics timestep
- Apply impulses for flotation & set input for character constraints
Is there a way to clear all impulses that haven't been used yet, or should I just change the execution order so the physics timestep happens between impulses being applied and time potentially being rewound (which would result in no left over impulses, right)?
Re: "Rewinding" Simulations
Applying an impulse immediately modifies the velocity, so there's no extra state to clear.
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
Constant timesteps are good for stability. Just wanted to rule out something like accidentally taking a negative timestep or other potentially catastrophic issues.
Unfortunately I can't think of any other likely causes. Might be time to start pruning stuff out bit by bit, working toward a stripped reproduction case (or finding more hints along the way). Ideally the issue could be reproduced without any kind of client/server architecture- just a local set of simulation interactions which cause a failure.
Unfortunately I can't think of any other likely causes. Might be time to start pruning stuff out bit by bit, working toward a stripped reproduction case (or finding more hints along the way). Ideally the issue could be reproduced without any kind of client/server architecture- just a local set of simulation interactions which cause a failure.
-
- Posts: 99
- Joined: Wed May 08, 2019 12:17 am
Re: "Rewinding" Simulations
So I have no idea if this will help, but I've found something very strange.
I added a basic cube with my flotation code to the simulation, so it has none of the other player-related stuff, and integrated it with my state system. What I don't understand is that cube falls to the ground (meaning it's not spawning inside anything else) and sits there. When it's underwater, buoyancy impulses are applied, and the velocity gets bigger and bigger, but it doesn't move.
Surely a velocity of 350,000m/s should be able move a 0.1kg cube upwards against the measly effect of gravity (which I've set to 9.8m/s/s)...
I'm not sure if it's related to the actual issues I'm having, but this is certainly very strange.
PS: during the duration of me writing this post, the velocity has increased to 500,000m/s...
EDIT: I tried it a few more times, and this result doesn't seem to be consistent after all. Sometimes it doesn't move but the velocity grows, sometimes the Y components of both velocity and position shoot to NaN almost instantly, and other times the Y components of both velocity and position get really large (like in the several hundred million range).
This is a bit of a noob question, but does it make a difference whether I set the position/velocity like this:
or like this:
I haven't worked with structs too much, so I'm not entirely sure if there's a difference in this case.
I added a basic cube with my flotation code to the simulation, so it has none of the other player-related stuff, and integrated it with my state system. What I don't understand is that cube falls to the ground (meaning it's not spawning inside anything else) and sits there. When it's underwater, buoyancy impulses are applied, and the velocity gets bigger and bigger, but it doesn't move.
Surely a velocity of 350,000m/s should be able move a 0.1kg cube upwards against the measly effect of gravity (which I've set to 9.8m/s/s)...
I'm not sure if it's related to the actual issues I'm having, but this is certainly very strange.
PS: during the duration of me writing this post, the velocity has increased to 500,000m/s...
EDIT: I tried it a few more times, and this result doesn't seem to be consistent after all. Sometimes it doesn't move but the velocity grows, sometimes the Y components of both velocity and position shoot to NaN almost instantly, and other times the Y components of both velocity and position get really large (like in the several hundred million range).
This is a bit of a noob question, but does it make a difference whether I set the position/velocity like this:
Code: Select all
gameObject.collider.Pose.Position = value.pose.Position;
gameObject.collider.Pose.Orientation = value.pose.Orientation;
gameObject.collider.Velocity.Linear = value.velocity.Linear;
gameObject.collider.Velocity.Angular = value.velocity.Angular;
Code: Select all
ref RigidPose _pose = ref gameObject.collider.Pose;
_pose.Position = value.pose.Position;
_pose.Orientation = value.pose.Orientation;
ref BodyVelocity _velocity = ref gameObject.collider.Velocity;
_velocity.Linear = value.velocity.Linear;
_velocity.Angular = value.velocity.Angular;
Re: "Rewinding" Simulations
Sounds like the object is asleep- many of the BodyReference's functions are intentionally barebones and do not wake the body up automatically. The XML docs for each should mention whether they do or don't. All the direct reference properties and apply impulse functions do not wake the body up. You can set the BodyReference.Awake to true to force the body awake.