Save/Load CharacterController for server reconciliation

Discuss any questions about BEPUphysics or problems encountered.
Post Reply
demiurghg
Posts: 14
Joined: Mon Jun 04, 2012 10:02 am

Save/Load CharacterController for server reconciliation

Post by demiurghg »

Hello,

I'm trying to implement client-side prediction and server reconciliation.

To perform client-side prediction I simulate entire physical world applying incoming user's command (like move forward, jump, crouch, etc).

To perform server reconciliation I get snapshot from server (that is actually outdated), apply position,
linear and angular velocities to physical objects and replay simulation (applying stored user's command) from this point to client's present time.

While position and velocities seem to be enough to store and replicate state of physical objects,
CharacterController requires more properties to be stored and restored.

For example, I press "Jump" and character controller start jump.
Then snapshot arrive and entire world is rolled back and re-simulated.
But character controller while being rolled back stays in some "jumping state" and starts jumping before applied jump command.

See the attached figure.
character server reconciliation.png
character server reconciliation.png (33.83 KiB) Viewed 28492 times
How to save and load CharacterController -OR- rollback physical world?


Thanks for advance.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: Save/Load CharacterController for server reconciliation

Post by Norbo »

The relevant dynamic state should be:
1) Entity properties (position, orientation, linear velocity, angular velocity)
2) ViewDirection, (used to interpret movement direction)
3) MovementDirection, and
4) tryToJump.

tryToJump is presumably what you're running into. I didn't consider this particular use case when making it private. This flag will be reset to false every single character update, though, so this behavior should be fairly rare under normal conditions. If you're seeing it all the time, something else may be contributing.

I've uploaded a changeset that makes that flag public.
demiurghg
Posts: 14
Joined: Mon Jun 04, 2012 10:02 am

Re: Save/Load CharacterController for server reconciliation

Post by demiurghg »

Direct setting TryToJump helped to solve jumping problem.

Now I've encountered the similar problem with landing.

See attached figure.
Character jumped and holding forward button turned in air by 90 degrees.
Latency is about 200 ms.

Big yellow cross — character position.
Red dots — position data from server
Magenta dots — re-simulated position from server data (retrospective).
Yellow — position that user actually see on screen (retrospective).

Image

What happens (in my opinion):
1. Character controller touched the ground (client side prediction).
2. Character starts moving along viewing direction (90 degrees)
3. Client received server state when character is still in air.
4. While re-simulation characters suddenly falls (because of internal state) on ground and start moving along viewing direction earlier than it should.
5. #3 and #4 repeat 12 times (12 parallel magenta trajectories).

P.S. Removing and Adding CharacterController before re-simulation almost solve this problem, but this solution seems odd.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: Save/Load CharacterController for server reconciliation

Post by Norbo »

Probably the support finder. If it had traction previously but there is no contact-based source of traction now, it will cast a ray down to find support. This is handy for responsiveness when going down steps and such, but it will cause problems like this too.

One option would be to clear the support finder's cache if the server's message says the character should not have traction. The SupportFinder.ClearSupportData function (now public) would work for this. There is also the option of more tightly controlling the Traction and Support states independently, but I doubt that would give you too much compared to just clearing support data.

Another option would be to modify the character controller to turn off the downcast, but that comes with side effects for responsiveness when going down steps.

I also avoided mentioning it for simplicity earlier, but you're already running into some of it, so: watch out for old contact caches sticking around after small corrections. If things seem to 'stick' sometimes near obstacles after server messages, it may be that the current contact cache is suboptimal. The invalidation heuristics usually catch this kind of thing transparently, but the character is very sensitive. If you encounter issues, you can call ClearContacts on each of the character's collision pairs. For example, in the character's jump handling code:

Code: Select all

                        foreach (var pair in Body.CollisionInformation.Pairs)
                            pair.ClearContacts();
                        SupportFinder.ClearSupportData();
As far as stability goes, it's best to not invalidate the pairs' contact caches if possible, so it's probably a good idea to avoid doing it until necessary.
demiurghg
Posts: 14
Joined: Mon Jun 04, 2012 10:02 am

Re: Save/Load CharacterController for server reconciliation

Post by demiurghg »

One option would be to clear the support finder's cache if the server's message says the character should not have traction. The SupportFinder.ClearSupportData function (now public) would work for this. There is also the option of more tightly controlling the Traction and Support states independently, but I doubt that would give you too much compared to just clearing support data.
Yes, it's been really helpful.
thank you for assistance.

I have another related question.
Since server and client could operate on different frame rates, packet loss and latency the server could receive more than one user commands per frame or no user command at all.
Note, that server could receive two commands from one client, and only one command from abother client.
What is the best way to update physical world to achieve close as possible results on client and server side?

Image
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: Save/Load CharacterController for server reconciliation

Post by Norbo »

If you're not already using one, a jitter buffer can smooth out variable delivery and increase synchronization at the cost of a baseline amount of latency. We tend to perceive constant latency as less objectionable than even lower variable latency, so this is often a win.

Depending on the structure of the networking system, it may be a good idea to only have the jitter buffer on one end. For example, I use a jitter buffer for the client, but the server just uses the latest input by logical ticks. This is acceptable in my case since the entirety of the client's simulation is in the server's time stream.

As far as physics updating itself goes, it's probably wise to make sure the client and server have timesteps that don't actively drift out of sync. This isn't a huge issue, but a pair of simulations running at 30hz and 60hz has common synchronization points that a pair running at 47.8hz and 91.1hz does not.
demiurghg
Posts: 14
Joined: Mon Jun 04, 2012 10:02 am

Re: Save/Load CharacterController for server reconciliation

Post by demiurghg »

If you're not already using one, a jitter buffer can smooth out variable delivery and increase synchronization at the cost of a baseline amount of latency. We tend to perceive constant latency as less objectionable than even lower variable latency, so this is often a win.

Depending on the structure of the networking system, it may be a good idea to only have the jitter buffer on one end. For example, I use a jitter buffer for the client, but the server just uses the latest input by logical ticks. This is acceptable in my case since the entirety of the client's simulation is in the server's time stream.
Do I understand correctly, that client-side jitter buffer must consume incoming snapshots
from the server and push them into the client-side game logic attempting to keep snapshot rate?
What is better to do in case of loss of snapshot? Just skip it, or reuse previous one?
What size of jitter buffer is enough for our purposes?

Which server update rate do you use?
I noticed that lower server update rate cause smoother client-side motion (in couple with interpolation).
Quake III uses 20 fps by default.


Another question is related to character controller.

I've studied Quake II source code and realized that they records user's command
and perform only client's player movement prediction.

Is it possible (and reasonable) to (re)simulate only character controller keeping entire other world unchanged?
I'm afraid that total (re)simulation of entire world could consume a lot of time, especially for high latency.
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: Save/Load CharacterController for server reconciliation

Post by Norbo »

Do I understand correctly, that client-side jitter buffer must consume incoming snapshots
from the server and push them into the client-side game logic attempting to keep snapshot rate?
Basically, yes.
What is better to do in case of loss of snapshot? Just skip it, or reuse previous one?
There are two common approaches: interpolation, or some form of extrapolation.

Interpolation works by having a long enough buffer that a missing update can be filled in by interpolating between the previous and next updates. In practice, this can work pretty well, but the required buffer length can introduce fairly significant latency, and things get tricky in the corner case where there is no next update to interpolate with.

Since I only send sparse updates based on priority instead of a simulation-wide snapshot, interpolation doesn't work so well for me. Instead, I just let the clientside simulation fill in the gaps. It generally does well enough in combination with the jitter buffer's smoothing effect and explicit graphical smoothing. This kind of simulation-extrapolation approach tends to work fine with smaller jitter buffers.
What size of jitter buffer is enough for our purposes?
In my setup, I think I found around 100ms worth of steps was enough for even terrible connections (100ms jitter range, massive packet loss, etc.). 20-40 seemed like the sweet spot for reasonably good connections if I remember correctly.

Interpolation approaches need big buffers. You might need 300ms worth of steps, or more. For more information, check out the gaffer on games article: http://gafferongames.com/networked-phys ... rpolation/
Which server update rate do you use?
I noticed that lower server update rate cause smoother client-side motion (in couple with interpolation).
Quake III uses 20 fps by default.
The server tends to send about 60 updates per second right now (this isn't a hard number- it can fall back or do more depending on per-client conditions). Each update, however, is not a full snapshot, but rather a prioritized subset of entities that the player is potentially aware of. The player's own character will tend to get 60hz updates on all but the absolute worst connections, while something very far away might only get 1 or 2 updates per second.

(Technically, the server itself is stepping at 300hz at the moment to compensate for BEPUphysics v2.0's new continuous collision detection implementation not existing. So, it could choose to send 300hz updates for individual objects, but that extra data would be wasted since all the clients currently run at 60hz.)

More updates tends to result in greater smoothness and responsiveness across the board in this approach, so long as the connection can handle it.
I've studied Quake II source code and realized that they records user's command
and perform only client's player movement prediction.

Is it possible (and reasonable) to (re)simulate only character controller keeping entire other world unchanged?
I'm afraid that total (re)simulation of entire world could consume a lot of time, especially for high latency.
It's possible, but pretty annoying. The engine doesn't have built in support for stepping individual objects, so this kind of approach tends to imply a separate simulation where the character in question is the only dynamic object.

This can work okay when all you're concerned about is collision with the static elements of the level, but things get complicated when you want to predict dynamic collisions with other characters and objects. You could use heuristics to determine a 'potentially interacting set' for the player to prune out unnecessary parts of the simulation, but in the worst case, it could end up identical to just running the full simulation.

That's part of the reason why rewind-replay prediction models are less common for heavily physical worlds.
demiurghg
Posts: 14
Joined: Mon Jun 04, 2012 10:02 am

Re: Save/Load CharacterController for server reconciliation

Post by demiurghg »

Thank you for detailed response.

I've spent two days and done de-jittering buffer that filters jitter pretty well and even adapts to the variable latency.

Since server and client+render lives in two separate threads, some jitter occurs too. Snapshots arrive one frame earlier or one frame later.
And I suspect that server loop (that controlled by Stopwatch) and client+render loop (that is VSync'ed) are out of sync.

Also, I've studied Quake2 and Quake3 source code again and perform some experiments.
Quake2's movement prediction algorithm filters out even extremely high jitter.
Quake3 also introduces prediction for items and triggers but with separate from main game codepath.


So, I think that character collision with static elements would be enough for first iterations.
More complex things are going to be added later.

Update:
Which kind of object to choose to replicate dynamic objects from server for player prediction: dynamic or kinematic?


I continue to go down to the rabbit hole...
Norbo
Site Admin
Posts: 4929
Joined: Tue Jul 04, 2006 4:45 am

Re: Save/Load CharacterController for server reconciliation

Post by Norbo »

It depends; both approaches have downsides.
1) If dynamic objects are represented as tightly controlled kinematic objects, then you don't have to worry about correcting the results of any other dynamic objects. This might be good if the dynamic objects in question are not going to be meaningfully affected by the character's motion- like a character walking into a car, for example.
2) For objects which are expected to respond to the player in a significant way, like a football, dynamic prediction could be important. If the football's prediction is kinematic, the character's prediction could get fully stopped by it, causing obvious hitching and rubber banding. However, for particularly chaotic simulations, the dynamic prediction may be way off the server's simulation, once again requiring heavy corrections.

I would probably try dynamic prediction first to see if it works well enough. Then, as necessary, either a) change individual predictors to kinematic, or b) remove individual objects from prediction entirely (e.g. gameplay irrelevant debris).
Post Reply