Physics performance on Xbox 360

[al9191]

I have setup the internal multi-threading, so BEPU runs on threads 1, 3 and 5. However, this doesn't seem to make any difference to the FPS of the game. So I'm not really sure what else I can do.

[Norbo]

If threading the physics doesn't help, that implies there's either something stopping threading from being effective (such as work being done on the cores where physics are trying to update) or the simulation is too small to benefit from threading. I would recommend profiling the game with SlimTune/ANTS/DotTrace/equivalent or just inserting timing code to see where time is being actually being spent. Also, make sure the CPU is the bottleneck before doing CPU-based optimizations- if the GPU is the bottleneck, reducing the CPU load to 0 won't increase the FPS at all.

[al9191]

Ah ok thanks, how can you tell if it is the CPU or the GPU that is the bottleneck?

[Norbo]

Here's a good article about it: http://blogs.msdn.com/b/shawnhar/archiv ... bound.aspx

[al9191]

I have read through that and followed it, and I think the game is GPU bound, as the GraphicsDevice.Present and adding Thread.Sleep(1) to update didn't change framerate.

Therefore, this might explain why the optimisations I have been trying don't improve the framerate. What would be the suggested course of action to improve when it is GPU bound? Or is this not an area you deal with?

[Norbo]

When it's GPU bound and performance is insufficient, measure and find out what needs optimizing. Identifying a GPU bottleneck can be a little trickier than CPU profiling.

Here's an old slide deck; the Bottleneck Identification slide in particular is at least useful for guiding an investigation:
http://origin-developer.nvidia.com/docs ... sation.pdf

While I do work with some of this stuff privately, it's not related to BEPUphysics, so it's probably a good idea to try a graphics-focused forum for more information.

[al9191]

Ok thanks for the help I will have a look at that.

I have ran some more tests and it seems that without the physics space being updated, the game runs smoothly at 60FPS the whole time, even though everything is still being drawn exactly as before. So therefore, somehow it is the physics that is slowly the whole game down.

- I have multi-threaded it -> no change in FPS
- I have replaced the ConvexHulls being used, with Prefabs.Box s instead -> No change in FPS
- I've optimised any code I can find
- Even made it so there is one enemy in the world, so that's 2 dynamic entites (enemy and player) and then the static objects dotted around the world. And the FPS still is slow.
- The FPS starts at 63, and slowly over time decreases more and more, until it hits around 15FPS after a while. This is just with you moving around, so the same stuff is happening every game loop really, no extra stuff happens.

I cannot see what it could be that is slowing it down.

[Norbo]

This is quickly entering the twilight zone of performance tuning where guesses are almost certainly going to fail. At this point, the best thing to do is start rigorously measuring and analyzing without trying to jump to any one solution. Grab a profiler of your choice or implement some detailed timing code to collect data.

- Even made it so there is one enemy in the world, so that's 2 dynamic entites (enemy and player) and then the static objects dotted around the world. And the FPS still is slow.

That is an extremely cheap simulation unless you have many hundreds of separate static objects on the Xbox or many thousands of static objects on the PC. That would also not degrade performance over time. Since this situation does not fit expected behavior, standard performance tuning advice ("lower iteration count! use simpler meshes! use approximations! use threads more efficiently!") won't help at all.

- The FPS starts at 63, and slowly over time decreases more and more, until it hits around 15FPS after a while. This is just with you moving around, so the same stuff is happening every game loop really, no extra stuff happens.

The fact that performance decreases over time is a good hint, but I'm wary of extrapolating from that without the raw data showing exactly where that time is being spent.

[al9191]

Ok thanks very much. I will do some profiler and get some data for you.

I have the time it spends in Update and Draw each time, and the time spent doing the updating of physics engine. What kind of information would be needed to tell what it was, would you need which part of physics engine, in which case I might need to put the source code for BEPU into the game so that I can profile it.

*EDIT*

Drawing - Total: 1688ms, count: 61, max ms: 29.85
Updating - Total: 2597ms, count: 261, max ms: 82.45
Physics update - Total: 1878ms, count: 261, max ms: 11.13


I am looking at the Space.update method and will try putting some timing code around different parts of it to see which bits are taking the longest. Which parts are the best to test?

[al9191]

Tested using SlimTune.

in update method:
-62% doing Space.Update()

in space.update:
-36% - updating BroadPhase systems
-20% - updating NarrowPhase systems
-17% - updating UpdateableSystems.UpdateManager.Update
-8% - updating Solver systems
-6% - deactivation management
-5% - force updater
-4% bounding box updater

Due to not knowing how much time should be spent where in the physics engine, I can't really see what is wrong, by looking at the statistics.

[Norbo]

Nothing in those relative space update statistics looks strange. That fits the profile of a relatively small simulation with few dynamic actors, a reasonable quantity of static objects, and a couple of characters. BroadPhase ends up being a larger percent due to the acceleration structure containing all the static objects, but it's still likely very cheap in absolute terms unless you have hundreds or thousands of static objects.

Updating - Total: 2597ms, count: 261, max ms: 82.45
Physics update - Total: 1878ms, count: 261, max ms: 11.13

This implies that the full update becomes substantially more expensive than the physics updates at some point.

Unfortunately it is going to be unlikely that I can help too much remotely. These kind of weird problems usually have local interactions that I am unable to predict. The best option is just to keep gathering data and measuring until something makes sense. My guess is that it isn't caused solely by the physics, but something external, or something external interacting with the physics. But such vague guesses aren't very valuable

A couple of things to try:
-Run the CLR profiler to check for memory allocation buildup. Frequent garbage collections can cause slowdowns, though unless you are generating more and more and more garbage, it shouldn't be a progressive slowdown.
-Profile only the period where the game has slowed down a bunch. You can do this with a profiler that has a timeline view like ANTS, or hook the profiler later, or enable the logging later.

[al9191]

Thanks very much for the help.

So even though stopping the physics updating, it could be that external interaction with the physics, makes updating it take longer?

Ok, the main problem is just I can't gleen any information from profiling really, as having not written the physics engine, how long it takes in certain parts doesn't really mean anything to me. The ms data above was given for the slowed down period.

The main problem is the progressive slowdown is only seen on the xbox, it does not happen on the PC. Therefore, the profiling tools I have for the %s aren't available. We have some form of profiling in the game that times sections, but it can only give out that ms data I gave first, no %s spent in each method.

[Norbo]

So even though stopping the physics updating, it could be that external interaction with the physics, makes updating it take longer?

I don't know the implementation, so I can't say. I wouldn't rule it out.

The main problem is the progressive slowdown is only seen on the xbox, it does not happen on the PC. Therefore, the profiling tools I have for the %s aren't available. We have some form of profiling in the game that times sections, but it can only give out that ms data I gave first, no %s spent in each method.

The PC is an order of magnitude faster than the Xbox, so the progressive slowdown might still be present- just much better hidden. It may not be reasonable to reproduce it that way, though.

Unless there are platform-specific codepaths, running CLR profiler will tell you if there are allocation issues.

For checking what's going on on the Xbox directly, using the XNA Framework Remote Performance Monitor might help too. It can tell you about allocations and GC counts too.

Ok, the main problem is just I can't gleen any information from profiling really, as having not written the physics engine, how long it takes in certain parts doesn't really mean anything to me. The ms data above was given for the slowed down period.

Unfortunately, my familiarity with the engine is not going to help much in identifying the problem. It's already relatively clear that it's not a standard expected performance problem sourced in high polygon objects or something trivial like that. That leaves things I can't really predict.

Fortunately, you don't need to know much about how the engine is implemented to flag something as strange.

Here's the rough guidelines to follow for the relative physics times:

1 ) Is any one stage greater than 40%? Small probability of something weird, unless there's a reasonable explanation (simulation type).

2 ) Is any one stage greater than 60%? Probably something weird going on, unless you specifically designed your simulation to stress that stage. For example: a world full of only hundreds of static objects will stress the broad phase, but not much else since no collisions or solving will happen.

3 ) Is any one stage greater than 90%? Almost certainly some shenanigans going on.

4 ) Most simulations see either the narrow phase or solver taking the top spot, both hanging around 20-40%.

5 ) Simulations that have few dynamic objects and more static objects in them pressure the broad phase more relatively speaking, like yours.

6 ) Simulations that have lots of dynamic objects floating in space will also pressure the broad phase more than other systems due to the rarity of sustained collision.

If something violates the above expectations, or if the physics time is just taking up massive amounts of time in absolute terms (say, growing to 100ms or something ridiculous and obviously wrong), then it would be worth taking a closer look at the parts of the simulation.

Until then, though, I would suggest broadening the investigation.

1 ) Assume to start with that the problem is not primarily caused by any external library, be it .NET or BEPUphysics or XNA. It may involve those external libraries, but the core problem is likely elsewhere.

2 ) Attempt to reproduce the problem on the PC where you have better tools. Most problems transfer from PC to Xbox360 in the absence of platform-specific code; the PC is just better at hiding them.

3 ) Use every profiler you can to gather information. Learn what each one can do. Use both performance profilers and memory profilers.

4 ) Attempt to reproduce an isolated version of the problem. Disable chunks of code or whatever else allows you to narrow it down.

5 ) Rigorously record the incremental changes and their results. If something doesn't make sense, make sure to note the confusion- it's a hint!

6 ) If disabling something helps but you do not know the mechanism by which it helps, re-enable it and disable a smaller portion. Keep searching deeper, narrowing the causes.

7 ) While going deeper, always keep in mind you might only be looking at one tendril of the problem. The monster may have multiple limbs that you need to cut off before you can fully understand why it is happening.

8 ) Make sure you have detailed and helpful information. The more, the better. When you hit a tantalizing data point, go deeper. For example, in the earlier post, total physics and update time were similar (within a factor of 2), but the max Update time was 8x higher. There's not enough information to know what's wrong with just that top level datapoint- it just hints that something isn't quite right.

9 ) Spending a couple of hours to create a live visualization tool that graphs time spent in various targeted sections of code might save you hours of labor hunting through raw data. Measure, remeasure, reremeasure.


Good luck!

[al9191]

Thanks so much for all of that help. I will keep it all in mind .

Quick question,

I tried putting all the enemies back in the game rather than having just a single enemy.
The whole game dropped from 55ish FPS with one down to 3FPS permanently.

around 25 static meshes, 1 dynamic object player, 1 dynamic object enemy with AI = 50-55ish FPS.
around 25 static meshes, 1 dynamic object player, around 15 dynamic object enemies at once with AI = 3-4ish FPS

Is there any obvious reason for having the extra dynamic objects (ConvexHulls), causing that?

[Norbo]

Working under the assumption that it's a 'regular' performance issue and assuming those numbers are on the Xbox, more objects are more expensive. However, 15 is not a very big number. The BEPUphysicsDemos handles stacks of 100 dynamic objects on the Xbox360 just fine. ConvexHullShapes can be more expensive than BoxShapes if they have many more vertices in them. A good maximum for a ConvexHullShape is around 20 vertices. Going beyond that doesn't give much in the way of extra noticeable detail, but linearly increases cost.

If the ConvexHullShapes are extremely complicated currently, that might be able to explain the drop. However, it may be something else. If they're character controllers, that will use up a bit more time. The AI might be slowing it down to, depending on what it does.

So, I would recommend again: measure, measure, measure