Character Controller & Garbage Collection

[Garold]

I have a player and 36 enemies. Each uses the character controller. As they move around the world there is garbage created. If they go over bumpier terrain more garbage is created, also if they bunch together. It's not really a problem on windows, but on the XBOX, the more characters I have, the more stutters I have. The problem is, I am writing the game specifically with the XBOX in mind.

Is there anything I can do to reduce the garbage created?

I have used the XNA Framework Remote Performance Monitor Tool for XBOX 360 to see when garbage is created, and it's definitely connected to the character. I used the CLR profiler for windows to try to determine where it is created, however it's confusing to use, the best I can determine, as wild guess, is it's something to do with collision pairs, maybe, I don't know.

I have updated to the latest development version, as I noticed you have made many changes to the character controller.

Any help at all would be greatly appreciated.

EDIT

I just noticed in this thread: viewtopic.php?f=4&t=1753 that If I use the character's body tag I must implement ICharacterTag. Is this the Body.Tag or the Body.CollisionInformation.Tag? As I currently do modify the character's body tag.

[Garold]

I ran the CharacterStressTestDemo on the XBOX. Once the characters had settled down the FPS was about 6 and the physics update time about 140ms with a garbage collection every 8 seconds or about every 50 frames. The screenshot below shows the managed objects / bytes allocated.



I reduced the number of characters to 64 characters, spheres characters and boxes to 0.

The performance went up to 60fps with the physics update taking around 11ms. The garbage was collected every 30 seconds or so. When the garbage is collected the stutter is pronounced, affecting gameplay.



In my game the enemies head towards the player and do bunch up if I don't kill them, maybe that accounts for why I get more garbage, because they are colliding with each other a lot more than in the example? When I have 25 characters bunched up I get about 1400 objects and 24000 bytes per frame.

Hope this helps in any way at all:)

[Norbo]

I've located and fixed a small source of garbage in the character, but it alone couldn't explain the quantity of garbage on the Xbox. There does indeed seem to be something else about interacting characters which is pumping out a whole lot of garbage on the Xbox, but not so much on the PC.

In a modified CharacterStressTestDemo with forced sustained character interaction, the PC accumulates about a megabyte of allocations (not necessarily garbage) in a minute. The same simulation running on the Xbox pumps out a megabyte of collected garbage every 3 seconds. This discrepancy, if it holds up, should make it easier to find the source.

I'll post again once I have more information/a fix. Thanks for the report!

I just noticed in this thread: viewtopic.php?f=4&t=1753 that If I use the character's body tag I must implement ICharacterTag. Is this the Body.Tag or the Body.CollisionInformation.Tag? As I currently do modify the character's body tag.

The character's Body.CollisionInformation.Tag is the one that needs to implement ICharacterTag. The character systems do not look at the Body.Tag, so it doesn't need to implement ICharacterTag.

[Norbo]

The development version now includes the previously mentioned character fix plus another contact manifold fix:
http://bepuphysics.codeplex.com/SourceC ... evelopment

This should take care of the issue as far as I know; let me know if something appears to still be leaking. Thanks again for the report!

[Garold]

Thanks Norbo, it's way better

I don't know what your goal is with garbage, so therefore I don't know if I am being too picky, I don't want to appear too cheeky. For my game I want about 50 characters and a lot of static mesh, I force a collection at the start of each round, so I think I can cope with the amount created. However if performance allows I would like to add some minor entities that the player could bash or shoot to make the game come alive.

When using the CharacterStressDemo with 64 characters:

The garbage settles down even reaching zero at some times. The longer I leave it running the less garbage. It feels like the characters learn where they have been? Only when they reach a new part of the mesh is garbage created? After leaving running for about five minutes, hardly any garbage is created at all

When using the CharacterPlaygroundDemo:

Garbage is never created when:

Colliding with the spinning plank.
Walking on the see-saw:
Walking or jumping on the diving boards.
Standing on an lift.

Created but then drops to zero:

Walking up and down the stairs; Initially around 300 objects 15000 bytes. After walking up and down the steps a few times it drops to zero.
Repeatedly firing a sphere at the same part of the mesh.
Walking into the mesh.
Firing a sphere at the barrel.
Walking up the tall curved ramp next to the stairs.

Always created:

Firing the sphere at a box.
Walking over a box.
Colliding with a box.

I can provide you with screenshots, but I didn't want to overload the post.

Hope this helps!

[Norbo]

Is it actually garbage? As the simulation expands (more objects, more interacting areas, more collisions, etc.), various systems must expand to match. If there's currently 100 pairs in the pairs pool and the simulation demands 500, allocations are required. Allocations are not always garbage, though; some (like the pair allocations example) are absolutely necessary and do not cause garbage collections because they are not thrown away.

Sometimes, the simulation expansion can result in some internal pool array resizing. The old pool array is tossed out and eventually collected. This is the main source of true garbage under regular conditions. However, this is generally tiny and rare compared to the true non-garbage allocations.

It is possible to pre-allocate resources in these pools, but it's typically not worth the trouble. (If you know for sure the simulation is going to grow very quickly to a large maximum size, it might be worth it. For example, dropping 15000 boxes in a pile will generate many thousands of collision pairs. Since they aren't in the pool by default, they have to be constructed. Further, the pool will rapidly resize over and over until it can contain the maximum. This isn't a free process, so doing it during load can help keep the simulation smooth at runtime.)

[Garold]

I ran the stress test again. I took a snapshot once the objects had hit the ground at 19:54 then another snapshot five minutes later.
On average the FPS was 6, the Physics Time was 150ms, Collision Pairs 2100, Active Objects, 1180. When I left it to run for about 15 minutes the Physics time increased to about 180ms and the collision pairs increased to 2850 but the active objects stayed about the same.

Here's a screenshot at the start after the entities had hit the ground:


And here's one five minutes later.


If you compare the two you can see how much garbage has been created. There were 5 collections.

I am not too sure exactly what the figures mean. I usually just look at Managed Bytes / Objects Allocated: these two figures are what I watch when testing my game, once the game is loaded and they then remain at delta zero then I know there's no garbage being created.

I hope you find this of help

P.S

Before I updated to the latest development version, I was using the Simple Character Controller for my 36 enemies as this would mostly create zero garbage. The reason I changed back to using the character controller was that when the simple characters bunched up or crowded the player then garbage was being created. I changed as you have added code relating to colliding characters.

[Norbo]

I chased some profiling lines down their rabbit holes and found a few more things. Most importantly, there was one more bug hiding out in the simulation island connections system.

The development version contains the changes:
http://bepuphysics.codeplex.com/SourceC ... evelopment

I haven't tested it on the Xbox360 yet, but barring some particularly well-hidden XBox-only chunk of code, it should behave similarly. There's still some resize garbage as mentioned, but it's pretty small and simulations of nonincreasing complexity should run practically flat.

[Norbo]

Something I did there actually broke the deactivation system pretty thoroughly. Oops! Let me just re-fix that...

Edit:
It should be fixed now.

[Garold]

I tried to run it on the Xbox, but there's a problem.

It is doing about 5 garbage collections per second and the demo never displays on the Xbox, just a blank screen, it's caught in some loop

If I make it the second demo, replacing WallDemo, then the CharacterPlaygroundDemo is still displayed when I switch.

I have to switch the Xbox off to exit.

I am using forks_rossnordby_development_6e71e16dfbfc

[Garold]

I ran it on windows and noticed that you now have 11 x 11 instanced meshes. I don't know if that's connected.

I ran it fine on the Xbox with 1 mesh, character, spherecharacter & box.

I will keep testing it by adding more characters etc, until I hit the problem. Maybe it's just a memory issue, as the Xbox doesn't really have that much. I don't know I am just guessing.

[Norbo]

The slowness/failure of the CharacterStressTestDemo load on the Xbox is almost certainly due to those instanced meshes. The BEPUphysicsDrawer isn't very clever about reusing data, so there's many hundreds of thousands of vertices and indices being created and a lot of garbage to go with it. However, optimizing the BEPUphysicsDrawer is not a high priority at this point

Those meshes/extra entities were just there to stress the simulation in different ways for some other performance investigation. The older CharacterStressTestDemo in the previous changesets will still work, though.

[Garold]

I would like to thank you for all your time and effort. I really do appreciate it.

You are a star.