Lag when adding a mesh to the simulation

[Telanor]

I've been having an issue where my game stutters when I add new meshes to the simulation. It happens with and without buffered states, with and without asynchronous updates, and it happens regardless of whether the mesh generation and adding is done on a separate thread or not.

Code: Select all

private void GenerateRegion(object o)
{
	BlockRenderer blockRenderer = o as BlockRenderer;

	if(blockRenderer == null || blockRenderer.SolidIndexArray == null || blockRenderer.SolidVertexArray == null)
		return;

	RemoveRegion(blockRenderer);

	Vector3[] vertexData = new Vector3[blockRenderer.SolidVertexArray.Length];

	for(int i = 0; i < vertexData.Length; i++)
	{
		vertexData[i] = blockRenderer.SolidVertexArray[i].Position;
	}

	var mesh = new StaticMesh(vertexData, blockRenderer.SolidIndexArray) { ImproveBoundaryBehavior = true };

	WorldSpace.SpaceObjectBuffer.Add(mesh);
	regionShapes.TryAdd(blockRenderer, mesh);
}

If I comment out the WorldSpace.SpaceObjectBuffer.Add(mesh); line, the stuttering goes away. Any ideas why this is happening?

[Norbo]

The fact that it still happens without asynchronous updates- meaning, presumably, that there is no contention- implies that this is not a contention issue (and even with asynchronous usage, the synchronization is so brief and rare that it should not be a problem).

The fact that the main thread still stutters in asynchronous updating (if I understand correctly) would imply that it's not something on the Space's thread of operation, but rather something external responding poorly to the new object in some way.

That leaves more traditional performance issues. I would recommend doing some profiling/timing with completely synchronous updates to see if something is taking significantly more time than it should during the stutters.

[Telanor]

I don't think its a performance issue, because I tried putting in a Thread.Sleep(1000) as well as a 1s busy loop in place of doing any actual loading and there was no stuttering. I ran the VS profiler in sampling mode and this is the result:

sampling.png (33.32 KiB) Viewed 10009 times

As you can see, the Space.Add hardly takes any time at all. I should also note that when I say its not running in its own thread, I don't mean that its running on the main thread. All our loading occurs on a separate thread from the main rendering thread. If the loading was on the main thread, we'd have even worse stuttering. As I had mentioned, I tried running the physics-specific loading in its own thread and still had this issue.

Edit: Something I just realized. When I have "synchronous" updates on, the Update method is actually called by the main thread. The loading is still happening on that secondary loading thread. So I suppose the way its set up, I don't actually have synchronous updates. I'm not sure if this would cause this problem or not.

[Norbo]

In the report, Space.Add is being called directly; is that from another thread? If so, is there any chance that space.Update is running when the asynchronous Space.Add is called? That could cause all sorts of problems (most worse than a stutter, though).

In the original post, it sounds like:
-Update on main thread.
-Space.SpaceObjectBuffer.Add called from separate loading thread.
was one of the approaches attempted, and resulted in the stutter- in other words, the main thread (which also handles rendering) was slowed. Is that true? If so, the only thing I can think of is that the new object requires some intense first-frame processing, such as generating hundreds of collision pairs. That would not show up in the Space.Add cost, but the next Space.Update stages could be more expensive.

[Telanor]

In the original post, it sounds like:
-Update on main thread.
-Space.SpaceObjectBuffer.Add called from separate loading thread.
was one of the approaches attempted, and resulted in the stutter- in other words, the main thread (which also handles rendering) was slowed. Is that true?

That's exactly correct. My terrain consists of blocks organized into 16x16x128 regions. After a region finishes loading, it generates a StaticMesh and adds it to the Space. The blocks only have faces generated when they're touching air. Would this setup cause that first-frame processing problem? Also, if that's the case, how come running the updates in its own thread doesn't solve the problem?

I ran another sampling profile with it set up the original way (as described in the quote) and looked at the Update method. Its not really showing much processing there, but I think that's probably because this lag is only happening once every thousand+ frames, so it's getting averaged out.

sampling2.png (30.9 KiB) Viewed 10005 times

Edit: I'm actually convinced now that this is due to an occasional slow update. This stuttering is less of an FPS problem and more of a movement issue. Since my players movement is entirely dependent on the physics, slow updates make the player move in a very choppy manner.

[Norbo]

My terrain consists of blocks organized into 16x16x128 regions. After a region finishes loading, it generates a StaticMesh and adds it to the Space. The blocks only have faces generated when they're touching air. Would this setup cause that first-frame processing problem? Also, if that's the case, how come running the updates in its own thread doesn't solve the problem?

Adding a generated object isn't a problem by itself. If the new object then causes hundreds or thousands of new collisions, then it can be a problem. A slowdown in the Space.Update on its own thread can force the render thread to use old data.

If the generated object doesn't cause a bunch of new collisions, then this first-frame processing explanation is less likely. I can't think of any other Space.Update-related mechanism which would trigger on the first frame after an add.

Another possibility is garbage collection. If there's garbage left over from the processing stage, it might get collected at some point after the addition. Depending on heap complexity and garbage quantity, this could freeze the application for a frame or two.

If the above doesn't work out, then I'm out of good theories for now. Measuring for time spikes in various places might help guide the investigation.

[Telanor]

Is mesh.Pairs.Count the right way to check the number of collision pairs on a StaticMesh? I put this in my update code:

Code: Select all

public void Update(GameTime gameTime)
{
	var sw = Stopwatch.StartNew();
	WorldSpace.Update((float)gameTime.ElapsedGameTime.TotalSeconds);

	if(sw.Elapsed.TotalMilliseconds > 5)
		Console.WriteLine("slow: {0}", sw.Elapsed.TotalMilliseconds);


	foreach(var region in regionShapes)
	{
		var count = region.Value.Pairs.Count;

		if(count > 2)
			Console.WriteLine(count);
	}
}

The slow output line seemed to match the stuttering, with it sometimes taking around 10ms and other times taking 90-100ms. I didn't get any output from the Pairs.Count though.

[Norbo]

Does it only happen once (the first time for a given type), or does it happen every time? It might be JIT related if it is only once and something still needs to be initialized.

That timing value does not rule out a GC-related issue, either; it may be worth checking it out with the CLR profiler.

[Telanor]

The stuttering happens every few seconds when moving in a straight line (and thus causing regions to be loaded and unloaded). I've never used the CLR profiler before, what should I be looking at/for?

[Norbo]

There's some better explanations around the web, but basically, it shows the managed memory in a variety of ways; the timeline is probably going to be the most immediately helpful. If there are collections or compactions (they'll be labeled on the axis) going on during runtime, jerky performance is possible/likely.

[Telanor]

Well in that case, according to the timeline, there are tons of GCs. About 2 per second if I'm reading this correctly. It looks roughly the same with Space.SpaceObjectBuffer.Add commented out though. Also, with that commented out, I no longer get the "slow" console output messages

clr.png (33.34 KiB) Viewed 9999 times

[Norbo]

Since they are common, GCs may still contribute (i.e. the stutter might get reduced if there wasn't any GC pressure), but it doesn't seem likely. Unfortunately, we've covered the easily guessable possibilities; I don't have much in the way of specific theories left. What is left is more measurement and perhaps constructing a demo which shows the problem in isolation to track it down easier (and if necessary, allows me to debug it directly).

[Norbo]

Judging by what I'm seeing in the reproduction demo, it appears unrelated to the physics. The stuttering appears to be roughly the same with or without commenting out the adds. It fits the profile of a GC-caused stutter very well.

The "slow" messages are indeed rarer without the adds, but that can be explained by the Space.Update being shorter due to not having any objects in it. By being shorter, it is less likely that a GC or other external influence will artificially bump its time. When it's longer, it's more likely that a GC will happen while it's updating, making it take much longer.

[Telanor]

Hmm, I haven't been able to get any stuttering at all with the add commented out. I'll have to spend some time investigating the GC issue and see if I can do anything about it. I'll post back here when I make some progress with that.

[Telanor]

I think I've dealt with the GC issue now, but the physics updates get increasingly slower as more regions are loaded now. I think I might have some kind of issue with overlapping meshes or meshes not being removed from the simulation or something. I can't really see the problem in the code though, and I didn't change much with the physics itself. Do you have any hints on debugging slow updates? I was going to try using the Debug Drawer but it seems that only works for XNA.