Path Following: Entities stop responding randomly

[jaja1]

From my previous thread I managed to implement a path system using the demo and the enemy moves as I want it to now. However, I have this weird problem on my server where on random restarts, entities stop responding totally to any input or automated movement patterns.
I checked and my space is being updated as well as the time parameter for the paths. The player which is controlled by an EntityMover (setting targetPosition) will move very slightly forward/backward when I press a key but will only move so far, then no more. The very small movement is due to the fact that whenever the player presses the forward/backward key, based on their orientation the player's position will be updated by small increments like so:

Code: Select all

if (!unityClient.isMoving)
            {
                unityClient.isMoving = true;
                Vector3 oldpos = unityClient.compactCollider.Position;
                Vector3 forward = unityClient.extendedCollider.OrientationMatrix.Forward;
                Vector3 newpos;

                float rate = 0;
                if (!unityClient.canRun)
                {
                    rate = unityClient.cacheCharStatsInfo.walk_speed * 0.01f;
                }
                else
                {
                    rate = unityClient.cacheCharStatsInfo.run_speed * 0.01f;
                }

                if (dir > 0)
                {
                    newpos = new Vector3(oldpos.X - (forward.X * rate), oldpos.Y, oldpos.Z - (forward.Z * rate));
                }
                else
                {
                    newpos = new Vector3(oldpos.X + (forward.X * rate), oldpos.Y, oldpos.Z + (forward.Z * rate));
                }

                if (newpos != oldpos)
                {
                    unityClient.charCompactMover.TargetPosition =
                    unityClient.charExtendedMover.TargetPosition =
                    unityClient.charGrabMover.TargetPosition = newpos;

                    unityClient.currentMapPos[1] = newpos.X;
                    unityClient.currentMapPos[2] = newpos.Y;
                    unityClient.currentMapPos[3] = newpos.Z;
                }
                else
                {
                    unityClient.isMoving = false;
                    return;
                }

                unityClient.isMoving = false;
                return;
            }
            else return;

What I cant figure out is why the player would only move once in either direction and no further. It seems based on observation every time something goes wayward with my monster's entities it affects all other entities in the space.

The monster which uses a path following system does not move at all even if my logs are telling me it is being updated. I am not sure how to isolate the problem...it seems totally random. Any ideas where I can start looking?

EDIT: I also noticed that I may not be handling my loops properly. My update method is basically run by a timer every 16 ticks. It starts when the server started. I tried restarting and once again the same problem shows up and it takes a very long time for my entity to even spawn. in the client. This has never happened before and I did not change my code at all before the restart.

[Norbo]

I am not sure how to isolate the problem...it seems totally random. Any ideas where I can start looking?

The behavior doesn't pattern match to any common oopsies, and the core problem seems to be too far from the engine for me to predict where the issue might be.

I can only suggest the usual universal strategies- binary search through older commits to see where the issue first started happening, start simplifying things down piece by piece to eliminate parts of the code as causes, walk step-by-step through suspected code and check everything against expected values, and so on.

[jaja1]

Well I managed to strike an exception, again on a completely random instance. Its a null reference exception on adding an entity to a space (a method in which tens...maybe even hundreds of times has never thrown this exception before). I am starting to think it is "normal" to get these random bugs and just restart my server when it happens. But this is BEPUphysics related so I fear it is a bug.

Here is the exception:

Code: Select all

2014-12-26 22:13:28,564 [17] DEBUG MukizuServer.MukizuServerClass [(null)] - System.NullReferenceException: Object reference not set to an instance of an object.
   at BEPUphysics.BroadPhaseSystems.Hierarchies.InternalNode.TryToInsert(LeafNode node, Node& treeNode)
   at BEPUphysics.BroadPhaseSystems.Hierarchies.DynamicHierarchy.Add(BroadPhaseEntry entry)
   at BEPUphysics.Space.Add(ISpaceObject spaceObject)
   at MukizuServer.PhysicsHandler.AddPlayerEntity(UnityClient unityClient) in C:\Users\jaja\Documents\Visual Studio 2010\Projects\MukizuServer\MukizuServer\PhysicsHandler.cs:line 88
   at MukizuServer.UnityClient.OnOperationRequest(OperationRequest operationRequest, SendParameters sendParameters) in C:\Users\jaja\Documents\Visual Studio 2010\Projects\MukizuServer\MukizuServer\UnityClient.cs:line 537
2014-12-26 22:13:28,658 [18] DEBUG Photon.SocketServer.ApplicationBase [(null)] - OnDisconnect - ConnID=0
2014-12-26 22:13:28,689 [14] ERROR Photon.SocketServer.ApplicationBase [(null)] - System.InvalidOperationException: Entry not present in the hierarchy.
   at BEPUphysics.BroadPhaseSystems.Hierarchies.DynamicHierarchy.Remove(BroadPhaseEntry entry)
   at BEPUphysics.Space.Remove(ISpaceObject spaceObject)
   at MukizuServer.UnityClient.OnDisconnect(DisconnectReason reasonCode, String reasonDetail) in C:\Users\jaja\Documents\Visual Studio 2010\Projects\MukizuServer\MukizuServer\UnityClient.cs:line 138
   at Photon.SocketServer.PeerBase.<>c__DisplayClass3.<Photon.SocketServer.IManagedPeer.Application_OnDisconnect>b__1() in h:\svncontent\photon-socketserver-sdk_3.4\src\Photon.SocketServer\PeerBase.cs:line 675
   at ExitGames.Concurrency.Core.DefaultExecutor.Execute(List`1 toExecute) in d:\dev\exitgames-libs\src\Core\Concurrency\Core\DefaultExecutor.cs:line 21
   at ExitGames.Concurrency.Fibers.PoolFiber.Flush(Object ) in d:\dev\exitgames-libs\src\Core\Concurrency\Fibers\PoolFiber.cs:line 216
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

And here is the method: (I added the try statement after I got the exception so at least other functions of the server don't go down with this method if it ever happens again)

Code: Select all

public static void AddPlayerEntity(UnityClient unityClient)
        {
            try
            {
                if (unityClient.currentMapPos[0] == 0.00f)//vandellia
                {
                    Vector3 init_pos = new Vector3(unityClient.currentMapPos[1], unityClient.currentMapPos[2],
                        unityClient.currentMapPos[3]);

                    unityClient.compactCollider = new Cylinder(init_pos, 1.5f, 0.2f);
                    unityClient.extendedCollider = new Box(init_pos, 1.5f, 0.5f, 0.8f);
                    unityClient.grabCollider = new Box(init_pos, 1f, 1f, 1f);

                    //offset local positions so that the colliders adhere to the "imaginary ground"
                    unityClient.compactCollider.CollisionInformation.LocalPosition = new Vector3(0, 0.3f, 0.05f);
                    unityClient.extendedCollider.CollisionInformation.LocalPosition = new Vector3(0, 0.4f, 0.7f);

                    //init orientation
                    unityClient.extendedCollider.Orientation = new Quaternion(unityClient.currentDir[0], unityClient.currentDir[1], unityClient.currentDir[2],
                        unityClient.currentDir[3]);

                    //set tags of colliders for recognition in collisions
                    unityClient.compactCollider.Tag = "[PLAYER] [body] " + unityClient.cache_charname_enter;
                    unityClient.extendedCollider.Tag = "[PLAYER] [weapon] " + unityClient.cache_charname_enter;
                    unityClient.grabCollider.Tag = "[PLAYER] [grab] " + unityClient.cache_charname_enter;

                    //enable collision on colliders
                    unityClient.compactCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;
                    unityClient.extendedCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;
                    unityClient.grabCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;

                    //initialize movers/rotators
                    unityClient.charCompactMover = new EntityMover(unityClient.compactCollider);
                    unityClient.charExtendedMover = new EntityMover(unityClient.extendedCollider);
                    unityClient.charGrabMover = new EntityMover(unityClient.grabCollider);
                    unityClient.charExtendedRotator = new EntityRotator(unityClient.extendedCollider);

                    #region map stuff
                    //spawn the character in the physics map
                    if (unityClient.currentMapPos[0] == 0.00f)
                    {
                        VandelliaPhysicsMap.Add(unityClient.compactCollider);
                        VandelliaPhysicsMap.Add(unityClient.extendedCollider);//this is line 88
                        VandelliaPhysicsMap.Add(unityClient.grabCollider);
                        VandelliaPhysicsMap.Add(unityClient.charCompactMover);
                        VandelliaPhysicsMap.Add(unityClient.charExtendedMover);
                        VandelliaPhysicsMap.Add(unityClient.charGrabMover);
                        VandelliaPhysicsMap.Add(unityClient.charExtendedRotator);
                    }
                    #endregion
                }
            }
            catch (Exception e)
            {
                ILogger Log = LogManager.GetCurrentClassLogger();
                Log.Debug(e);
                unityClient.Disconnect();
            }

I commented where line 88 is in the code.

Now get this, As soon as my player is added to the game with this method, the update function stops working totally. This is why everything froze in game. As soon as the player leaves the map, the update function begins working again. I checked using a log file. Something is wrong with the above method, I am just not sure what.
Also note the exception I mentioned earlier does not coincide with the update function freezing randomly. The two occured on two separate occasions at seemingly random times.

Here is my update function. I used a timer that starts an event every 16 ticks which is about 60 fps by my calculations.
The timer is enabled when the server starts in a separate class. This is a System.Timer.

Code: Select all

public static Timer updateFrame = new Timer(16.666666666666666666666666666667);
public static void Update(object sender, EventArgs e)
        {
            VandelliaPhysicsMap.Update();
            foreach(MonsterType m in MonsterHandler.AllVandelliaMonster.Values)
            {
                if (m.blood > 0)
                {
                    m.Update();
                }
            }
        }

As for the m.Update() method, that is to update the monsters position along the curve that was constructed by the ai script. Here it is:

Code: Select all

public void Update()
        {
            if (wp_move.Enabled)
            {
                pathTime += compactCollider.Space.TimeStepSettings.TimeStepDuration;
                compactMover.TargetPosition = positionPath.Evaluate(pathTime);
                extendedMover.TargetPosition = positionPath.Evaluate(pathTime);
            }
        }

[Norbo]

It looks like there is some unsafe asynchronous access going on. This is likely why you see 'random' bugs- there are race conditions.

Treat any potentially simultaneous access to shared resources as a bug until there is a proof otherwise.

In particular, with very few exceptions:
-Do not modify the state of the Space, its stages, or objects within the Space while the Space is simulating.
-Do not modify the state of anything from multiple threads without guaranteed exclusive access. This includes adding and removing things from the Space.
-Even with locks or other exclusive access guarantees, be careful with ordering. For example, the SpaceObjectBuffer can be used to queue adds and removes from other threads, but it's easy to end up putting adds and removes out of order when using multiple threads. Adding an object twice or removing an object that isn't there will cause an exception.

These may not be the source of all of the observed problems, but until they are fixed, it could be very difficult to reason about the causes of other bugs.

[jaja1]

In particular, with very few exceptions:
-Do not modify the state of the Space, its stages, or objects within the Space while the Space is simulating.
-Do not modify the state of anything from multiple threads without guaranteed exclusive access. This includes adding and removing things from the Space.
-Even with locks or other exclusive access guarantees, be careful with ordering. For example, the SpaceObjectBuffer can be used to queue adds and removes from other threads, but it's easy to end up putting adds and removes out of order when using multiple threads. Adding an object twice or removing an object that isn't there will cause an exception.

With regards to point 1: Could you elaborate further on how I can achieve this. I can't quite see how I can not manipulate the space while the space is simulating. I will always have to add objects and modify them as the world is very dynamic.

EDIT: Point 2/3: I managed to find some docs that are helping understand the concept of threads. I also realized now that the sdk for my server is multithreaded. This could be the reason why I am getting race conditions (if this is the cause of the bugs). Also, is BEPUphysics multithreaded? And should I use a memory barrier when adding and removing entities from a space? And last, which should I remove first, a mover/rotator or the entity associated with it?

[Norbo]

With regards to point 1: Could you elaborate further on how I can achieve this. I can't quite see how I can not manipulate the space while the space is simulating. I will always have to add objects and modify them as the world is very dynamic.

By 'while the Space is simulating', I meant 'while some thread is in the middle of processing the Space.Update function'. Changes to state should only be made while no thread is in Space.Update. Changing state between updates is fine.

A few examples:
1) Perform the state changes from the same thread as the Space.Update is called from.
2) Use synchronization to ensure that the space isn't currently updating. Watch out: in this model, it's easy to stall your state-changing thread for many milliseconds waiting for the Space to finish updating.
3) Use a synchronized queue of state changes which is fed by other threads. The queue is flushed and its changes are applied by the thread responsible for Space.Update between calls.

Also, is BEPUphysics multithreaded?

It is if the Space is given a IParallelLooper (with more than one thread in it) in its constructor. If the empty constructor is used, it defaults to single threaded execution.

Those threads are just used by the space within the Space.Update function to speed things up. They do not keep running outside of the Space.Update. Under normal conditions, their use is transparent; they have no effect on correctness because they are isolated from view. (There is one exception to this: events which are fired from within the internal threads, like immediate (present tense named) collision events and Entity.PositionUpdated. The implementation of the attached events must take this into account.)

And should I use a memory barrier when adding and removing entities from a space?

If multiple threads are adding and removing entities simultaneously, some form of exclusive access is needed, like a lock statement. Memory barriers alone do not provide exclusive access.

Note that the lock statement in C# has full memory barriers built in on entry and exit; you don't need to worry about including them if you use it.

And last, which should I remove first, a mover/rotator or the entity associated with it?

By convention, I'd say remove the mover/rotator first. But nothing should go wrong if the entity is removed first.

[jaja1]

Once again thanks for the major help Norbo! I took your advice and I am finding myself having to deal alot less with these race conditions and "random" bugs. I also started a thread on my server sdk's forum and figured out how to solve some of the issues I was facing. I am now using a dedicated Fiber for BEPUphysics which several threads work on but in queue, so that anything that does not belong to BEPU must be queued through the fiber and SHOULD never cause any "out of turn" access. It also improves the overall ordering of requests to my server because the client no longer waits for a request to be acknowledged by the server, but rather it just be queued in the fiber and a response is sent to the client whenever those calculations are done.
I also found locks to be the most convenient (and effective) way to prevent race conditions from occurring WITHIN BEPUphysics and its members.

[jaja1]

Sorry to post on a slightly old thread, but the issue came up again and I just can't seem to narrow down the cause. Let me explain exactly what is going on.

The exception is a null reference exception. Here is my call stack and the exception:

Code: Select all

2015-01-10 23:06:09,421 [17] DEBUG MukizuServer.CharacterGate [(null)] - System.NullReferenceException: Object reference not set to an instance of an object.
   at BEPUphysics.BroadPhaseSystems.Hierarchies.InternalNode.TryToInsert(LeafNode node, Node& treeNode)
   at BEPUphysics.BroadPhaseSystems.Hierarchies.DynamicHierarchy.Add(BroadPhaseEntry entry)
   at BEPUphysics.Space.Add(ISpaceObject spaceObject)
   at MukizuServer.PhysicsHandler.AddPlayerEntity() in c:\Users\jaja\Documents\Visual Studio 2010\Projects\MukizuServer\MukizuServer\PhysicsHandler.cs:line 128
2015-01-10 23:06:19,003 [21] DEBUG Photon.SocketServer.ApplicationBase [(null)] - OnDisconnect - ConnID=0
2015-01-10 23:06:19,034 [17] DEBUG MukizuServer.UnityClient [(null)] - 127.0.0.1 has disconnected.
2015-01-10 23:06:19,096 [13] DEBUG MukizuServer.SaveCharData [(null)] - System.NullReferenceException: Object reference not set to an instance of an object.
   at MukizuServer.PhysicsHandler.RemovePlayer() in c:\Users\jaja\Documents\Visual Studio 2010\Projects\MukizuServer\MukizuServer\PhysicsHandler.cs:line 155

The AddPlayerEntity is called via a task within a Fiber pool. The same Fiber pool also schedules an update function that updates the Spaces I am adding entities too. Since both methods are called under ONE static "PoolFiber", I highly doubt entities are being added while the space is updating. This is the AddPlayerEntity method:

Code: Select all

public void AddPlayerEntity()
        {
            try
            {
                float[] pos = playerPhysicsProfiler.init_pos;
                float[] dir = playerPhysicsProfiler.init_dir;

                Vector3 init_pos = new Vector3(pos[0], pos[1], pos[2]);

                playerPhysicsProfiler.compactCollider = new Cylinder(init_pos, 1.5f, 0.2f);
                playerPhysicsProfiler.extendedCollider = new Box(init_pos, 1.5f, 0.5f, 0.8f);
                playerPhysicsProfiler.grabCollider = new Box(init_pos, 1f, 1f, 1f);

                //offset local positions so that the colliders adhere to the "imaginary ground"
                playerPhysicsProfiler.compactCollider.CollisionInformation.LocalPosition = new Vector3(0, 0.3f, 0.05f);
                playerPhysicsProfiler.extendedCollider.CollisionInformation.LocalPosition = new Vector3(0, 0.4f, 0.7f);

                //init orientation
                playerPhysicsProfiler.extendedCollider.Orientation = new Quaternion(dir[0], dir[1], dir[2], dir[3]);

                //set tags of colliders for recognition in collisions
                playerPhysicsProfiler.compactCollider.Tag = "[PLAYER] [body] " + charStatsProfiler.char_name;
                playerPhysicsProfiler.extendedCollider.Tag = "[PLAYER] [weapon] " + charStatsProfiler.char_name;
                playerPhysicsProfiler.grabCollider.Tag = "[PLAYER] [grab] " + charStatsProfiler.char_name;

                //enable collision on colliders
                playerPhysicsProfiler.compactCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;
                playerPhysicsProfiler.extendedCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;
                playerPhysicsProfiler.grabCollider.CollisionInformation.CollisionRules.Personal = BEPUphysics.CollisionRuleManagement.CollisionRule.Normal;

                //initialize movers/rotators
                playerPhysicsProfiler.charCompactMover = new EntityMover(playerPhysicsProfiler.compactCollider);
                playerPhysicsProfiler.charExtendedMover = new EntityMover(playerPhysicsProfiler.extendedCollider);
                playerPhysicsProfiler.charGrabMover = new EntityMover(playerPhysicsProfiler.grabCollider);
                playerPhysicsProfiler.charExtendedRotator = new EntityRotator(playerPhysicsProfiler.extendedCollider);

                #region map stuff

                if (playerPhysicsProfiler.currentMap == 0.00f)//vandellia
                {
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.compactCollider);
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.extendedCollider);//THIS IS LINE 128
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.grabCollider);
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.charCompactMover);
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.charExtendedMover);
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.charGrabMover);
                    VandelliaPhysicsMap.Add(playerPhysicsProfiler.charExtendedRotator);
                }
                else return;

                #endregion

                isPlayerAdded = true;
                return;
            }
            catch (Exception e)
            {
                Log.Debug(e);
                return;
            }
        }

I commented in the location of line 128.
Also, here is where I implement the task to call this method.

Code: Select all

PhysicsHandler p = new PhysicsHandler();
                p.uClient = uClient;
                p.charStatsProfiler = charProfiler;
                p.playerPhysicsProfiler = physProfiler;
                Task t = new Task(p.AddPlayerEntity);
                PhysicsHandler.PhysicsFiber.Enqueue(t.Start);
                t.Wait();

I created a new instance of the physics handler class to call the method and avoid race conditions (apparently did not work). However, I state againt the fiber pool is static.

The second exception is called on this method:

Code: Select all

public void RemovePlayer()
        {
            PlayerPhysicsProfiler pprof = uClient._playerPhysicsProfiler;
            try
            {
                pprof.charCompactMover.Space.Remove(pprof.charCompactMover);//THIS IS LINE 155
                pprof.charExtendedMover.Space.Remove(pprof.charExtendedMover);
                pprof.charGrabMover.Space.Remove(pprof.charGrabMover);
                pprof.charExtendedRotator.Space.Remove(pprof.charExtendedRotator);
                pprof.compactCollider.Space.Remove(pprof.compactCollider);
                pprof.extendedCollider.Space.Remove(pprof.extendedCollider);
                pprof.grabCollider.Space.Remove(pprof.grabCollider);
                pprof = new PlayerPhysicsProfiler();

                if (SaveCharData.isWaitingToSwitchChar)
                {
                    OperationResponse response = new OperationResponse((byte)15);
                    response.Parameters = new Dictionary<byte, object> { { 15, null } };
                    SendParameters sendParameters = new SendParameters();
                    sendParameters.Unreliable = true;
                    uClient.SendOperationResponse(response, sendParameters);
                }

                return;
            }
            catch (Exception e)
            {
                Log.Debug(e);
                return;
            }
        }

Once again I commented in the location of line 155.
Here is where I queued the method: (I did not use Task.Wait() in this case because I had no code to call after the method had completed. In retrospect I'm thinking it was still necessary? )

Code: Select all

PhysicsHandler p = new PhysicsHandler();
            p.uClient = uClient;
            PhysicsHandler.PhysicsFiber.Enqueue(p.RemovePlayer);

It is one of those "random" errors so I guess its another race condition. But what is the exact cause of this one is what I can't seem to find. The strange thing about it too is that the exception is always called on these exact same lines. The exception may happen randomly (and too frequent a case to simply ignore), but the same lines always bug out.

I also have another "random" issue on a monster entity when it dies and respawns. I can't provide an debug log because there is no error...client side nor server side. So for now I will assume it is a logical error and if the previous issue is solved I will try to tackle this one with new knowledge.
EDIT: It was a logical error so the monster issue is fixed....the problem mentioned above about the player however still persists. I did try locks.

So, is there any further advice you can give me on this problem Norbo?

[Norbo]

The only times I've ever seen that exception in the DynamicHierarchy are when there is some threading issue in play. The most likely source is adding things to the Space from multiple threads in an unprotected way.

Unfortunately, my ability to remotely debug threading issues is extremely limited. I would just suggest simplifying the relevant code and test as much as possible until you have the bare minimum to reproduce the bug. Even if it doesn't immediately find the problem, it should make it easier to reason about.