Infinite Update function

[nikolaiko]

Hi.

I faced the following problem. I am using separate thread to update the physical world, running this method in separate thread :

Code: Select all

public void Run() {
            try {

            Stopwatch Timer = new Stopwatch();
            Timer.Start();

            IsRunning = true;
            while (IsRunning)
            {

                Log.DebugFormat("Updating physics thread");

                    if (Timer.Elapsed < TimeSpan.FromSeconds(1/30f)) {

                        if (ThreadRegionServer.NumPlayers <= 0) {

                            Thread.Sleep(1000);
                            Timer.Restart();
                        }
                        
                        if ((int) (1000.0f/30.0f - Timer.Elapsed.Milliseconds) > 0) {
                            Thread.Sleep((int)(1000.0f / 30.0f - Timer.Elapsed.Milliseconds));
                        }
                        continue;
                    }
                    
                    Update(Timer.Elapsed);
                    Timer.Restart();
                }
                               
            }
            catch (Exception e)
            {
                Log.DebugFormat("Exception in physics thread : {0}", e.ToString());
            }            
        }

And this Run method simply call :

Code: Select all

private void Update(TimeSpan Elapsed) {
            try {
                Log.DebugFormat("BEFORE Updating physics");
                Space.Update();
                Log.DebugFormat("AFTER Updating physics");               
            }
            catch (Exception ex) {
                Log.DebugFormat("Physics update exception {0}", ex.ToString());
            }
        }

After some time of playing everything on client is freezing, I tought - server died. No. Messages between client-server still going. I found that objects doesn't moving because server didn't send events about coordinates change. It happends because at some time server went into Space.Update() and after that never returns (Because I see : BEFORE Updating physics, but didn't have : AFTER Updating physics after it, never). So whole physics update thread hanging in Update function. Any reasons for such behaviour? I doing something totaly wrong?

Thanks.

[Norbo]

The first suspect is event handlers. They're going to be called from the physics thread. Any interthread communication or data access within those event handlers is dangerous by default; a tiny mistake could deadlock it.

But there's no need to guess- attach the debugger and break into the process when it appears to freeze. Then, check the parallel stacks debug window (at least that's what it is called in VS2010). The physics thread's call stack will show exactly where it's stuck.

[nikolaiko]

Hm, will try to debug it, and wrote about results, but anyway, you told what I need to call event handlers from same thread as call Update method?

[nikolaiko]

I added BEPU sources to my projects, added some trace to Space.Update and found that we going into Solver.Update -> protected override void UpdateMultithreaded(). So, somethere here we are stucking.

Code: Select all

         protected override void UpdateMultithreaded()
        {
            ParallelLooper.ForLoop(0, solverUpdateables.Count, multithreadedPrestepDelegate);
            //By performing all velocity modifications after the prestep, the prestep is free to read velocities consistently.
            //If the exclusive update was performed in the same call as the prestep, the velocities would enter inconsistent states based on update order.
            ParallelLooper.ForLoop(0, solverUpdateables.Count, multithreadedExclusiveUpdateDelegate);


            ++PermutationMapper.PermutationIndex;
            ParallelLooper.ForLoop(0, iterationLimit * solverUpdateables.Count, multithreadedIterationDelegate);
        }

Working on it.

[Norbo]

but anyway, you told what I need to call event handlers from same thread as call Update method?

Sorry, I'm not sure what the question means. Event handlers triggered by BEPUphysics will be called from whatever thread executed them. This could be the physics thread or, in the case of immediate events and multithreading, a physics worker thread.

Solver.Update -> protected override void UpdateMultithreaded(). So, somethere here we are stucking.

That's a very strange place to get stuck. There are no events fired from within the solver. The only potential for a deadlock would be in something external making use of the Entity.Locker object improperly. If it is truly stuck somewhere in the solver, I'd begin to suspect some strange platform or environmental issue.

Using the Parallel Stacks debug window would be very helpful in narrowing down.

[nikolaiko]

Hi, again. It's in ParallelLooper->ForLoop it stucks in loopFinished.WaitOne(); because waiting for signal from other thread. Continue working.

The problem in debugging what I am not starting it as program. My server is .dll. I build it and then using PhotonServer to start it. So I can't to debug it directly. I trying to attach to already started process, but then I didn't see my physical thread in list, sometimes. Maybe I just don't know how to work with it properly, so I keep working on it too.

Thanks for your assistance.

[nikolaiko]

Hmm it's strange for me. In the ForLoop of ParallelLooper there is a comment : //CANNOT CALL THIS WHILE BUSY!!!! ASSUME THAT IS GUARANTEED.. But in the same time then we create space only one Looper instance is used and it passed to many other objects : Solver, ForceUpdater and others. I guess they all use ForLoop async, so I think we have ForLoop call then it busy. Or not?

[Norbo]

Space stages run strictly in sequence. Each one has fully exclusive access to the ParallelLooper; there is no danger of stages stomping on each other within a single Space.Update call.

If different threads use the same ParallelLooper without guaranteeing that it is not busy- for example, two Space objects being updated asynchronously but using the same ParallelLooper- that would indeed break things.

[nikolaiko]

Ok, I found the reason (or at least 100% sight of deadlock).

Then player walking around - it's ok. Then I start shoting rockets, for each rocket I create CharacterController (same as for player) with body, then set :

Code: Select all

Controller.HorizontalMotionConstraint.MovementDirection = new Vector2(Orientation.X * Physics.GetMoveSpeed(), Orientation.Z * Physics.GetMoveSpeed()); 

to give rocket a direction for moving. After that I keep counting time inside rocket class, and destroy it after some time past. So, what happends...some times right after creation class : EntityBase throws exception in function :

Code: Select all

public void ApplyLinearImpulse(ref Vector3 impulse) {

                linearVelocity.X += impulse.X*inverseMass;
                linearVelocity.Y += impulse.Y*inverseMass;
                linearVelocity.Z += impulse.Z*inverseMass;
                linearVelocity3 = linearVelocity;
                MathChecker.Validate(linearVelocity);                  
        }

at the line :

Code: Select all

MathChecker.Validate(linearVelocity);

because linearVelocity values : NaN. That happends because impulse (function parameter - NaN). I Checked stacktrace. Function ApplyLinearImpulse called from : HorizontalMotionConstraint, function : ExlusiveUpdate :

Code: Select all

public override void ExclusiveUpdate()
        {
            //Warm start the constraint using the previous impulses and the new jacobians!
            Vector3 impulse;
            Vector3 impulse1;
            Vector3 torque;

            float x = accumulatedImpulse.X;
            float y = accumulatedImpulse.Y;
            impulse.X = linearJacobianA1.X * x + linearJacobianA2.X * y;
            impulse.Y = linearJacobianA1.Y * x + linearJacobianA2.Y * y;
            impulse.Z = linearJacobianA1.Z * x + linearJacobianA2.Z * y;

           characterBody.ApplyLinearImpulse(ref impulse);
          
            if (supportEntity != null && supportEntity.IsDynamic)
            {
                Vector3.Multiply(ref impulse, -supportForceFactor, out impulse);

                x *= supportForceFactor;
                y *= supportForceFactor;
                torque.X = x * angularJacobianB1.X + y * angularJacobianB2.X;
                torque.Y = x * angularJacobianB1.Y + y * angularJacobianB2.Y;
                torque.Z = x * angularJacobianB1.Z + y * angularJacobianB2.Z;


                supportEntity.ApplyLinearImpulse(ref impulse);
                supportEntity.ApplyAngularImpulse(ref torque);
            }
        }

Impulse is NaN, because : linearJacobianA1 is NaN too (I traced them). I went further and found linearJacobianA1 set in Update function. And for some reason, this function for some Rockets have movementDirection3d = 0,0,0 (but I set HorizontalMotionConstraint.MovementDirection and most of the rockets fly normally). According to this, program went in if branch for !tryToMove condition and at the end of it we have linearJacobianA1 - NaN. The other Rockets have normal value of movementDirection3d and have normal values in movementDirection3d.

After a few cycles with NaN values all update is stuck.

I am trying to find out there is my mistake. I guess in rockets creation, I think I need to sync it with physical engine or something like this. But still didn't get how it all works.

[Norbo]

I am trying to find out there is my mistake. I guess in rockets creation, I think I need to sync it with physical engine or something like this. But still didn't get how it all works.

If rocket creation and movement management is on a different thread than the physics engine update without proper synchronization, yes, that would cause a lot of different kinds of problems.

An easy rule is that no thread except the physics update thread should affect the physics engine.

If a non-physics thread wants to create a new object, rather than creating it directly it sends a message to the physics thread. If a non physics thread wants to change the properties of a physics objects, it sends a message to the physics thread rather than doing it itself.

Personally, I avoid interacting asynchronous systems whenever possible. They introduce a lot of complexity for every simple task. Often, I just update the physics from the main thread and ensure every stage takes advantage of the whole CPU. In the worst case, the rendering thread might be separate, since pre-DX12 it's sometimes difficult to get good multithreaded CPU utilization out of a renderer. On the upside, a renderer is a read-only system. Asynchronous game logic can expose its information to the renderer in thread safe ways, like a double buffer with lock on buffer flipping.

Code: Select all

Controller.HorizontalMotionConstraint.MovementDirection = new Vector2(Orientation.X * Physics.GetMoveSpeed(), Orientation.Z * Physics.GetMoveSpeed()); 

Note that the MovementDirection is normalized. Setting it to a scaled direction just results in the scaling being removed. If you want to change the speed of a character, adjust the CharacterController.StandingSpeed and related properties.

[nikolaiko]

Ok, I moved creation and deletion of objects in physics thread and seems like problem is gone. But I still have some object properties change in other threads (movement direction).
Thanks.