var lineDisplayObject = displayObjects[i] as LineDisplayObject<T>;
if (lineDisplayObject != null && lineDisplayObject.LineObject.Equals(o))
despite displayObjects is not null, the resulting cast (not sure of the desired goal here) in lineDisplayObject is always null (at least in my tests), so lines are actually not removed.
Included in patches and sources for both DemoGame (add LineDrawer to demo constructor + a few change in demo order, some unrelated) and RobotArmDemo to show the issue. Enable constraints display, press 0 on the numpad and see what happened, moving arms part could also show it ore obviously.
There was a bug in the handling of SolverGroups in the remover. It should be fixed now (though I'll admit I didn't do much testing). Thanks for the report!