Did you verify that the worm and sector were centered with the wheels straight ahead before removing it. Remember the play increases between the two as you move away from lock to lock center.
Also keep in mind a little play out of the car, becomes A LOT when a 16" steering wheel is introduced

And no, the LCA pivots should not be moving side to side as you turn the wheels. There was a post on that recently and the repair method generally used. Maybe someone will point you to it. keep us posted