You more than likely have a bent lower control arm. The problem is unless you now how to recognize a bent lca you won't see the damage. I may still have a really nice looking bent one to show you a pic. Let me go look.
The lca cast-machined part that the ball joint attaches to bends.