I can see part of your problem in your pic. First is
the distance from the centerline of the ball joint to
the centerline of the tire.... we will say thats 3"
and as you apply the brakes it wants to torque back
and now it has a 3" moment arm. We all use wheels that
tend to hang past the ball joint centerline. Now look
at the distance from the steering arm to the centerline
of the outer tie rod end with that spacer in there....
with the bolt. I know that bolt isnt a press fit
into both parts. As the brakes apply the tire tries
to move rearward torquing on the steering arm and
finding the slop in the bolt to the sreering and
heim plus the distance between the 2 parts