How do you know the shaft was made with the correct diameter or even the correct taper? If that shaft is too small it could definitely pull on the ball before it makes contact with the arm. Castle nuts are designed to have the cotter pin go through the towers, not to let it loosen a couple of turns before it contacts the towers. All I'm saying there's something not right there.
Here's a suggestion, why don't one of you guys buy the tie-rod off the OP and install it on your car? We'll see how many of you are willing to do that.