Are you still using the original style idler arm that has the single bracket for the mount? May want to switch to the later model and add that top brace with the thru bolt instead of the single bracket. My Bro n law had one of those early A bodies and that idler arm would move all over the place.
Let me try to find my old Mopar manual that shows how to modify the brackets so the idler arm, pitman arm and both tie rod ends are the same level on the drag link.