Well I think I found out what it was... Should have been the first thing I should have checked!!!

The wheels were toed OUT, only slightly but but about a 1/16-1/8.

I toed them back in and it is MUCH better, need to get a real alignment now!

The steering box was definitely loose, which was a factor, or maybe a symptom, and it might never have been completely tight.