Just to update on this...

A few weeks back I jack the car up to pull the gearbox out and see whats going on. After I pull the driveshaft I look up and see one of the trans to bellhousing bolts is missing, and the others are loose.

There was almost a 1/4" gap opened up between the bell and trans. I find another 7/16" bolt, tighten them up, stick the driveshaft back in and test drive it. No more popping out of forth.

I should have checked but it didnt even occur to me that it would be something like that.
Now I'm sure the input shaft being at an angle like that didn't do anything any good, but I've been driving it since then a lot and it seems fine so far.