People forget that when new, the factory installed the cam bearings then actually "line bored" the whole she-bang. New bearings take you back to square one.
I've had this same situation happen. I pulled out a '509 to put in a Lunati Solid and while the '509 fit easily and spun with almost no resistance, the Lunati would not fit. I had to make THIS:

Cam bearing tool 2.jpg