The oldest trick in the book is to install the bearings and then use an old cam with a notch on each journal to scrape the new bearings to fit. This works like a charm and would definitely solve your problem.

I also don't think there's anything wrong with your cam bores, the issue is with the bearing manufacturers not making the bearings to fit properly. The factory never had any problems and they didn't do anything to the bearings once they were installed !!

Another popular theory is "core shift" in the block. But core shift occurs during casting. The machining process would still yield straight bores.

Try the cam trick !!