If you have two “V” blocks place the cam on them using #1 and 5 cam journals then check the center bearing for runout.
Compare the cam that fits nicely to the one that does not. Your theory holds water.

Of course you want to make sure the journals are very close in diameter between the cam that fits and the one that does not.