When you tear the enging down you will see the metal from the cam got to all the bearings and oil pump
I thought the same as you to just put a new cam in, but when I checked the oil pump and bearings I ended up doing a minor rebuild. Had the block re-honed, then replaced the piston rings, all the bearings, the oil pump, cam and lifters.