I wonder if you are actually seeing wear on the cam or just the polished area where the lifter rides. Make sure to put a sealer on the intake bolts. If you look you can see that they go into the heads then into air space in the lifter valley. Very common for oil to walk its way up the threads and pool in the valley tin. Oil deposits could also becoming from guide wear.