In general, I agree that if the scrub path is small and a safe distance away from the stem tip's edge I'd leave it alone.
The roller diameter is a bit of a design puzzle. Most mfg. have settled on a 3/8" OD roller as a reasonable compromise between 2 extremes:
1. A really small roller (like 1/4") punishes the stem tip, and has very few needle bearings actually loaded = early failure. Only a few needles at the top of the roller take the entire load, and they don't actually rotate - they oscillate a few degrees.
2. A really big roller (like 1/2") lives longer, loads many more needles, and locally depresses the stem less in psi (yes, I know in theory it's only line contact!), but that's the worst possible place to add both weight and inertial resistance - the roller must reverse rotation instantly or it scuffs across the stem.

Roller tips effectively increase the stem height (compared to pallet ends, as far as geometry is concerned) by their radius, since the axle is the "end" of the stem.
When you run a big roller, the rocker doesn't change (as to geo), but now the valve is too long.

Warren Johnson played with geo on his "stock" Olds 350 motors 100 years ago.