From what I have seen over the years....... The Schubeck/tool steel type lifters are not the hot ticket for "street" cars.
I know of numerous people who have been through this.

After you get the cam out, if you can, get some close up pics of the nose of the lobes, so I can see if they look like others I've had in the shop.

When it's a real street car(more driving, less racing), and won't be running a roller cam...... My preference is not to use real aggressive lobes, get the cam nitrided, and use normal edm lifters.
And of course high zinc oil....... And I'm not in the "light oil" camp.

I prefer the Driven or Brad Penn myself.

Edit: I didn't see the cam pic in the first post initially.
The wear and pitting on the nose like you have is what I usually see when using this style of lifter on iron cam cores when "normal" cam profiles are used.
"Cheater", or dwell at full lift type lobes seem to hold up much better when this style of lifter is used, even with much higher spring loads.
Nitriding doesn't help at all.

I've never gotten a good answer on what's happening, but I have a theory.

It's not "defective" parts, and I don't think different oil would solve it.