My experience with the xe hyd cams in bb mopars, especially with aftermarket rockers is, they are noisier than many solid lifter grinds, and the noise definitely isn't as uniform.
The sound is not particularly confidence inspiring, but that's how they are.
Just run it and live with the noise, or change to a quieter lobe design if you can't get past the noise.

Btw, the upper rpm limit of those lobes with std lifters and the springs you have is somewhere around 5500rpm. It might go a little higher, but if you feel it stop pulling, it started getting unhappy a few hundred rpm lower than where you felt it flatten out.
If you run it higher than that, don't be surprised if you start having issues.

As a side note....... In most cases the noise isn't usually coming from the lifters themselves.
I think if you go to the Rhodes lifters website and look through their info, they have a good description of where the noise comes from when the lifters are are in a somewhat collapsed state.