Here is a point of reference for you:

528hemi, 270advertised 235 @.050 solid flat tappet 108 LSA, 10.0:1 compression. Cam installed dot to dot. Factory intake with cloned Carter-brocks. 16in vacuum at 750 rpm idle.

Mine acted like the way you are describing when I first put it together, I had a massive vacuum leak. Good luck with your Hemi.