Holley carbs never have been my choice of a go to carburetor, but it's what my kid bought for his 440.
Which port is the ported vacuum for the vacuum advance? Seems like they all have full manifold vacuum on them when I put my thumb over them.

Should be in the manual. If it is the same as a QF, the one on the outside is full vac and the inside one is ported vac. If it has a nipple on the primary metering block, it is ported vac, but pulls less vac than the ported vac port on the base plate (at idle, with my setup anyway).

If you want no vac advance at idle, I'd day the metering block port.

IDK where my kid bought this carb but it was brand new in the box when he did, if it wasn't Summit/Jegs it was a similar type place, though he got it online.... wasn't from a private individual.... but there was no "manual" of any kind among the paperwork in the box, just a reference to "Holley.com". on the outside of the box..... I prefer the "old way" of doing most things..... I'm not a "dot com" kind of person.

You're on a "dotcom" now. Just google search holley street avenger manual and look for the pdf to read or download. Mine came with a manual, the one he bought may have been a return, display item, or simply missed getting a manual going down the packaging line. The "old way" to me is taking it to a shop and having them fix it, so i prefer the new way.

If the timed port has a vacuum on it with then engine at slow idle, then the throttle is open too far.
Reduce the throttle stop, and if need be, turn the distributor for a degree ot two more initial timing. That alone will bring up the idle speed enough on stock or mildly hot rodded engine.