Really interesting problem.

I'm inclined to also think you're running lean, or on the ragged edge at idle. (hate those step up rods) I've had problems with throttle shafts leaking, and screwing up idle quality. Other than that, you're obviously competent enough to know if the base of the carburetor is warped, or the gasket not sealing well. If you spray carb cleaner around the base of the carb while idling does anything happen?

The air/fuel mixture density in the combustion chamber will be quite low compared to when the throttle is opened appreciably. That type of mixture tends to burn more slowly, and in my experience was a primary reason for the vacuum advance, which you say you don't have connected.

I know I struggled with a warmed over 331 SBC years ago that had a Mallory dual point in it, with no vacuum advance. Played with springs, weights, on and on, and things really only got better when I installed a distributor with a vacuum advance. It really cleaned up with idle/off idle characteristics of the engine.