Yes, manifold will need to come off, and break out the chisels. Does the manifold get warm (hot) at the crossover area soon after cold start?
Before you do that check to see if the choke pull off is perhaps pulling the choke open too far, this would give you choke when cold, but no enrichment for cold run, soon after start.
If it is the pull off, adjustment is done by carefully bending the link.
Guessing, is this a 318 2v?
My '85 has a transplanted 318 4v, thermoquad, and it took a lot of fooling around to get things working properly, and I'm still not sure ...