For what it's worth I've had both shoes and drums cause a pull. Even with a very nice finish on both drums that were cut to the same diameter. Switching the shoes or drums from side to side moved the direction of the pull. I would agree in your case to check the wheel cylinders,shoes,and drums since the pull goes away with use or temperature change. I would eliminate suspension,tires,return springs in your case. Another factor over looked are the backing plate pads. I have welded grooved plates and caliper mounts where they are worn.
Doug