The mass is more significant as it moves away from the center of rotation. This inertia effect increases with the square of the radius. But, the negative effect of inertia is perportional to change of rotational speed (acceleration).
Anyone that spends big dollars on gun drilled axles will likely be disappointed and wont pick up more than the actual weight removed.
Going from new street tires to close to bald (may drop quite a bit of weight in some cases) is weight where it is more significant. A pound of tread verses a pound of lug nuts has 38 times more effect one the overall inertia of the wheel/tire (given a 28 inch tire. and 4.5 in. bolt circle). In fact, the total weight of a wheel/tire combo could be less than another but still result in more loss due to increased inertia.
Running a smaller/lighter converter or damper (or even paying the extra to have the crank turned down when you have a bunch of material to remove) could payoff.