Give your buffing pads a good clean by soaking them in a bucket with either the proper buffing pad cleaner or dishwashing liquid, then rinse them in clean water until they look like new.

Wash those two panels with dishwashing liquid instead of a car wash solution and hit those panels with your swirl remover again. Use a high speed (3/4 to full speed)setting on the polisher and really work the polish in until there is hardly any residue left on the panel.

If that clears it up then do a couple of passes with the polish and you should be good. If it doesn't clear it up you might have to try a lighter grade swirl remover to help clean it up.