I've been painting with Rustoleum pro outside and have also had some issues with dirt specks and pollen in the paint.

I've noticed that if the paint is just the least bit tacky when the dirt hits it, it sort of sits up on top of the paint and doesn't really get "in" the paint. When the paint sets up the dirt and muck is pretty easily removed without any flaws in the paint. Usually don't need sandpaper to do it, just wet it all down and run my hand over it or a fingernail to remove it. It hasn't been a real problem.