I asked the same question once. The guy used to work in a body shop, and he said that the sanding, even with a very high grit, makes tiny scratches on the paint, and light reflection is different than in the no "scratched areas". when you go in circles or swirls, those scratches are going to reflect light in all directions, not matter where you are looking at the car. It is going to look like a shiny "spider web". In the other hand, when you go in straight lines, the reflections from those scratches, it is going to match with the reflections on the panel if they are parallels. if the reflection is perpendicular to the reflection of the panel, it is going to go in a different directions but not to the observer, so the scratches are less evident.
I hope my explanation is understandable.
To make it easy, It is the same when you are next to a pond and you throw a stone in it. the reflection of light is going to go to your eyes wherever point you are standing at. But when there is waves, you can see the reflection just when you are in the direction of source of light. But you can't see it when you are perpendicular to the source of light.