It really does sound like it is coming from the mirror, but as you speed up the sound moves to the center of the windshield.

Also, putting your hand out the window and blocking airflow to the mirror has no effect.