Depending on what anchors you use i would go at least 3 inches down. Typical wedge anchors need some room to catch and expand. If the hole is slightly larger it may not catch or need more space. Also the closer to the surface you run the risk of cracking/chipping off the surface when you tighten it. I have installed plenty of these.

[Linked Image]