By hat do you mean the braket that the caliper bolts to?

If so that should not be shimmed and is used to retain the bearing in the housing.

The shims (if needed) would go between that and the caliper itself.