After welding in a patch panel, at what point am I doing more damage than good when grinding the weld down? Does it have to be completely smooth before applying filler?

Thanks