The ground on the relay is not your problem. You wouldn't get a click, plus only original manual trans cars (and not all of them) were grounded through the case of the relay.

It is most likely a low voltage problem (which could be a bad connection reducing the voltage to the starter) or a bad starter or a starter with grounding issues (loose maybe).