I forgot to mention that I already loosened the master cylinder a bit so that the push rod wouldn't touch the master cylinder - didn't improve anything.
Maybe I have to open the bleeders and check again.