Yup, that's a bit of my worry as well. In fact, for better or for worse - although I never ran into an issue having done so - when I originally cleaned up my current rims I actually popped out each rivet so I could get to clean face of the rim. Ended up polishing each rivet by hand separately and then just popping them back in. None ever came out, not even started to separate out.