Never had a problem even with the brake shoes in place. It doesn't take that much of a pry to release the seal. been doing it this way for years now.

Clean it, if it's Dirty. Oil it, if it Squeaks. But: Don't fix it, if it Works!