It looks like dirt, I bet it's rust imbedded in the rubber. I'd try Evapo-Rust -- it works really well on removing rust from metal and I've tried it on rubber and it works good on that. I wouldn't leave it for hours and hours though. Try it for an hour and see how it does. If it needs more you can just leave it in longer. It's a great rust remover.