Depends on usage - if you are at low load most of the time (most street drivers are, even with the odd blast down the quarter) you will probably be OK. If you are at high load frequently (eg towing a trailer) you will probably get valve seat recession. If you do, is it a problem? Probably not, as per peoples experience above. Eventually if left long enough you would have to fit hardened seats, but that could be a very long time.
You do increase the size of the combustion chamber therefore lower CR as the valves recede, again on a street driver this is probably not going to be an issue.

My 2c, if the engine were torn down, it is really not THAT expensive to have the seats done, I'd get it done and never wonder again. If the engine is running fine, carry on for now!