Well, to a point. Oil does pick up contaminants as time and miles roll on, especially with vehicles that only see short trips and often don't reach operating temps for long enough periods of time to burn off the contaminants. If your vehicle runs for 30 minutes after it reaches operating temp, the oil will last a long time. If it only runs 10 minutes at a time and gets shut off for hours afterwards, the oil picks up a lot of junk the can damage your engine, you need to change the oil and the oil filter.
Also, the oil itself does not break down, but the additives in the oil can. As the additives in the oil break down, it can cause engine wear, how much additional wear may be subject to question. Will there be enough wear to cause an engine failure? Don't know. Most engines are designed to run a lot of miles, but some don't last nearly as long as others of the same design, why is that? Gene