Slick sidewalls are thin. If they dry out enough, they will crack. Not good if it happens while going down the track.

As for visual inspections - I had a brand new 16" trailer tire always stored in the trailer or the garage. It looked "perfect" even after 7 years when I finally installed it to replace a flat. When I got around 1000 miles on it, the tread peeled off like a banana. Luckily it held air despite riding on the steel belts because it took a few miles before I was able to find a spot to pull off.