I've run as close as 1/4" for years with no ill effects. Actually the inner side of the quarter panel is almost always closer than the lip.
Doug