One downside to smaller filters: Less filter media means a greater pressure differential before and after the media. This will cause the filter’s internal bypass to open sooner when the conditions are right (colder higher viscosity oil, high flow rates due to higher rpm).
I personally don’t like the idea of open bypass, but they are necessary.