Use a reamer. They are not that expensive in the smaller sizes and they do a better job, leaving a smoother surface inside the hole.

I have used this one: https://www.amazon.com/gp/product/B00DMAUAPA/ref=ppx_yo_dt_b_asin_title_o07_s02?ie=UTF8&psc=1
But it is a bit on the short side, and I held it in a drill extension, but worked good for $12

If you can get a longer one it would make the job really easy.