This works for antique furniture with worn out holes for caster wheels too.

Over drill the hole, apply glue to a dowel of the same species, tap in, trim flat.

When the glue is try you can re-drill the correct size hole for the caster wheel.