If the OP decides on shims:
Buy a bunch of body shims - basically squares with slots or sliding around a bolt - in varying thicknesses.
Paint jobbers, maybe even auto parts store, should have them.
1/4" thick might be all that's needed.
But my money's on the door frame being bent; oval pin hole would be obvious when replacing the pin.
A photo or two of upper & lower hinges and door face where they attach would likely be helpful.