Do you have a plat already? I’d show him and see if he will be reasonable. I’ve had it out with neighbors over such things and it can get ugly quick. You need to get a survey done it sounds since he moved the pin. I’d get that booked ASAP will trying to confront him with what you have now. If he’s a jerk, and you feel it’s on your land/survey proves it, you could escalate and take out the driveway gift he gave you just the same as he did to you putting it there. Depends on how you want to play it. I once planted a decent sized tree where a driveway was installed on my property. That got the message across on that one. Others got nasty when I got sick of the bs and put up a fence.