Generally the guy would have no recourse unless he could prove he suffered a direct loss as a result of your changing your mind.

For example, if he could show a judge he had acted on the agreement to buy by breaking a CD to finance the purchase. Even then, the judge would most likely tell you you had to pay him for his losses but not force you to sell.