Agree on the epoxy or self etch primer first. Some of the epoxies recommend pretty coarse paper for bare metal... down to 80 grit.

I still wonder why your primer peeled. Could there have been some contamination that got on the surface? A good wipe with lacquer thinner before painting might be a good idea.