I think you did real well to get most of it off without hurting the paint underneath. If its latex (which I doubt, but who knows) there are removers that will take it off without hurting the enamal. One product is called "goof off" as I remember. Your hardware store can help you.

If its dry oil based primer, I don't see any alternative to sanding or stripping and re-painting. The only upside is finish, gloss ect isn't as critical on the jambs. You might try it at home with an inexpensive sprayer and let the body shop do the outside.