I would check the upper control arm bushings. Might even have to take them out to check.

I would bet that the bushing is allowing the upper A arm to move under had accell, causing the car to move one way, and under decell causes the car to move back the other. I would say only one side is really bad.