Anything that would affect the field current or the wiring from the alternator, through the ammeter, to the battery will give you those symptoms. I'd bet that it is on the output side of the alternator. If you have a poor connection in that part of the circuit the battery voltage will be unstable. When this happens the regulator starts chasing it and the entire system becomes unstable.

Just a comment, you can't intelligently work on something without know how it works and what it is supposed to do.