When I said its probably not the alternator because this was happening with the previous alternator as well. I had a broken mounting tab on that one so I decided to replace it hoping thats all it was...

Ok, I have the two wire alternator so I found the correct wire ground wire and grounded that terminal. It increased voltage by about .1-.2 volts at both the alternator B+ post and at the battery.

At the voltage regulator the green wire has power and key on and the blue wire has ground. Should that be right? Are those the same wires that came from the alternator?