How is your alternator wired? I bet something is going on there, if it is wired to the battery side of the switch it could have a problem and be the cause of the drain. It is possible for an alternator to be bad and drain the battery when the key is off, but still function properly when the car is running. That would be my guess.

I like to run a continuous duty solenoid up front that breaks alternator power when the disconnect switch is shut off. It does complicate the wiring some but it insures zero hot power when the disconnect is off.

What kind of battery do you have and what kind of charger? When you get a battery drained down bigtime like you mentioned it takes alot of charge time to get them up to 100% properly. Most automatic chargers won't get one there without a few extra steps.