The braking with the current setup works decent but not great. I didn't notice much if any improvement when I switched from 2.6 piston front calipers to the 2.75 set. I did use new pads with the new calipers.

Regarding the OEM style brake booster, that linkage piece that is mounted on the passenger side of the firewall: Does anyone have an under-dash picture to show how it connects? I may be interested in running the stock setup if I can get the parts and see how they are supposed to fit.

Still dealing with the substandard brakes.
To recap, the front to rear steel line was replaced with a new line in 2006. The front caliper hoses were replaced in 2012 along with the calipers. The rear disc kit was installed new in 2006. I think the only old parts are the steel line that runs across the firewall and the proportioning valve. I disassembled the valve and cleaned it out with no improvement. I've tried several others as well with little to no improvement.
I'm still leaning toward a conversion to the original style B body booster plumbed as DAYCLONA mentioned. I like the idea of the dual diaphragm if it will provide more boost but if a stock 66-70 single diaphragm unit is rated higher than my current 75 Dart booster, that could work too. I'm still having a hard time envisioning the way the linkage under the dash works.

After MUCH searching, I found THIS picture. It totally cleared up my confusion. The middle section of the crossbar is where the shorter power brake pushrod connects but it was the section below it that confused me. This picture showed that the bottom section connects to the "cage" that the pedal swings from. I looked under my own dash and did see two mounting holes that I never noticed before. Now it is a matter of collecting some parts to do the swap.

I'm somewhat lurking on this thread. My car is 512 six pack Challenger with very similar background. Was factory non power drum, converted to Wilwood Dynalite front disc, still rear drum, no power,,,don't wish for booster because of clutter.

Wilwood pads on front, almost like one of those old coaster rear brake bicycles with no front brake. Brake pressures are excellent tested with gauge.

Installed Hawk high performance so called low dust pads. Have somewhat ok but still not good braking,,,at least some semblance that fronts are contributing to the cause.

However 20 minutes of driving after cleaning rally wheels, they are as dirty as what you normally see on a Mercedes.

Now considering going to a 72 non power brake, disc brakes MC. Been suggested that May work better.

All new steel lines all the way to calipers. Hard pedal that feels good,,,just mediocre brake performance with dusty wheels.

As I said, lurking and watching to see how this plays out.

Last week I replaced the front brake pads with carbon metallic ones. It stops better but still isn't very impressive. I am leaning toward switching to the correct b body Bendix Dual diaphragm booster.
Oddly, through everything, the car stops better in reverse than going forward. I have no idea what that means.

I had the same problem with a '64 sport fury about 8 years ago. Went from power drums to '73 a body manual disc. I weigh 210 and couldn't stop the car with all of the force I could apply. I drilled a new hole in the brake pedal shaft higher up to give it more leverage. It worked, and it would stop again. Easy fix, give it a try.

To do that, I'd think you'd need to weld over the existing hole to keep from weakening the pedal lever to avoid potential failure. I wonder how the new position affects the angle of the brake pushrod.
I do have a spare pedal assembly. Its not a bad idea.

I just measured the center-to-center of the brake pedal mount to pushrod on a 73 Duster and a 70 B body. The Duster was 1 7/16" and the B body setup was 1 7/8". This clearly shows that using the A body booster assembly on my Charger put me at a 7/16" disadvantage in leverage force. I'm sure that the loss of leverage along with the weaker single diaphragm booster explains why I've been unhappy with the braking performance from the power setup. I'm still unsure of why the manual brakes were as bad as they were.
Since the car originally had manual 4 wheel drum brakes with NO proportioning in the system....Why did it perform so poorly with a manual 15/16" MC with 4 wheel discs? Maybe a REpositioned pedal pushrod is a crutch, but if it works, I'm okay with that.

It's all about leverage. Just move the hole higher up. I crawled under my dash, and drilled another hole while it was in place, I think it was an inch, and a half that I moved it up. I guess I got lucky because it worked fine. I would get rid of the booster, just go manual.

Using Mustang rear calipers.

Did you adjust the caliper self adjuster so that there is minimal clearance? If not you get EXACTLY what you are seeing as your symptoms.

I don't know how to adjust the rear calipers! I didn't know that they were adjustable... The Dr Diff stuff didn't come with any instructions. Maybe I can search youTube for videos on this task.

One strange and embarrassing thing I found: I have a disc/drum proportioning valve in the system. Sometime during my attempt to switch to manual brakes 2 years ago, I was trying a bunch of different master cylinders, calipers and ditribution blocks/proprtioning valves and somehow I ended up leaving this prop. valve in.

Pull the caliper outboard until the inner shoe (pad) is firmly seated against the rotor, and measure the clearance between the outer shoe (pad) and the caliper. The clearance must be is 1/32" to 3"32". If it is not, remove the caliper, then readjust the piston the obtain the required gap.
Rotate the shaft counterclockwise to narrow the gap and clockwise to widen gap. (1/4 turn of the piston moves it approximately 1/16")

That's what I found, though when I had my 96 Stang I set them a bit tighter. Just an ever so slight drag, like the fronts. Just so you know, the E brake is what compensates for wear on the rear pads, you have to use it. It also has to be properly adjusted and as the pads wear you may have to readjust the E brake.

Check the rear caliper adjustment, get rid of that disc/drum valve and test.

Thanks SuperCuda. I looked on YouTube but didn't see that point on measuring for clearance. I plan to get out there in a few minutes to work on it.

As a cheat, you could try to partially apply the E brake to take up some slack and test. One thing about the E brake on the Stang calipers is that it only works with a certain amount of clearance, too much and it will not work. Too little clearance will cause the pads to drag and heat up.

I found out about this the hard way, replaced the pads, which requires rotating the piston all the way in, poor brakes and no E brake, till I adjusted them properly.

Good news: Step by step, things are improving.
First up, I swapped IN the drum/drum distribution block. The system was bled. I looked at the rear calipers and it seemed that they have very little slack/clearance. I took the car for a drive and was quite happy to find that while initial free-play was more than I'd like, the brakes do work much better once I take up the slack. I may need to REbleed the system. The pedal is firm once the brakes engage and does not sink.
I removed the MC from the booster and adjusted the length of the pushrod. OEM A body systems have a threaded insert at the end of the pushrod that fits between the MC and the booster diaphragm. On the test drive, the brakes felt better than ever! The pedal free-play was gone and the car stopped great....THEN the brakes started to smoke. I had to get back home QUICK!
I pulled the MC and adjusted the pushrod again. I set it about halfway back to where it was originally. The car was able to roll freely but the pedal free-play was back. I think if I rebleed the system and maybe fine tune the adjustable pushrod, I should be able to call it good. Maybe I'll even try engaging the parking brake slightly to see if the rear calipers do need to be adjusted out a bit.
Thanks guys!