Thanks. I figured it out: The speaker singals from the headunit was going to the stock amp, where it was getting amped again. So that was working fine all along, you just couldn't turn the volume up much.
Then when I added the sub, I bypassed the stock amp, so when I had the volume low on the headunit the signal was really weak going to the sub amp.
So what I ended up doing was just using the fader to fade the sound to the rear speakers, which aren't installed anymore, so that the fronts were only getting a light signal. Then you can turn up the volume and it matches with the sub.
Sounds odd, but it will work fine for now until I replace the stock speakers and can get rid of the stock amp.