I have zero knowledge about e46 audio systems, however.
In my e36, adding an aftermarket headunit made an enormous difference. I'm not an audiophile, but i do enjoy clean sounding music, and even just having upgraded my HU, the sound became clear and crisp, perhaps only really lacking in bass. Once i added a subwoofer/amp to the system, it started really sounding great.
I'd just do HU first, although for e46 i'm not sure what options you have. I'm sure you could get a single din deck, but it may look strange.
Hot: 2001 Estoril M Roadster
ZCP 19's, ZHP knob, JL 8W3
Cold: 2002 TiAg M3 6mt
ZCP 19's supersports, ZHP knob, UUC short shift GROM, OEM 18's w310's,
http://absolutehid.ca hid kit