Don't think of it as a harsher ride, think of it as more precise handling. After all, the suspension takes all of the load. The ball joint is just going to make the suspension geometry more precise.
When you replace the rubber control arm bushing with an M3 bushing/urethane/polyurethane, then you'll notice a firmer ride. I replaced mine with Bavarian Auto urethane on the front, and with Bilstein Sport struts they were a bit pogo-ee on dips in the road, but the handling and braking was excellent. The installers did complain about how hard it was to install them though... I just regret not replacing the rear suspension bushings with something firmer, as the rear didn't perform as well. But the strut bearings were stock (although I replaced those too because they were worn)--overall the ride was very comfortable but firm; never an issue summer or winter, and with 17" rims I even enjoyed a few autocrosses with the BMW club. I did use the Eibach ProKit (only 1-1.5" drop), so really your choice of springs/struts/bushings are going to determine how harsh the ride is, not the ball joints.