Also consider where you're driving, for me even my 335xi feels kind of large in tight spots downtown. On the open road though, I love the feel of it and the extra space.
I've driven a 5 as well, although it's heavier and larger, it doesn't feel much slower. I love the look of the 5 but not really the size. A bit too big for me for at my current age.
As for the infant, if you have a rear facing baby seat it would be very tight in the 3 depending on how long your legs are. The 5 has much more room, but if the kid is a bit older, the 3 has enough space imo. (Also depends how big your stroller is... so many things to consider!)