I think the first question you have to ask yourself is why you want to do it. If you have aspirations of becoming some kind of champion, or even some day making a little money at it, then you have to start in Karts. The thing is that this is not for everyone... If you're not winning at Karts after your first or second season (and doing so at each level when you're moving up), then it's time to pass on the dream unless you're loaded.
If you just want to compete for the sake of competing, then I'd say start with some good HPDE schools, like the ones put on by BMW Club at the bigger tracks like MoSport and Tremblant. It sounds like you've never been on the track, so you will really want to try these first because you will find that it's an entirely different ball game from being a badass on the street. You may well find yourself progressing quickly and being just as comfortable being on the edge at the track as you are blasting through twisty public roads but in many cases that is far from reality and you may just get pissed off and frustrated - so it will just be a cheap way to find out if it is or is not what you really want, before you start dumping $ into a track car, which, like Bruno said, is endless.