Weeman - you're only partially correct. You can opt to not stop at the US border and properly "export" your car with the required paperwork. However, if you go back into the US and are pulled over by the police, you face a $5000 fine for possession of an improperly exported item...or some nonsense like that.
I recently helped a friend move from the US to Canada and I explicitly asked the Canadian Border Services Agency about this so I have first hand info.
That being said, I don't know why you wouldn't stop at the US border to get your "stamp." I've imported 2 cars from the US and each time it took me less than 5 mins to get my stamp. Mind you, I was the first person in line (e.g. Got there as soon as it opened) and had done all of my homework and was well prepared. If you get there at 4:59pm on a Friday, you may have a different experience.