I noticed that if you try multi-tasking while watching a video on quick time, it becomes like what you said or the video itself becomes choppy.
But if you watch the video and you're not multi-tasking at the same time while the sound becomes choppy, then it could be the the user who encoded the video. (i.e. Problems with encoding may make the video choppy)
These are just some symptoms.
That's why I hate it when people encode their videos using Real Media or Quicktime and sometimes the *.wmv format.
-=Slow In Fast Out=-