Because yes, many SEOs are really afraid of that, they are afraid that Google can see that data and use it against the website.
The point is: this makes no sense. Users search for an information on search engines, they click on a result, they find the answer… and they leave. Happy.
What’s wrong with this behavior? Nothing! Why should a SEO be worried about the bounce rate then? No reason.
Rand correctly uses the example of a Q&A website. I go further: if you go on Google, search for something, and leave Google without coming back for more results, doesn’t it mean Google actually provided you with a good result?
Ok, technically speaking, searching for a key-phrase and clicking on a result doesn’t increase a bounce rate, since you are visiting two pages, not just one. But still, the time spent on Google is short. The shorter, the better.
You should be worry about the short time spent on your website for other reasons, maybe. But that data, out of context, means nothing. So, don’t panic.