Edited Answer, see below.
Every time I've studied conversations patterns and rules of speaking, the other person has always been called either hearer or addressee. Here I'll quote an example from my notes:
"Speakers and hearers constantly adjust their internal registry of deictics to keep up with the conversation."
"Directives = Speech acts in which the words are aimed at making the hearer do something;"
I'm not really sure about listener, but honestly I can't recall using it.
Since I've been downvoted, I'll link a further reference which proves me right. The book, Speech acts: an essay in the philosophy of language, is written by John R. Searle, a rather known name in the Linguistics field and to whoever studied Linguistics in an academic setting.
Speech Acts are utterances that don't simply convey information, but can also "perform actions" such as "I hereby declare you company and wife." (This one, for example, is a Declaration according to Searle's classification of Speech Acts.
If you search words in that book, there are 34 entries for "hearer" and 0 entries for "listener".