Generally, old in the sense of former or previous is not used in such constructions at all. In the given context, the word old can only mean aged; My old teacher cannot be the opposite of My new teacher.
However, old is sometimes used in such expressions as my old house (to mean the previous), and an old friend (long-time friend).
On a different note, the use of old for former or previous is standard in some Oriental languages, where old is the antonym of new in every context.
"But should it be avoided in this context anyway?" Yes.
The adjective applies to time (indirect, elided reference) rather than the noun mentioned (person). As such, it can be used only where its reference is unambiguously to time. "My old friend" has the elision: "My old (-time) friend". Not all contexts are self-explanatory and can be ambiguous, as in "My old teacher" occurring without context.