Both sentences have multiple interpretations which largely overlap.
One key issue is that the verb to think can represent multiple degrees of conviction.
X thinks that Y could mean that X is quite convinced that Y is true, or it could mean that X only suspects that Y might be true.
"I don't think you understood me" could be used in a situation where the same speaker could say either "I suspect that you possibly might not have understood me" or "it quite obvious and therefore I strongly believe that you did not understand me".
Like wise, "I think you misunderstood me" supports multiple such interpretations.
Therefore, these sentences are not distinguishable in the speaker's degree of conviction.
However, there are additional factors at play here here. Firstly, not to understand is not the same thing as misunderstand.
Someone who misunderstands something is under the impression that he or she in fact understands: to misunderstand is to "understand wrongly". Just as "misappropriate" does not mean "not to appropriate" but to "appropriate wrongly", and "miscalculate" is not the same thing as "not to calculate" or "not to be able to calculate", but to "calculate wrongly". Other "mis-" words follow similar patterns.
Someone who does not understand could be misunderstanding (not knowing that he or she does not understand) or could be aware that he or she doesn't understand.
Thus, "I don't think you understood me" could mean "I have some degree of conviction in the suspicion that you did not understand me and that you know this is the case", or it could mean "I have some degree of conviction in the suspicion that you understood me incorrectly". The former could occur for example when speaking with a learner of English, and that person makes a puzzled face. The latter would occur when the learner of English appears to understand, but makes an inappropriate reply.
By contract, "I think you misunderstood me" has only the second of these two meanings.
Secondly, to understand has an additional shades of meaning; it can refer to emotional rapport between people. Emotionally, someone either understands you, or does not understand you. The word misunderstand does not usually take on this shade; it refers to some factual blunder.
Thus, "I don't think you understood me" could be a remark uttered by someone who is making a comment about, say, a past relationship. Such a remark would not likely be expressed using "I think you misunderstood me".