I know what 娘惹 means (wikipedia), however, I couldn't find any reference on its origin. Is it a Chinese transliteration of a Malaysian word, a Cantonese/Hokkien word or a Mandarin word? What's the literal meaning?
It is definitely transliterated from Malay (it's noted on the zh Wikipedia as well).
Baba and Nyonya (pronounced /ɲoɲə/) are descendants of Hokkien Chinese who migrated to Malaysia centuries ago. They have come to refuse to be recognized as descendants of the Chinese and instead, identified themselves as British subjects (as per Bahasa Malay WP entry).
According to Wikipedia, the term baba is a religious honorific term that originated from Persian and entered Southeast Asian languages possibly through traders. It originally was used as an honorific for one's grandparents in Malay. Its meaning changed over time among the Baba-Nyonya to refer to all males of this particular ethnic group.
Nyonya is a loanword from Javanese that was likely borrowed from Portuguese don(h)a (compare Italian: donna; Spanish: doña) meaning lady (夫人). The term was used by the Malay people to refer to all "foreign" women including female peranakan. With the departure of European colonists, the meaning of the term slowly narrowed down to only peranakan women.
For further reading on the cultural aspect of this term: Southeast Asia: A Historical Encyclopedia, From Angkor Wat to East Timor, Volume 1.