The move was welcomed by Shia Houthi rebels, who brought down the previous government last month.
It is hoped that the former UN envoy's appointment will bring an end to Yemen's political unrest.
Last week the rebels rejected the nomination of Ahmed Awad bin Mubarak for prime minister but suggested Mr Bahah as an alternative.
The new prime minister hails from Hadramawt in the east of the country. He studied business and administration in Yemen and India before being appointed as the country's ambassador to Canada.
He later returned to Yemen as its oil minister before becoming envoy to the UN.
Abdul Maliik al-Ijri, a member of the Houthi political bureau said: "We believe [Khalid Bahah] is the right person. His appointment will help the country overcome the difficulties it is going through."
The Houthis seized key areas of Yemen's capital Sanaa last month.
Under a UN-brokered deal aimed at stopping the deadly clashes, it was agreed that a new government would be formed, while the Houthis and southern separatists would appoint a new prime minister.
The Houthis, also known as Ansar Allah (Partisans of God), have staged periodic uprisings since 2004 to win greater autonomy for their northern heartland of Saada province.
The president of Yemen, Abdrabbuh Mansour Hadi, has appointed Khalid Bahah as his prime minister.