After the 4,000 years or so of peace the world fell into a long period of darkness and winter known as the Long Night. During this time evil creatures known as the Others, or White Walkers, came down from the Land of Always Winter in the extreme north of the continent of Westeros.
The Children of the Forest and the First Men teamed up to fight the Others and eventually pushed them all the way back to the north. This is when the First Men and Children of the Forest decided to construct The Wall and form the Night's Watch.
The Pact began 4,000 years of friendship and peace between the two peoples. This is known as the Age of Heroes. During this time the First Men took took up the same gods as the Children of the Forest.