The Scottish Building Federation's latest quarterly construction monitor saw confidence fall 22 points to -19.
It was the first negative overall confidence rating by employers since the second quarter of 2013.
SBF said employers were unsettled by the economic volatility seen following last week's vote to leave the EU.
There was also concern that investment decisions could be postponed indefinitely.
The federation's survey responses were collected during June, with firms being given the opportunity to update their confidence rating following the EU vote.
SBF managing director Vaughan Hart said: "The results of our latest quarterly survey reflect much of the informal feedback I have received from individual members.
"Construction employers are unsettled by the economic volatility we have witnessed following the vote to leave the European Union last week.
"General uncertainty about the economic outlook has prompted concern that investment decisions could be postponed indefinitely.
"The potential impact on interest rates also risks provoking a sustained slowdown of activity across different sectors of the property market."
He added: "If the current economic volatility is sustained over a longer period of time, the UK Treasury may be forced to take evasive action come the time of the autumn statement with a knock-on impact on the Scottish government's budget and on local government funding.
"There is also a more general concern that the process of negotiating the UK's withdrawal from the European Union could result in paralysis within government that means important priorities such as the delivery of more housing, the development of skills, training and apprenticeships and critical improvements to the country's infrastructure risk being sidelined."
Confidence within Scottish construction has fallen to its lowest level for three years following the outcome of the EU referendum, a survey has found.