Research by the Federation of Small Businesses (FSB) found two-thirds of small Scottish firms with EU staff were worried about future skills shortages.
A total of 37% said they would consider reducing operations if Brexit led to extra barriers on hiring EU citizens.
Almost a fifth said they might close their business, while 12% would consider moving their business abroad.
FSB said EU workers helped to plug chronic skills gaps across a range of sectors.
The UK-wide research found 26% of Scottish small employers currently had a member of staff from elsewhere in the EU, with this figure rising to 41% in the Highlands.
By comparison, about a fifth of UK firms employed an EU worker.
Almost 90% of those surveyed said they had recruited their EU staff when they were already living in the UK.
The vast majority of UK respondents (95%) said they had no experience of using the UK's points-based immigration system to recruit non-EU workers.
FSB national chairman Mike Cherry said: "There is real concern among small firms with EU staff that they will lose access to the skills and labour their business needs to survive and grow.
"EU workers are a vital part of our economy, helping to plug chronic skills gaps across a wide range of sectors, and filling jobs in an already tight labour market.
"From packers, to mechanics, to graphic designers, small employers need to be able to hire the right person, for the right job at the right time."
Scottish policy convener Andy Willox said: "Our data shows that our members predominantly recruit non-UK EU citizens because they're the best candidates.
"If our immigration system is set to change, then our skills system needs to do the same."
The survey included responses from 577 Scottish respondents between 31 March and 12 April.
Small firms have raised fresh concerns about the impact of Brexit on employing EU workers.