The US central bank was concerned about the impact of the referendum on financial markets, minutes from the meeting show.
Policy makers were also concerned about May's slow job growth.
The central bank decided to keep interest rates between 0.25% and 0.5% when it met last month.
"Members generally agreed that, before assessing whether another step in removing monetary accommodation was warranted, it was prudent to wait for additional data on the consequences of the UK vote," the minutes revealed.
"Most participants noted that the upcoming British referendum on membership in the European Union could generate financial market turbulence that could adversely affect domestic economic performance."
May's surprising dip in the hiring rate also appeared to rattle members of the Federal Open Market Committee (FOMC).
The US labour market added just 38,000 jobs in May, the lowest number since September 2010.
"The minutes show there's more chance of pigs flying than us getting a US rate any time soon," said Luke Bartholomew of Aberdeen Asset Management.
"We can see clearly see that the Fed was worried that a Brexit vote could unsettle markets. It is safe to say the growth and risk sentiment shock from Brexit will keep the Fed on hold."
A few members of the FOMC did raise concerns that keeping interests rates at such a low level for a prolonged period could become problematic.
"Some of them emphasized that, with labour market conditions and inflation at or close to the Committee's objectives, taking another step in removing monetary accommodation should not be delayed too long," the minutes said.
However Federal Reserve members also stressed the need for the US market to be able "withstand a possible downward shock to demand" if international economies wavered.
China's economy also raised some concern, as growth in the world's second largest economy slowed in the second quarter. Policy makers noted the "continued uncertainty" around the country's foreign exchange policy and the mounting levels of Chinese debt.
The next meeting to decide on interest rates is on 26-27 July.
The Federal Reserve believed it was "prudent to wait" for the UK vote on European Union membership before raising interest rates last month.