The risks involved with "the largest UK government-run IT project in history" were "staggering", a report said.
It recommended that the government drastically scale back the programme or abandon it altogether.
Smart Energy GB, the independent body set up to publicise smart meters, said the IoD wanted to take the UK "back to an analogue dark age".
Energy-saving digital smart meters, designed to replace existing analogue gas and electricity meters, should help householders to monitor their energy-use far more accurately, and energy companies to do away with estimated bills.
By some estimates, the new meters could save us £17bn on our energy bills.
But the IoD believes the government's plan to roll out smart meters to all 30 million UK households by 2020 is far too ambitious.
"The pace of technological innovation may well leave the current generation of meters behind and leave consumers in a cycle of installation, de-installation and re-installation," it said.
Under the scheme, energy companies must begin offering free smart meters to their customers from the autumn. Despite the £11bn estimated cost to the industry, it will not be compulsory to have one.
Responding to the IoD report, Sacha Deshmukh, chief executive of Smart Energy GB, said: "The IoD does not understand what's needed to secure Britain's energy infrastructure for the future.
"The smart meter rollout must be for everybody. It will only deliver the national transformation Britain needs if every home is part of this national upgrade."
Nearly 1.4 million households have already had a smart meter installed, he added.
But the IoD isn't the only body concerned about the smart meter programme.
Earlier this month, the Parliament's Energy and Climate Change Committee said the project was in danger of becoming a costly mistake.
The MPs are concerned that allowing each energy company to come up with its own smart meter solution, rather than adopting an industry-wide approach, could lead to inefficiency and further delay.
The government's smart meter scheme could be an "IT disaster", the Institute of Directors (IoD) has said.