Recent reports of abuse within football coaching came as no surprise to social workers, Sharon Shoesmith told PM on BBC Radio 4.
But social workers kept quiet because "you just didn't have the confidence", said Ms Shoesmith.
It was vital to counter vilification of the profession, she said.
Baby Peter Connelly died in 2007 after months of abuse at the hands of his mother, her boyfriend and his brother.
He had more than 50 injuries, despite being on the at-risk register and receiving 60 visits from social workers, police and health professionals over eight months.
His mother, Tracey Connelly, her boyfriend, Steven Barker, and his brother, Jason Owen, were jailed in May 2009 for causing or allowing the child's death.
Ms Shoesmith was sacked as director of children's services for the London borough of Haringey in December 2008 by the then children's secretary Ed Balls.
But in 2011 the Court of Appeal ruled that she had been unfairly dismissed, describing her as "unfairly scapegoated". She was later awarded Â£679,452 in compensation.
She has since written a book, Learning from Baby P, based on a PhD she has completed.
She told PM presenter Eddie Mair that she has been "up and down the country" talking to social workers about her experiences.
"And they are telling me they have taken great strength from the book, from sitting down and understanding this story.
"My main message to the social work profession is there is only one set of people who can change this vilification of social workers, which is very, very embedded in our culture, and it is they themselves who have to take this on.
"And we have talked in some considerable depth and I have run workshops and so on about how the social work profession might begin to counter this because I think it's absolutely vital that they do.
"There is not a social worker in the country who has been surprised that there has been child abuse within the football coaching regimes because we know it - no surprise to them - and yet we don't talk about it - they are quiet about these things.
"I would never have talked in any depth while I was director of children's services about children's social care and what the challenges were in a place like Haringey.
"I would never have done that because you just didn't have the confidence to do it.
"So this book is about in many ways empowering the social work profession."
The full interview is available as a podcast.
The social services boss sacked over the death of Baby P says it is vital that social workers are empowered to speak out about child abuse.