The HM Inspectorate of Prisons report suggested boredom and the availability of drink and drugs at HMP Stoke Heath were among reasons for poor behaviour.
The jail, near Market Drayton in Shropshire, had high levels of violence and "some concerning finds of weapons".
Stoke Heath houses 745 prisoners, 40 of them young offenders.
It is a category C "closed" prison - where inmates are not trusted in open prison conditions but are considered unlikely to try to escape.
The report criticised the fact that one prisoner, a foreign national, was being held solely on immigration grounds despite his jail term being completed two years ago.
A high proportion of prisoners had been involved in domestic violence offences, but there was no work done at the jail to address this behaviour.
The inspectorate said: "Many prisoners were under-occupied and boredom contributed to poor behaviour.
"We found 40% of men - unemployed, part-time workers and domestic workers - milling about aimlessly on the wings during the working day."
Chief inspector of prisons Nick Hardwick said HMP Stoke Heath's priorities for the future should include "a focus on tackling violence, keeping men fully occupied and doing more to reduce the risks that they will reoffend after release".
Michael Spurr, chief executive of the national offender management service, said: "The Governor and his staff have worked hard to make real improvements at the same time as managing significant change.
"The prison will now use the findings of this report to develop and improve further, including through an enhanced strategy to tackle and reduce prisoner violence."
A prison should be working harder to reduce violent outbreaks and stop newly-released inmates from reoffending, according to a new report.