The Labour peer, who died in December 2015, is alleged to have abused youngsters over a 30-year period.
His family said that, following Lord Janner's death, the inquiry would be a "proxy trial".
Inquiry chair Prof Alexis Jay ruled a hearing was not necessary as she had already dismissed the family's fears.
Prof Jay, chair of the Independent Inquiry Into Child Sexual Abuse, said the investigation of the evidence would continue but no hearings would take place until it was clear they would not duplicate investigations being carried out by other agencies.
The Independent Police Complaints Commission is investigating the way the Janner case was dealt with by police.
Daniel Janner QC, Lord Janner's son, said the the inquiry has "inexplicably and unfairly" turned down his request to make oral representations.
"I am certain this is because the inquiry is afraid that their decision to hold a separate strand investigating my wholly innocent late father would not stand up to scrutiny forensically," he added.
"This is pure Alice in Wonderland - sentence first, verdict afterwards."
The major inquiry into historical child sex abuse in England and Wales will examine claims made against local authorities, religious organisations, the armed forces, public and private institutions and people in the public eye.
The family of the late Lord Janner have lost their battle to have a hearing to discuss concerns about the inquiry into allegations against him.