Majid Akhtar was on a day trip to Kings Mill Reservoir near Mansfield in September 2012 when he died.
Carer Nikki Deaney spent 19 minutes on the phone in a 34-minute period while supervising him.
He disappeared from her sight while on a walk and was later found dead.
Mrs Deaney, a day centre care assistant at Spring Wood Day Centre in Mapperley, Nottinghamshire, also sent eight text messages in the same time period.
Mr Akhtar, who had a mental age of four, had taken part in water sports with other people at the reservoir earlier in the day.
A jury of eight men and three women at Nottingham Coroner's Court heard Mr Akhtar's care plan said he was "very vulnerable, with no concept of danger, and needed one-to-one care at all times".
He had severe learning difficulties and was epileptic.
Mrs Deaney told police in a statement she had lost sight of Majid for about three seconds.
But the solicitor for the Akhtar family, Georgina Cursham, said it was "likely you were walking for several minutes while on the telephone call before realising Majid wasn't with you.
"I suggest you weren't paying any attention to Majid at all."
His body was found by two people who were walking their dog at the reservoir.
Paramedics were called to the scene and attempts were made to revive him but he was pronounced dead at King's Mill Hospital, Sutton-in-Ashfield.
The inquest continues.
A care worker responsible for a 28-year-old man with learning difficulties was "on the phone" and texting during the half-hour before his death, an inquest has heard.