Pro-Hezbollah media said the forces were attacking the Jroud Arsal area and Qalamoun mountains from two directions.
The area is home to about 1,000 militants, including Islamic State (IS) group and al-Qaeda's former affiliate.
The operation has raised fears for the safety of thousands of Syrian refugees around nearby Arsal in Lebanon.
Earlier this week, Lebanon's prime minister said the army was preparing its own offensive against militants, who have used Arsal's refugee camps as a safe haven.
Last month, five militants blew themselves up as troops searched camps for suspects and weapons. A young girl was killed and three soldiers wounded by the blasts.
Hezbollah-affiliated media said the operation to flush out "armed terrorists" would carry on indefinitely.
It said Syrian forces and Hezbollah shelled the area, then advanced from Flita on the Syrian side of the border and from the south of Arsal, on the Lebanese side.
Television pictures showed columns of smoke rising as Hezbollah fired artillery towards hills where the militants are based.
Hundreds of militants belonging to Hayat Tahrir al-Sham (HTS), formerly al-Qaeda's official branch in Syria, are believed to be in Jroud Arsal, and a similar number of IS fighters are thought to be in a neighbouring area.
Reuters news agency quoted an unnamed Lebanese security source as saying refugees were fleeing towards Arsal.
The area around Arsal is home to tens of thousands of refugees from the war in Syria. The influx has heightened sectarian tensions inside Lebanon since the conflict began in 2011.
A Sunni enclave surrounded by Shia villages, Arsal was the scene of an attack in 2014, when more than two dozen Lebanese security force members were seized by militants from al-Qaeda and IS who had crossed the border from Syria.
Sixteen have since been released and four killed by their captors.
Syrian troops and their Lebanese Hezbollah allies have launched a long-anticipated offensive against militants in the border area, officials say.