The easiest authors tend to be the ones you've spent the most time with. I know that isn't helpful, but I've found that a new author is always difficult when first starting. After you read for a while, you get a hang of the vocabulary, syntax, grammar, expressions, etc. that they like to use. Cicero is considered a "difficult" author because of long sentences, but I have a much easier time reading him than, say, Tacitus, because I've spent a lot of time with Cicero. I might feel like I know all the vocab there is, but when I open a new author, especially poetry, I'm confronted by a bunch of unknown words and difficult constructions.
With that said, I've found Boethius, Apulleius, and Tacitus to be very difficult. They are all idiosyncratic. Tertullian is also said to be very tough. Note that all of these authors wrote after the so called "golden age." Seneca is an author that I found confusing at first, due to strange vocabulary and syntax, but got easier quickly.
If you stick with one time period or genre, you'll also have an easier time. As noted, poetry is more difficult than prose in almost all cases.
Horae quidem cedunt et dies et menses et anni, nec praeteritum tempus umquam revertitur nec quid sequatur sciri potest. Quod cuique temporis ad vivendum datur, eo debet esse contentus. --Cicero, De Senectute