In my opinion he's not a pervert...he acts like one because he never experienced a life with girls etc. I would act the same way if it happened with me tbh..but im not a pervert now! :P ive got a gf and im happy about it:)
Actually he acts like a pervert on occasions such as the haiku poems but that was because he does not know how to act around women. Although there is an example of Yoichi just being a pervert; the differentiation of the fake Ayame incident. Other then that Yoichi is almost always just "unfortunate" (in a way he actually is very fortunate in those "unlucky scenes").