As ChrisF mentions, any mechanical device causes some wear when it operates. With that said, let us start by checking with the National Electrical Code (NEC).
404.11 Circuit Breakers as Switches. A hand-operable circuit breaker equipped with a lever or handle, or a power-operated circuit
breaker capable of being opened by hand in the event of a power
failure, shall be permitted to serve as a switch if it has the
required number of poles.
Informational Note: See the provisions contained in 240.81 and 240.83.
So according to the NEC, circuit breakers can be used as switches with some provisions.
240.81 Indicating. Circuit breakers shall clearly indicate whether they are in the open “off” or closed “on” position. Where circuit
breaker handles are operated vertically rather than rotationally or
horizontally, the “up” position of the handle shall be the “on”
240.83 Marking. (D) Used as Switches. Circuit breakers used as switches in 120-volt and 277-volt fluorescent lighting circuits shall
be listed and shall be marked SWD or HID. Circuit breakers used as
switches in high-intensity discharge lighting circuits shall be listed
and shall be marked as HID.
If the circuit breaker clearly indicates the "on" and "off" position ("on" being up in vertically oriented situations), and the breaker is labeled "SWD" and/or "HID". Then the circuit breaker can be used as a switch, as far as NEC is concerned.
As for whether or not using a circuit breaker in this way causes damage to the circuit breaker, we'll look to the NEC's definition of a circuit breaker.
Circuit Breaker. A device designed to open and close a circuit by nonautomatic means and to open the circuit automatically on a
predetermined overcurrent without damage to itself when properly
applied within its rating.
Notice it specifically mentions "without damage to itself". So in the eyes of the NEC. A device used manually or automatically to open a circuit that causes damage to itself, can not be used as a circuit breaker.
More evidence is provided in this pdf document from Schneider Electric describing the markings on a circuit breaker.
4.) SWD – 15- or 20-A circuit breakers rated 347 V or less may be marked “SWD,” meaning that they are suitable for switching fluorescent
lighting loads on a regular basis (NEC 240.83(D)). These circuit
breakers are evaluated for high endurance use, since they will be used
similar to a light switch.
5.) HID – 50 A or less circuit breakers rated 480 V or less may be marked “HID,” meaning they are suitable for switching high intensity
discharge or fluorescent lighting loads on a regular basis. These
circuit breakers may employ a different construction than a standard
SWD circuit breaker in order to address the high inrush current
resulting from the lower power factor created by the HID lighting (NEC
240.83(D)). These circuit breakers also undergo additional endurance evaluation to demonstrate their ability to perform the switching duty.
While the information presented here, is based on information that may only pertain to the United States. Other countries will have similar rule and markings, to indicate which devices can be used safely in different situations. Circuit breakers rated for this use are tested for durability similarly to switches, and can safely be used to manually open and close a circuit without too much wear and tear.