We've been developing a design for a deep-space optical transceiver for flight in the second half of the next decade. We're concentrating on an off-axis Gregorian fore-telescope, primarily for reduced optical scattering characteristics. Such a design eliminates the problems associated with the obscuration of the telescope pupil associated with an central obscuring secondary mirror.
For most telescope systems, an on-axis secondary is not much of a problem: it only obscures about 10% or less of the aperture and has a minor effect on the spreading of a diffraction-limited image formed by an incident plane wave. These are usually acceptable trades for implementing a more compact design, in which light beams traverse the telescope tube 3 times on their way to the aft optics. Couple that advantage with the fact that on-axis mirrors use a much larger (and easier to produce) focal ratio (F/#) and have historically been cheaper and easier to fabricate, and you'll understand why the majority of astronomical telescopes follow that general design.
However, an optical communications transceiver introduces some additional concerns. Whereas a 10% (by area) centralized obscuration will only reduce the received power by 10%, it reduces the transmitted power by almost 30% because of the Gaussian intensity distribution of light in the pupil. This is a lot of power to throw away, and has to be effectively dealt with when it is reflected back into the transmitter. The main special concern associated with an optical transciever is that it has to work well both while pointing to deep space, and when pointing very close to the Sun. In the latter case, a secondary mirror, and the window or spider structure required to hold it in place act as strong scattering sources. If the scattered light is too bright for the communications terminal to see the Earth-based beacon, then it can't effectively point, and the signal beam will be lost.
The off-axis system eliminates the scattered sunlight from the secondary mirror edge and its support structure, By baselining a Gregorian system in which two concave paraboloids face each other, a focus is generated right after the primary mirror, where a field stop can be placed to limit stray light entering the rest of the optics train. Finally, modern diamond-turning optical fabrication techniques have substantially reduced the difficulty and cost of generating fast off-axis mirrors, making this type of design much more feasible and cost-effective.
These factors are beginning to tip the balance toward the implementation of an off-axis transceiver for use under the harsh conditions of deep space.