|Publication number||US6980485 B2|
|Application number||US 10/004,070|
|Publication date||Dec 27, 2005|
|Filing date||Oct 25, 2001|
|Priority date||Oct 25, 2001|
|Also published as||US20030081504|
|Publication number||004070, 10004070, US 6980485 B2, US 6980485B2, US-B2-6980485, US6980485 B2, US6980485B2|
|Original Assignee||Polycom, Inc.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (21), Referenced by (64), Classifications (8), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
1. Field of the Invention
The invention relates generally to videoconferencing systems and more specifically to camera tracking of videoconference participants.
2. Description of Related Art
When people wish to communicate from remote locations, a videoconferencing system is a convenient way to share ideas. Typically, there is more than one person participating in a videoconference. A camera must, therefore, be positioned to frame everyone in the room. However, such camera shots are impersonal and do not allow the recipient of the image to pick up on nuances of the speaker's facial expressions because they lack sufficient detail. Manually tracking the camera from one speaker to the next is inconvenient and distracting. There are a number of prior art patents in this field directed to solving this problem.
For example, U.S. Pat. No. 5,686,957 to Baker teaches automatic tracking by using a plurality of microphones and a crude audio detection circuit. Because the audio detection mechanism is limited in its ability to locate sound, a special camera is used to enhance the peripheral portion of the field of view. The camera rotates in approximately 30 degree increments in order to frame the general location of the speaker.
U.S. Pat. No. 5,778,082 to Chu teaches automatic tracking by using a plurality of microphones and a processor. The processor determines when an acoustic signal begins at each microphone, and then locates the direction of the source based on a comparison of data from each microphone. The system, however, requires processor intensive Fast Fourier Transform calculations to be made continuously.
U.S. Pat. No. 5,940,118 to Van Schyndel teaches automatic tracking using an optical transducer that takes visual cues (e.g., a moving mouth) to point the camera toward the location of the speaker. The method requires a very advanced processor, an optical transducer, and is subject to many false signals (e.g., one participant whispering to a neighbor).
U.S. Pat. No. 5,959,667 to Maeng teaches automatic tracking using a microphone array and a set of preset values. The preset values include a set of camera parameters that define a particular camera shot. The microphone array locates the position of the speaker, and the position is compared to the preset values. The camera then tracks to the closest preset position. The method also requires a very powerful processor because the speaker location has to be continuously calculated.
U.S. Pat. No. 6,005,610 to Pingali teaches a method for localizing a speaker based on concurrent analysis of a video image and plural microphone signals. The position of the speaker is tracked as the speaker changes position.
U.S. Pat. No. 5,438,357 to McNelley teaches a method for image manipulation that creates the impression of eye contact between video conferencing parties. The camera is manipulated to track a speaker such that their image is centered on a display.
U.S. Pat. No. 5,528,289 to Cortjens et al. teaches a system for controlling devices on a computer network. The pan, tilt, and zoom of each camera is controlled via a pointing device such as joystick or mouse.
U.S. Pat. No. 5,581,620 to Brandstein et al. teaches a method for enhancing the reception of signals received at several locations using beamforming techniques. The location of a speaker is calculated and the information is used to align the phase of signals from multiple audio detectors. The signals originating from the speaker's location are added while signals from other locations are attenuated.
U.S. Pat. No. 5,583,565 to Cortjens et al. teaches a camera system that can be remotely controlled over a computer network for the purpose of video conferencing. Preset operational parameters are stored within the camera to increase the ease of operation. A user input device is used to control the camera.
U.S. Pat. No. 5,844,599 to Hildin teaches a video system in which a camera is directed at a voice activated emitter associated with a speaker. The emitters are detected using an infrared position signal and a video camera is directed at the emitter's location.
In one embodiment, the invention provides an acoustic tracking system for determining the location of a sound producing acoustical source from a set of at least two user-defined locations. The system includes an array of microphones and a processor for performing various calculations. The processor can be any device that can make the necessary calculations such as a CPU, FPGA, and the like.
The calculations result in a set of beamforming parameters, a set of modified signals, and a result location. Each beamforming parameter within the set of beamforming parameters is associated with each user-defined location within the set of at least two user-defined locations. The set of modified signals is determined by applying each beamforming parameter within the set of beamforming parameters to the sound received by the array of microphones. The result location is then associated with the best modified signal within the set of modified signals. The best modified signal can be determined as a function of signal strength and duration.
In another embodiment, the invention provides a method for identifying the location of an audio source. The method includes the steps of defining at least two points where the audio source may be located, calculating beamforming parameters for each of the at least two points in space, receiving sound by a microphone array, applying each set of the beamforming parameters to the sound received by the microphones in the array, determining which set of beamforming parameters maximize the sum amplitude of the signals received by the microphone array; and identifying one of the at least two points in space associated with the set of beamforming parameters that maximize the sum amplitude of the microphone signals.
Movable microphone 170 receives the sound that is transmitted as the audio component of the teleconference to a remote location. Movable microphone 170 is typically kept away from video camera 160 so it does not pick up extraneous noise from the normal operation of video camera 160.
In one embodiment of the invention, microphone array 150 is positioned proximate to video camera 160. As will be described, microphone array 150 consists of at least two microphones, although a typical system would includes five or six microphones. Additionally, if the location of movable microphone 170 relative to videoconferencing system 140 was known, microphone 170 could also be configured to act as part of microphone array 150. In order to simplify the description of the invention all references to a microphone or microphones in this specification will refer to the microphones within microphone array 150.
Three potential audio sources 180, 184, and 188 are shown in room 100. Sound waves 190 are formed when one of potential audio sources 180 generates sound (i.e., a person who is sitting in chair 120 speaks). As will be described in connection with
A first microphone 220, located an r, distance 230 from audio source 210, receives the signal at a time t1, and a second microphone 240, located an r2 distance 250 from audio source 210, receives the signal at a time t2. As will be seen, it is advantageous to describe t2 in terms of Δt2, the difference in time between t1 and t2. Although Δt2 is a positive number in
By arbitrarily assigning the origin (0,0) of a Cartesian coordinate system to second microphone 240 and orienting the coordinates so first microphone 220 is on the X-axis, beam-line 270 can be described as a function of Δt2 using the following equations:
x(t 1)=(D 1 2+2S 2 t 1 Δt 2 +S 2 Δt 2 2)/2D 1 (1) and
y(t 1 , x)=±[S 2 t 1 2−(x−D 1)2]1/2 (2).
Equations (1) and (2) are derived by applying Pythagoras' Theorem to the two right triangles with the hypotenuses r1 distance 230 and r2 distance 250, noting that r1=S t1 and r2=S t2, and substituting (t1+Δt2) for t2. Once Δt2 is known, the x position and y position will change as a function of t1. While
If the time t1 that the signal takes to reach first microphone 220 is known, then equations (1) and (2) can be used to reduce the possible positions of audio source to one of two points ((x, y) and (x, −y)). However, since the t1 time is usually not known, the most information that can be derived from microphone array 150 with only two microphones is that audio source 210 originated along beam line 270. A third microphone is required to determine the location of audio source 210 along beam-line 270.
t 1=(D 1 D 2 2 +D 1 S 2 Δt 3 2 −D 2 S 2 Δt 2 2 −D 1 2 D 2)/(2D 2 S 2 Δt 2−2D 1 S 2 Δt 3) (3)
can be derived.
By combining the three equations, audio source 210 can be located by knowing D1 distance 260 (the spacing between first microphone 240 and second microphone 220), D2 distance 330 (the spacing between second microphone 220 and third microphone 310), and the Δt2 and Δt3 time intervals (the length of time between when second microphone 240 and third microphones 310 receive the signal relative to the time first microphone 220 receives the signal).
The three microphones 220, 240, and 310 are shown to be co-linear for simplicity. However, it should be appreciated that a non-co-linear arrangement of the three microphones enables identification of a pair of source positions, (x, y, z) and (x, y, −z), in three-dimensional space. The three non-co-linear microphones form a plane, and all points on one side of the plane are represented as positive z and points on the other side are represented as negative z. Although a fourth non-planar microphone can then be used to determine which side of the plane the source is located, all the potential audio sources are normally disposed on only one side of microphone array 150, so a fourth microphone is not necessary to locate a single point in space. However, additional microphones are desirable to provide error checking and accuracy.
Using these principals, the flowchart in
After the potential sources are identified, next step 420 in
As an example of steps 410 and 420,
If microphone 520 was at location (X, Y, Z) in Cartesian coordinates, one of potential sources 180 was located at (ρ, φ, θ) in spherical coordinates, and the origin (0,0) of both the Cartesian coordinates and the spherical coordinates was the camera, the distance could be found with the following equation:
Distance=[(X−ρ cos θ sin φ)2+(Y−ρ sin θ sin φ)2+(Z−ρ cos φ) 2]1/2 (4).
The time that it takes for a signal to reach any given Distance can be found by dividing the Distance by the speed of sound (about 1129 feet/second at 68 degrees Fahrenheit). Since knowing the absolute time required for a signal to travel to each microphone is not necessary only the change in time from one microphone to the next is stored in step 420.
In another aspect of the invention, the beamforming parameters are calculated in the frequency domain instead of the time domain. As is well known in the art, Fourier theory (including both the Fourier Series and the Fourier Transform) relates the time domain and the frequency domain. Furthermore, it is often useful to use a Fast Fourier Transform (FFT) as an approximation to the Fourier Transformation. The effects of background noise can be easily reduced in the frequency domain by, for example, applying greater weights to some frequencies or implementing a bandwidth filter. In the frequency domain, the beamforming parameters represent the changes in phase from one microphone to the next, and not the changes in time.
Referring back to
As an example of steps 430 through 460,
In order to apply the beamforming parameters to the signal at microphone B, the entire signal is shifted by an amount equal to Bb1. Therefore, the arbitrary component 620 within signal 610 peaks at a time (tb−Bb1) 640. Component 620 peaks at microphone C at a time (tc−Bc1) 645. If the source of signal 610 were located at the first potential source, then both (tb−Bb1) 640 and (tc−Bc1) 650 would be equal to ta 630.
There are many methods that may be used to determine whether the modified signals for each microphone match. For example, adding the modified signals from each microphone generates an aggregate signal for a potential source. The potential source that generates the loudest aggregate signal would be where the signal originated. When aggregate signal 660 in
In another aspect of the invention, calculations are performed in the frequency domain. As previously mentioned, a phase shift in the frequency domain is equivalent to a time shift in the time domain. Therefore, a method similar to that already described can be applied in the frequency domain.
In the last step 470 a decision is made whether to continue camera tracking. Camera tracking can be performed only once, be performed only at certain intervals, be continuous, or only be performed only if certain conditions are met (e.g., when sound is received). Unnecessary camera movement may be avoided by waiting until a certain amount of time has elapsed before moving the camera to ensure a new person is speaking, thereby limiting erroneous tracking due to coughing or errant comments. Camera tracking can also be performed by using multiple cameras and only activating the camera that frames the speaker or by using a PTZ (pan/tilt/zoom) camera (mechanical or electronic).
Although the invention has been described in its presently contemplated best mode, it is clear that it is susceptible to numerous modifications, modes of operation and embodiments, all within the ability and skill of those familiar with the art and without the exercise of further inventive activity. Accordingly, that which is intended to be protected by Letters Patents is set forth in the claims and includes all variations and modifications that fall within the spirit and scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4858012||Mar 18, 1988||Aug 15, 1989||Hiromasa Hino||Camera having automatically controllable shooting features|
|US5068735||Aug 13, 1990||Nov 26, 1991||Fuji Photo Optical Co., Ltd.||System for controlling the aiming direction, focus, zooming, and/or position of a television camera|
|US5396287||Feb 18, 1993||Mar 7, 1995||Fuji Photo Optical Co., Ltd.||TV camera work control apparatus using tripod head|
|US5438357 *||Nov 23, 1993||Aug 1, 1995||Mcnelley; Steve H.||Image manipulating teleconferencing system|
|US5528289 *||Jan 17, 1995||Jun 18, 1996||Videoconferencing Systems, Inc.||Method for automatically adjusting a videoconferencing system camera to center an object|
|US5561519||Jun 6, 1995||Oct 1, 1996||Parkervision, Inc.||Remote-controlled tracking system for tracking a remote control unit and positioning and operating a camera and method|
|US5581620 *||Apr 21, 1994||Dec 3, 1996||Brown University Research Foundation||Methods and apparatus for adaptive beamforming|
|US5583565 *||Jan 17, 1995||Dec 10, 1996||Videoconferencing Systems, Inc.||Method for automatically adjusting the pan and tilt of a video conferencing system camera|
|US5631699||Feb 6, 1995||May 20, 1997||Konica Corporation||Video camera system for use in fixed and free modes in which, when coupled to a base in the fixed mode, video functions are automatically set by a control|
|US5684528||Dec 4, 1995||Nov 4, 1997||Fujitsu Limited||Camera controlling apparatus for television conference system|
|US5686957 *||Jun 30, 1995||Nov 11, 1997||International Business Machines Corporation||Teleconferencing imaging system with automatic camera steering|
|US5719622||Feb 23, 1996||Feb 17, 1998||The Regents Of The University Of Michigan||Visual control selection of remote mechanisms|
|US5778082 *||Jun 14, 1996||Jul 7, 1998||Picturetel Corporation||Method and apparatus for localization of an acoustic source|
|US5844599 *||Aug 6, 1997||Dec 1, 1998||Lucent Technologies Inc.||Voice-following video system|
|US5859663||Jul 29, 1997||Jan 12, 1999||Intel Corporation||Audio control system for video teleconferencing|
|US5940118||Dec 22, 1997||Aug 17, 1999||Nortel Networks Corporation||System and method for steering directional microphones|
|US5940188 *||Dec 18, 1996||Aug 17, 1999||Minolta Co., Ltd.||Image processing and communicating apparatus that allows processing according to function of external apparatus and image communication method|
|US5959667 *||May 9, 1996||Sep 28, 1999||Vtel Corporation||Voice activated camera preset selection system and method of operation|
|US6005610 *||Jan 23, 1998||Dec 21, 1999||Lucent Technologies Inc.||Audio-visual object localization and tracking system and method therefor|
|US6008837||Sep 30, 1996||Dec 28, 1999||Canon Kabushiki Kaisha||Camera control apparatus and method|
|US6014167||Jan 21, 1997||Jan 11, 2000||Sony Corporation||Tracking apparatus and tracking method|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7473040||Oct 14, 2005||Jan 6, 2009||Lifesize Communications, Inc.||High definition camera pan tilt mechanism|
|US7572073||Oct 17, 2005||Aug 11, 2009||Lifesize Communications, Inc.||Camera support mechanism|
|US7593539||Apr 17, 2006||Sep 22, 2009||Lifesize Communications, Inc.||Microphone and speaker arrangement in speakerphone|
|US7623156 *||Nov 24, 2009||Polycom, Inc.||Natural pan tilt zoom camera motion to preset camera positions|
|US7667728||Feb 23, 2010||Lifesize Communications, Inc.||Video and audio conferencing system with spatial audio|
|US7667762||Aug 1, 2006||Feb 23, 2010||Lifesize Communications, Inc.||Dual sensor video camera|
|US7688345||Mar 30, 2010||Lifesize Communications, Inc.||Audio output in video conferencing and speakerphone based on call type|
|US7717629||Apr 14, 2006||May 18, 2010||Lifesize Communications, Inc.||Coordinated camera pan tilt mechanism|
|US7720232||Oct 14, 2005||May 18, 2010||Lifesize Communications, Inc.||Speakerphone|
|US7720236||Apr 14, 2006||May 18, 2010||Lifesize Communications, Inc.||Updating modeling information based on offline calibration experiments|
|US7760887||Jul 20, 2010||Lifesize Communications, Inc.||Updating modeling information based on online data gathering|
|US7826624||Nov 2, 2010||Lifesize Communications, Inc.||Speakerphone self calibration and beam forming|
|US7903137||Apr 17, 2006||Mar 8, 2011||Lifesize Communications, Inc.||Videoconferencing echo cancellers|
|US7907745||Mar 15, 2011||Lifesize Communications, Inc.||Speakerphone including a plurality of microphones mounted by microphone supports|
|US7970150||Jun 28, 2011||Lifesize Communications, Inc.||Tracking talkers using virtual broadside scan and directed beams|
|US7970151||Jun 28, 2011||Lifesize Communications, Inc.||Hybrid beamforming|
|US7991167||Aug 2, 2011||Lifesize Communications, Inc.||Forming beams with nulls directed at noise sources|
|US8054336||Nov 8, 2011||Lifesize Communications, Inc.||High definition pan tilt zoom camera with embedded microphones and thin cable for data and power|
|US8116500||Apr 17, 2006||Feb 14, 2012||Lifesize Communications, Inc.||Microphone orientation and size in a speakerphone|
|US8237770||Apr 18, 2006||Aug 7, 2012||Lifesize Communications, Inc.||Audio based on speaker position and/or conference location|
|US8248448||Aug 21, 2012||Polycom, Inc.||Automatic camera framing for videoconferencing|
|US8249298||Aug 21, 2012||Polycom, Inc.||Ultrasonic camera tracking system and associated methods|
|US8395653||Mar 12, 2013||Polycom, Inc.||Videoconferencing endpoint having multiple voice-tracking cameras|
|US8457614||Jun 4, 2013||Clearone Communications, Inc.||Wireless multi-unit conference phone|
|US8743204||Jan 7, 2011||Jun 3, 2014||International Business Machines Corporation||Detecting and monitoring event occurrences using fiber optic sensors|
|US8842161||Aug 20, 2012||Sep 23, 2014||Polycom, Inc.||Videoconferencing system having adjunct camera for auto-framing and tracking|
|US8878891||May 10, 2012||Nov 4, 2014||Lifesize Communications, Inc.||Providing audio playback during a conference based on conference system source|
|US8879761||Nov 22, 2011||Nov 4, 2014||Apple Inc.||Orientation-based audio|
|US8903108||Jan 4, 2012||Dec 2, 2014||Apple Inc.||Near-field null and beamforming|
|US8923529||Aug 26, 2009||Dec 30, 2014||Biamp Systems Corporation||Microphone array system and method for sound acquisition|
|US9020163||Dec 6, 2011||Apr 28, 2015||Apple Inc.||Near-field null and beamforming|
|US9030520||Jun 20, 2011||May 12, 2015||Polycom, Inc.||Automatic camera selection for videoconferencing|
|US9084057 *||Oct 19, 2012||Jul 14, 2015||Marcos de Azambuja Turqueti||Compact acoustic mirror array system and method|
|US20060012671 *||Jul 16, 2004||Jan 19, 2006||Alain Nimri||Natural pan tilt zoom camera motion to preset camera positions|
|US20060082655 *||May 2, 2005||Apr 20, 2006||Vanderwilt Patrick D||High definition pan tilt zoom camera with embedded microphones and thin cable for data and power|
|US20060083389 *||Apr 18, 2005||Apr 20, 2006||Oxford William V||Speakerphone self calibration and beam forming|
|US20060088308 *||Oct 17, 2005||Apr 27, 2006||Kenoyer Michael L||Camera support mechanism|
|US20060093128 *||Oct 14, 2005||May 4, 2006||Oxford William V||Speakerphone|
|US20060104458 *||Oct 17, 2005||May 18, 2006||Kenoyer Michael L||Video and audio conferencing system with spatial audio|
|US20060104633 *||Oct 14, 2005||May 18, 2006||Kenoyer Michael L||High definition camera pan tilt mechanism|
|US20060132595 *||Oct 14, 2005||Jun 22, 2006||Kenoyer Michael L||Speakerphone supporting video and audio features|
|US20060238611 *||Oct 17, 2005||Oct 26, 2006||Kenoyer Michael L||Audio output in video conferencing and speakerphone based on call type|
|US20060239443 *||Apr 17, 2006||Oct 26, 2006||Oxford William V||Videoconferencing echo cancellers|
|US20060239477 *||Apr 17, 2006||Oct 26, 2006||Oxford William V||Microphone orientation and size in a speakerphone|
|US20060245601 *||Apr 27, 2005||Nov 2, 2006||Francois Michaud||Robust localization and tracking of simultaneously moving sound sources using beamforming and particle filtering|
|US20060256974 *||Apr 11, 2006||Nov 16, 2006||Oxford William V||Tracking talkers using virtual broadside scan and directed beams|
|US20060256991 *||Apr 17, 2006||Nov 16, 2006||Oxford William V||Microphone and speaker arrangement in speakerphone|
|US20060262942 *||Apr 17, 2006||Nov 23, 2006||Oxford William V||Updating modeling information based on online data gathering|
|US20060262943 *||Apr 13, 2006||Nov 23, 2006||Oxford William V||Forming beams with nulls directed at noise sources|
|US20060269074 *||Apr 14, 2006||Nov 30, 2006||Oxford William V||Updating modeling information based on offline calibration experiments|
|US20060269080 *||Apr 11, 2006||Nov 30, 2006||Lifesize Communications, Inc.||Hybrid beamforming|
|US20060269278 *||Apr 14, 2006||Nov 30, 2006||Kenoyer Michael L||Coordinated camera pan tilt mechanism|
|US20070279483 *||Nov 13, 2006||Dec 6, 2007||Beers Ted W||Blended Space For Aligning Video Streams|
|US20080030611 *||Aug 1, 2006||Feb 7, 2008||Jenkins Michael V||Dual Sensor Video Camera|
|US20080095401 *||Oct 15, 2007||Apr 24, 2008||Polycom, Inc.||Ultrasonic camera tracking system and associated methods|
|US20100123785 *||Nov 17, 2008||May 20, 2010||Apple Inc.||Graphic Control for Directional Audio Input|
|US20110103612 *||Nov 2, 2010||May 5, 2011||Industrial Technology Research Institute||Indoor Sound Receiving System and Indoor Sound Receiving Method|
|US20110164761 *||Aug 26, 2009||Jul 7, 2011||Mccowan Iain Alexander||Microphone array system and method for sound acquisition|
|US20130100233 *||Oct 19, 2012||Apr 25, 2013||Creative Electron, Inc.||Compact Acoustic Mirror Array System and Method|
|US20150341545 *||Jan 9, 2013||Nov 26, 2015||Lg Electronics Inc.||Voice tracking apparatus and control method therefor|
|EP1914563A1||Oct 19, 2007||Apr 23, 2008||Polycom, Inc.||Ultrasonic camera tracking system and associated methods|
|EP2388996A2||May 11, 2011||Nov 23, 2011||Polycom, Inc.||Videoconferencing endpoint having multiple voice-tracking cameras|
|EP2538236A2||May 11, 2012||Dec 26, 2012||Polycom, Inc.||Automatic camera selection for videoconferencing|
|WO2014109422A1 *||Jan 9, 2013||Jul 17, 2014||Lg Electronics Inc.||Voice tracking apparatus and control method therefor|
|U.S. Classification||367/119, 348/E07.079|
|International Classification||G01S3/808, H04N7/14|
|Cooperative Classification||G01S3/8083, H04N7/142|
|European Classification||G01S3/808B, H04N7/14A2|
|Oct 25, 2001||AS||Assignment|
Owner name: POLYCOM, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MCCASKILL, JOHN;REEL/FRAME:012358/0436
Effective date: 20011019
|May 21, 2009||FPAY||Fee payment|
Year of fee payment: 4
|Mar 18, 2013||FPAY||Fee payment|
Year of fee payment: 8
|Dec 9, 2013||AS||Assignment|
Owner name: MORGAN STANLEY SENIOR FUNDING, INC., NEW YORK
Effective date: 20130913
Free format text: SECURITY AGREEMENT;ASSIGNORS:POLYCOM, INC.;VIVU, INC.;REEL/FRAME:031785/0592