US 20140125810 A1
An imaging device includes an image sensor and an array of wafer lenses. The image sensor has rows and columns of pixels partitioned into an array of sensor subsections. The array of wafer lenses is disposed over the image sensor. Each of the wafer lenses in the array of wafer lenses is optically positioned to focus image light onto a corresponding sensor subsection in the array of sensor subsections. Each sensor subsection includes unlit pixels that do not receive the image light focused from the wafer lenses and each sensor subsection also includes lit pixels that receive image the image light focused by the wafer lenses. A rectangular subset of the lit pixels from each sensor subsection are arranged to capture images.
1. An imaging device comprising:
an image sensor having rows and columns of image pixels partitioned into an array of sensor subsections; and
an array of wafer lenses disposed over the image sensor, each of the wafer lenses optically positioned to focus image light onto a corresponding sensor subsection in the array of sensor subsections, wherein each sensor subsection includes unlit pixels that do not receive the image light focused from the wafer lenses, each sensor subsection also including lit pixels that do receive the image light focused from the wafer lenses, wherein a rectangular subset of the lit pixels from each sensor subsection are arranged to capture images.
2. The imaging device of
3. The imaging device of
4. The imaging device of
5. The imaging device of
6. The imaging device of
7. The imaging device of
8. The imaging device of
9. A system comprising:
an image sensor having rows and columns of image pixels partitioned into an array of sensor subsections;
an array of wafer lenses disposed over the image sensor, each of the wafer lenses optically positioned to focus image light onto a corresponding sensor subsection in the array of sensor subsections, wherein each sensor subsection includes unlit pixels that do not receive the image light focused from the wafer lenses and lit pixels that do receive the image light focused from the wafer lenses, and wherein a rectangular subset of the lit pixels are arranged to capture images;
shutter control circuitry coupled to the image sensor to initiate an exposure period of the image pixels; and
processing logic coupled to the image sensor to receive image data from the image sensor and to generate at least one image from the image data.
10. The system of
11. The system of
12. The system of
13. The system of
14. The system of
15. The system of
16. The system of
17. The system of
18. The system of
19. A method of generating an image comprising:
initiating an exposure of an image sensor to image light;
reading out image data generated during the exposure from the image sensor;
sorting the image data into first, second, third, and fourth image data groups from first, second, third, and fourth sensor subsection of the image sensor, respectively, wherein first, second, third, and fourth wafer lenses are optically positioned to focus the image light onto the first, second, third, and fourth sensor subsections of the image sensor, respectively, and wherein each of the sensor subsections include unlit pixels that do not receive the image light focused from its respective wafer lens and each of the sensor subsections also include lit pixels that do receive the image light focused from its respective wafer lens;
cropping the first, second, third, and fourth image data groups to generate first, second, third, and fourth images;
generating a composite image from the first, second, third, and fourth images.
20. The method of
21. The method of
22. The method of
23. The method of
generating a 4-D light field from the first, second, third, and fourth image data groups; and
refocusing the composite image using the 4-D light field.
This disclosure relates generally to optics, and in particular but not exclusively, relates to digital imaging.
Conventional digital imaging devices or cameras have a lens (which may include multiple lens elements) that focuses image light onto an image sensor that measures the image light and generates an image based on the measurements.
Conventional digital imaging devices may have a lens such as lens 120 with a Z axis 121. The size of image sensor 101 and the corresponding size of the image plane that lens 120 focuses image light 199 on may influence the depth of Z axis 121. The image plane and corresponding depth of Z axis 121 to achieve a high resolution image may be relatively large. A larger depth of Z axis 121 may limit the applications or uses of digital imaging device 100, because of space constraints. Therefore, a device or method that allows a digital imaging device to capture high resolution images with a reduced lens depth (Z axis) would be advantageous.
Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.
Embodiments of a system and method for generating an image are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.
Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Directing attention to
Wafer lens array 205 may include multiple wafers stacked on top of each other so that the first through ninth lenses in wafer lens array 205 have multiple lens elements that make up each lens. In this case, first lens light 213 would propagate through the multiple lens elements of the lens in the Z axis of the lens. Using wafer lens technology in wafer lens array 205 may give low-profile camera 200 a substantially smaller Z axis dimension than a conventional camera. Wafer lenses used in wafer lens array 205 may be purchased from Heptagon™. In one embodiment the lenses in wafer lens array 205 have an f-number of f/2.5. It is appreciated that additional filters and optics (not shown in
The nine rectangular subsections 418 illustrated in
Considering the quantity of unlit pixels illustrated in
The embodiment illustrated in
As can be seen in
It is appreciated that the lenses in wafer lens array 205 may not focus image light 199 in a circular pattern as illustrated in
In on embodiment, outside lenses (e.g. first lens 212, second lens 222, and the third lens 232) are configured to have a field of view of 0° to 90° and outside lenses on an opposite side of a wafer lens array (e.g. the seventh lens 272, eighth lens 282, and ninth lens 292) are configured to have a field of view of 0° to −90°. First lens 212 and the seventh lens 272 may have red filters, second lens 222 and eighth lens 282 may have green filters, and the third lens 232 and ninth lens 292 may have blue filters to give each field of view a color image.
One potential advantage of using one standard image sensor (as compared with nine separate image sensors) in low-profile camera 201 is reduction of the software and processing complexity to manage nine separate image sensors. Using multiple discrete image sensors would also likely require a bigger footprint and more complex input/output (“I/O”) connectivity, which may increase cost. Using only one image sensor allows for an existing interface to control the exposure time and line-by-line readout of the one image sensor to capture a plurality (e.g. nine) of images for generating a composite (e.g. super-resolution) image. Furthermore, as describe in connection with
If MFP 525 is disposed over image sensor 201, the pixels in image sensor 201 would be partitioned into four sensor subsections, each capturing an image. The sensor subsection under the red filter would receive and capture the red light of a scene, the sensor subsections under the green filter would receive and capture the green light of a scene, and the sensor subsection under the blue filter would receive and capture the blue light of a scene. The four images could then be used to generate a full color super-resolution image. In MFP 526, the extra green filter is traded for a white filter that passes visible light, but may not pass IR light. The pixels in the sensor subsection under the white filter can be used to capture the overall intensity or brightness of the scene, while the other three sensor subsections capture the red, green, and blue light from the scene. In MFP 527, an IR filter passes the IR light, but filters out the visible light. The IR filter may be centered around a certain infrared wavelength that matches an IR illuminator (e.g. an IR light emitting diode “LED”) that is connected to a camera in which image sensor 201 is disposed in. The pixels in the sensor subsection under the IR filter can be used to capture the IR light from a scene, while the other three sensor subsections capture the red, green, and blue light from the scene.
Readout circuitry 631 may include one or more analog-to-digital converters. Image logic 690 may include a processor, Field-programmable-gate-array (“FPGA”), or other logic. Image logic 690 may include a memory to store image data and super-resolution image 699.
In one embodiment, image sensor 201 is replaced with a high-dynamic-range (“HDR”) image sensor configured to expose first selected pixels for a first integration period (e.g. 5 ms) and expose second selected pixels for a second integration period (e.g. 30 ms). Each sensor subsection of the HDR image sensor has a quantity of first pixel and quantity of second pixels. The first pixels may be in first lines (columns or rows) and the second pixels may be in second lines (columns or rows) of the HDR image sensor. The first lines may be interlaced with the second lines in an every-other configuration. The first pixels and the second pixels may measure image light for different integration times. Shutter controller 605 may initiate the first and second integration period at the same time, but the first integration period may be shorter than the second integration period. With the shorter integration period, the first pixels will measure fewer photons from image light. When the HDR image sensor is readout, the image date from each sensor subsection will include shorter and longer exposure data. Having image data from shorter and longer integration periods may be used to create a composite HDR image from the different sensor subsections that increases the dynamic range between the lightest and darkest sectors of the image.
In the illustrated embodiment, low-profile camera 200 is mounted in the center of a frame 725 of wearable glasses 700, on or about the nose-piece. In the illustrated embodiment, an IR LED 755 is mounted in the bridge or noise-piece of wearable glasses 700. IR LED 755 may be used to illuminate a photography subject to be imaged by low-profile camera 200. IR LED may be used in connection with MFPs 525, 526, or 527. IR LED may emit IR light and camera 200 may image the emitted IR light reflected off of a subject (e.g a hand) to decipher how close the hand is, as a way to measure depth. Multiple images of reflected IR light can be the basis for motion or even gesturing detection.
In one embodiment, two low-profile cameras 200 are disposed on wearable glasses 700. Wearable glasses 700 may include processing logic 650 disposed in right arm 740. Processing logic 650 may include an integrated circuit with hardware, firmware, and/or software logic. Processing logic 650 may be used to receive, transmit, and process data. Processing logic 650 may receive and process image data and generate images based on software algorithms. Processing logic 650 may be located in a location or locations other than in right temple arm 340, such as in left temple arm 730.
With regard to low-profile camera 200 in HMD 700, it desirable to have a large depth of field for camera 200 because a user/wearer of HMD 700 may not spend time to focus the picture. As mentioned above, using wafer lens technology in wafer lens array 205 may give low-profile camera 200 a substantially smaller Z axis dimension than a conventional camera. Some of this savings in the Z axis may be used to construct a wafer lens array that gives camera 200 a large depth of field, which will help capture images that are in focus.
In process block 805, an exposure period of an image sensor is initiated to allow image pixels of the image sensor to measure image light. For example, shutter controller 605 may initiate an exposure period (a.k.a. integration period) of image sensor 201. After measuring the image light, image data measured during the exposure period is read out from the image sensor in process block 810. In process block 815, the image data is sorted into first, second, third, and fourth image data groups. If an HDR image sensor is used (such as the one discussed in connection with
In process block 820, the first, second, third, and fourth image data groups are formatted (e.g. cropped) to generate first, second, third, and fourth images. Each image data groups may be cropped to generate a rectangular image that does not include unlit pixels. The image data groups may also be cropped to generate the largest rectangular image that does not include unlit pixel and that also has a common field of view with the other sensor subsections. As discussed above, the sensor subsections may have different fields of view due to the slightly different physical location of the sensor subsections. However, their fields of view may at least partially overlap giving them a common field of view. A composite image is generated from the first, second, third, and fourth images, in process block 825. The composite image may be a super-resolution image.
One potential feature of low-profile camera 200 is capturing images (with different sensor subsection) that are from slightly different perspectives. Using trigonometry in accordance with the principles of parallax, the distance to an subject/object imaged by camera 200 may be determined by analyzing two images captured by different sensor subsections. For example, an image from one sensor subsection may be compared with an image from another sensor subsection. A common point in each image can be determined by matching edges or corners. Then, an angle(s) of the incoming light can be determined based on which pixel in each image that that common point appears in. The angles are determined, at least in part on how far away the pixels are from each other in the image sensor. The depth information that can be calculated using parallax principles can be useful in gesture detection (e.g. isolating the image of a hand against a background).
The principles of parallax as applied to using camera 200 may also be useful for after-capture-focusing of an image generated by camera 200 (super-resolution or otherwise). Since the angle of image light incident on a pixel can be determined using trigonometry related to how far apart the pixels from different subsections are, a four dimensional (“4-D”) light field can be generated. With the distance of objects in the image also known based on trigonometry, after-capture-focusing can be applied to an image. In one example, image blur is applied to an image based on a calculated distance of objects in the image. In this way, a captured image that is generally in focus (as a result of the large depth of field of the wafer lenses) can have artistic blur correctly added to selected portions of the image (e.g. blurring out objects in an image more than 40 feet from the camera to blur the background). Different scales or layers of blurring may be added to a captured images based on the depth of the object from the camera. In one embodiment, the calculations and after-capture-focusing are completed by processing logic 650.
The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.
A tangible non-transitory machine-readable storage medium includes any mechanism that provides (i.e., stores) information in a form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-readable storage medium includes recordable/non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).
The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.
These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.