|Publication number||US8131062 B2|
|Application number||US 10/551,290|
|Publication date||Mar 6, 2012|
|Filing date||Mar 25, 2004|
|Priority date||Mar 29, 2003|
|Also published as||CN1799070A, CN100382110C, US20090310858|
|Publication number||10551290, 551290, PCT/2004/1262, PCT/GB/2004/001262, PCT/GB/2004/01262, PCT/GB/4/001262, PCT/GB/4/01262, PCT/GB2004/001262, PCT/GB2004/01262, PCT/GB2004001262, PCT/GB200401262, PCT/GB4/001262, PCT/GB4/01262, PCT/GB4001262, PCT/GB401262, US 8131062 B2, US 8131062B2, US-B2-8131062, US8131062 B2, US8131062B2|
|Original Assignee||Atelier Vision Limited|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (5), Non-Patent Citations (6), Referenced by (2), Classifications (10), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention relates to an image processing or manipulation technique, and in particular to a technique that may be used to enhance our appreciation of 3D depth and form depicted in base 2D images and derivative virtual reality (VR) environments. Images processed in this manner may be of improved believability, thus the invention allows an increase in the believability threshold.
When three-dimensional scenes are displayed or depicted using conventional two-dimensional techniques, for example by printing on to paper or the like or displaying on a television, computer monitor or the like, or by projection, although the brain perceives the images displayed as being to some extent three-dimensional in nature, there are occasions when the perception of depth and form in the displayed image is not particularly good. This is thought to be due to the absence of sufficient monocular depth cues in the image to allow the brain to interpret the displayed image fully.
In the following description a distinction is made between ‘pictures’ and ‘images’. The following definitions of pictures and images have been made by Prof Jan Koenderink (Vision Scientist), and the terms as used herein should be interpreted accordingly.
Picture: “a real image, an ordered record, such as a video-signal, a photograph, an activity pattern in a neural structure. The order of the picture is purely conventional and exists only with respect to an external agent.”
Images: “always contain more than pictures because the whole structure of the receiver (a lifetime's experience) is summed up in them.”
This invention relates, at least to some degree, to the isolation of the techniques that transform pictures to images.
The technique of the invention has been developed from theoretical work identifying a new form of illusionary space which has been termed ‘image space’ (thus distinguishing itself from traditional picture space). Image space acknowledges the individual nature and specialisms of peripheral vision and central vision and also that brain function ‘creates’ important facets of visual perception by making ‘relativistic judgments’ between the two. Key to the novelty of this approach is the realization that these new cues rely on as yet unappreciated monocular cues and the composition of monocular projection. These monocular cues distinguish this method of achieving apparent 3D depth from other techniques that rely on stereo cues deriving from binocular disparity etc. By correctly representing and aligning two projections used in monocular vision by the brain to compose perceived images (the projection known as vision) it is possible to induce the brain to replicate depth and form perceptions that could otherwise only be created by directly observing the ‘real setting’ from which the pictorial information derived.
‘Image Space’ can be achieved/induced in a number of ways pictorially.
1. By imbuing traditional picture space (perspective projection or photographic/film record) with the required cues.
2. By using the principles to model a new illusionary space to generate novel virtual reality (VR) environments.
It is an object of the invention to provide a system, which may be used in combination with existing techniques, whereby an image may be created or modified to enhance the perception of the depth in the original picture. In each case, image space relies on the incorporation of stimuli from which the brain is able to deduce additional perceptions of depth and form associated with our sense of visual perception.
According to one aspect of the invention there is provided an image processing technique comprising the steps of:
(a) stretching an original picture in the Y-direction by a factor falling within the range of 2-10%;
selecting a fixation point and disordering the image centering the disordering operation around the fixation point, and
rotating the image through an angle falling within the range 3-9°;
(b) stretching a copy of the original image in the X-direction by a factor falling within the range of 2-10%; and
selecting an area of the image around the selected fixation point; and
(c) merging the selected area of the image formed in step (b) with the image formed in step (a).
In step (a), the rotation of the image is preferably in a clockwise direction. A minority of individuals may prefer the image to be rotated in the opposite direction and which the technique is used to produce an image to be viewed by such individuals, the rotation may be undertaken in the anti-clockwise direction.
The disordering operation preferably involves disordering in line with a chosen self-similar fractal disorder pattern.
The image formed as a result of the transformation set out in step (a) will hereinafter be referred to as a Type 1 image, and that produced in the transformation set out in step (b) will be referred to as a Type 2 image.
This technique can be thus be used to:
i) mimic the appearance of the two base projections used by the brain in our monocular projection of the real world by transforming known picture environments.
ii) replicate a typical composition/structure of these base projections used by the brain to form perceived visual images.
The key to forming a valid composition is the selection of a fixation point, the establishing of a fixation area (conforming to known constraints of central vision) and the correct imposition of conditions prevailing in ‘the fixation volume’, peripheral vision and around the interface between the Type 1 image and the Type 2 image. As in real time observation, a fixation point needs to be established on the media. The fixation point is the point around which the various transformations techniques are to be centered. It is thought that the enhanced 3D depth and form cues are established by:
Without a central point of alignment (fixation point) from both data sets, it would be impossible for the brain to make the required calculations. The fixation area establishes an area around the fixation point broadly analogous to the extent of central vision. As such, the fixation area establishes the extent of the image to be subjected to the specific conditions of the fixation volume. The fixation volume is a spherical area of influence contained within the fixation area. Any form (object), surface or part of a form that projects into this area of influence is projected in the brain as per the qualitative and quantitative characteristics of image type 2 above.
It follows that any other visual reference to objects outside of this fixation volume contained in the fixation area are referenced as per the image type 1 above.
Areas of central vision outside the fixation volume are referenced as per the characteristics of ‘peripheral vision’.
There are a number of typical compositions used by the brain, which utilize aspects, and juxtapositions of the two image types in away capable of generating the relativistic judgments from which the new cues are deduced.
Images processed in this manner are thought to conform more fully with our projection of real-time monocular observations (visual images) allowing the brain to correctly interpret a fuller range of three-dimensional cues from the 2D image or virtual reality setting. The projection of monocular vision would appear to be compiled from two data sets which allows the brain to have certain degrees of freedom when compiling visual field. It has been noted that these potential compositions have been adapted to suit the specific task being undertaken by the viewer. For example, reading 2D literature requires little 3D information. This order of information could even provide a distraction to the task in hand. In this situation only one data set is used reducing the visual information being processed. A specialist composition is used when detailed spatial analysis is required. One where half of central vision is suppressed allowing for aspects of Type 1 information to spread into central vision up to the fixation point. This process allows the important brain function responsible for making relativistic judgements and hence enhances form cues to play over the actual object held in fixation. Aspects of this composition are often revealed in the work of visual artists painting still life material.
The nature and structure of information contained in peripheral vision is still largely unresolved by vision science. This invention uses the qualities of self-similar (fractal) disorder (not blur) to replicate the order of information projected outside central vision (peripheral vision). This invention identifies a novel step in attributing peripheral vision with an independent 3D dimension centred on fixation. ie, the level of disorder generates out from fixation in every direction. Hence, 2D pictures are transformed to replicate this order of information by building layers of disorder backwards, forwards and outward on the vertical plane from the targeted fixation. This establishes a new form of spatial dimension in the image. For still images this technique is seen to enhance depth perception in the image. However, the real advantage is thought to lie in its ability to impart accurate renditions of perceived movement in moving images. Once mastered, movement around the targeted fixation will now be closer to our perception of movement observed from ‘real settings’. This 3D information can be captured for use in the transformation process as a ‘depth map’ using specialist equipment, for example making use of laser imaging or LED lighting technologies combined into film cameras.
Moving images will also require changes to the targeted fixation points as the action depicted on them develops. Techniques for inducing a change in fixation will be incorporated into action depicted in moving media to lead the viewer through the eye movement procedure.
The processing technique may require additional steps of fine-tuning around the boundary between the images formed in steps (a) and (b).
Step (a) of the processing technique may further include steps of altering the contrast of Type 1 information by a factor falling within the range of +10-40% and/or decreasing the brightness of the image by a factor falling within the range 2-40%. If contrast is increased in this way, then colour saturation should preferably be reduced to an accommodation level (usually to match the increase of level in contrast).
The technique defined hereinbefore may be performed using a computer on image data derived, for example, from a scanner or digital camera, or on digital picture data obtained or created from other sources including computer programs. However, it will be appreciated that the technique may be performed in other ways, and the invention is not restricted to the specific arrangements or applications described herein.
The invention will further be described, by way of example, with reference to the accompanying drawings, in which:
The invention will, initially, be described with reference to the image illustrated in
In accordance with the invention, the base image of
Another copy of the image is stretched by a factor of 2-10% in the X-direction (horizontal). Preferably, it is stretched to ensure a differential with the image in the previous transformation of around 5-20%. A fixation point on the image is selected, and an area around the fixation point is selected, the remainder of the image being deleted. The result of this operation is shown in
The two images produced in this manner are superimposed upon one another. The images may be superimposed using several different techniques.
Referring next to
This percept of 3D form and depth is, in some ways, analogous to the percept obtained by perspective space. However, it is now thought that perspective cues are not the principle method by which spatial depth is obtained in central vision or through the full extent of visual field. Rather, we see spatial depth and form by a process that involves our visual system making relativistic judgments between two distinct data sets or image types. The arrangement identified above stimulates the brain to engage with this process when observing 2D space, providing us with cues more conversant with our appreciation of real settings.
The new image space allows us to create depth in a similar way to the way it is perceived in our actual visual appreciation of our surroundings. Looking at the new image space will be closer to an actual act of real time observation.
Referring next to
For example; around the neck of the bottle the stones of the wall, although within the circumference of the fixation area of
For example; one arm of the candelabra penetrates the fixation volume and so is referenced as per image Type 2 while the rest of the object is outside this area of influence and so is rendered as per image type 1.
It is thought that images processed using the techniques described hereinbefore are of improved or increased resemblance to images projected by the brain. Using this structure/armature in digitised medial, it will be possible to enter many facets of the differences known to exist between peripheral vision and central vision. Clear and accurate representation of perceived visual image compositions allows an observer of the 2D rendition to engage in additional computations leading to a more viable perception of the real setting being depicted.
An observer of the transformed image will need to look at (fixate on) the selected fixation point to perceive/appreciate the full impact of the additional cues.
In the description hereinbefore reference has been made to a step of disordering the image. Although a number of disordering techniques may be used, a preferred technique is to incrementally disorder the image using, for example, the sunflower-like fractal pattern shown in
It is thought that images processed using the techniques described hereinbefore provide an improved or increased perception or projection of depth to the part of the image around the fixation point and through peripheral areas. The enhanced perception of depth is apparent both where a viewer views the image through both eyes and when the image is viewed using only one eye. Stereo blind subjects perceive the additional depth these transformed images contain. In contrast, the organised depth of stereo based assemblages fail to engage stereo blind individuals.
Although the techniques have been described hereinbefore in relation to the processing of pictures, either in the form of computer generated pictures or photographs, it will be appreciated that this need not be the case and the invention may be applied to a wide range of technologies and display techniques including computer or television displays, projected images, virtual reality displays, animations and printed media. The invention may also find application in, for example, cameras (either for stills or for moving images) which are designed to incorporate a display on which the transformed image can be displayed in “real time” so that the photographer, director or other person involved in the capturing of the image can see how it will be transformed before the image is captured, or completion of the capture takes place. In order to accommodate this, the camera may be provided sensors to allow the depths of objects in the field of vision to be measured at the same time as the image is captured, these two components being used in conjunction with one another and in conjunction with the techniques described hereinbefore in the transformation of the image. Additionally, the location of the fixation point in the field of view will need to be input. This could be set at the centre of the field of view or, using appropriate input means, may be selected by the director, photographer or other individual.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US6157733 *||Apr 18, 1997||Dec 5, 2000||At&T Corp.||Integration of monocular cues to improve depth perception|
|US7113650 *||Jul 30, 2005||Sep 26, 2006||Microsoft Corporation||Real-time wide-angle image correction system and method for computer image viewing|
|US20020139920 *||Aug 23, 2001||Oct 3, 2002||University Of Washington||Image acquisition with depth enhancement|
|US20020167512||May 8, 2001||Nov 14, 2002||Koninklijke Philips Electronics N.V..||N-view synthesis from monocular video of certain broadcast and stored mass media content|
|EP0514266A2||May 13, 1992||Nov 19, 1992||Sony Corporation||Apparatus and method for transforming an input image|
|1||*||Favaro, "Depth from Focus/Defocus", http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL-COPIES/FAVARO1/dfdutorial.html, Jume 2002.|
|2||*||Favaro, "Depth from Focus/Defocus", http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL—COPIES/FAVARO1/dfdutorial.html, Jume 2002.|
|3||*||Pentland "ANew Sense for Depth of Field", Pattern Analysis and Machine Intelligence, IEEE Transactions, Jul. 1987, pp. 523-531.|
|4||*||Subbarao et al. "Depth from Defocus: Spatial Domain Approach", International Journal of Computer vision, 13, 1994, pp. 271-294.|
|5||*||Swain "Intergration of Monicular Cues to Create Depth Effect", IEEE, 1997, pp. 2745-2748.|
|6||Swain, C.T.: "Integration of Monocular Cues to Create Depth Effect", Acoustics, Speech, and Signal Processing. 1997. ICASSP-97, 1997 IEEE International Conference on Munich, Germany Apr. 21-24, 1997, Los Alamitos, CA, USA. IEEE Comput. Soc., US Apr. 21, 1997, pp. 2745-2748, XP010225724, ISBN: 0-8186-7919-0 abstract, p. 2746, left hand column, line 27-line 31.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US9076269 *||Jan 31, 2012||Jul 7, 2015||Sony Corporation||Method and electronic device for creating a combined image|
|US20140219581 *||Jan 31, 2012||Aug 7, 2014||Sony Mobile Communications Ab||Method and electronic device for creating a combined image|
|U.S. Classification||382/154, 356/12, 345/419|
|International Classification||G06K9/00, G01C3/14, G06T15/00|
|Cooperative Classification||H04N13/0018, G06T11/00|
|European Classification||G06T11/00, H04N13/00P1B|
|May 19, 2008||AS||Assignment|
Owner name: ATELIER VISION LIMITED, UNITED KINGDOM
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:JUPE, JOHN;REEL/FRAME:020972/0072
Effective date: 20051205
|Oct 16, 2015||REMI||Maintenance fee reminder mailed|
|Mar 6, 2016||LAPS||Lapse for failure to pay maintenance fees|
|Apr 26, 2016||FP||Expired due to failure to pay maintenance fee|
Effective date: 20160306