US 7525543 B2
A method and system for shading large volumetric data sets using partial derivatives computed in screen-space. The method and system provide a fast and efficient shading a large datasets. Screen space derivatives are computed efficiently by evaluating neighboring pixel information together. The method can be efficiently implemented using GPUs (graphics processing units) that provides some access to information about neighboring pixels using ddx and ddy fragment processing functions.
1. A shading method for rendering volumetric data having a plurality of pixels, each of the plurality of pixels having a scalar value, comprising:
direct rendering of the volumetric data, including the steps of:
determining a partial derivative with respect to the x direction of a first of the plurality of pixels in a screen-space;
determining a partial derivative with respect to the y direction of the first of the plurality of pixels in the screen-space;
determining a partial derivative with respect to a z direction of the first of the plurality of pixels in an object-space; and
determining a gradient based on the partial derivative with respect to the x direction, based on the partial derivative with respect to the y direction and based on the partial derivative with respect to the z direction.
2. The method as claimed in
3. The method as claimed in
determining a partial derivative with respect to the x direction of each of the remaining plurality of pixels in the screen-space;
determining a partial derivative with respect to the y direction of each of the remaining plurality of pixels in the screen-space;
determining a partial derivative with respect to the z direction of each of the remaining plurality of pixels in the object-space; and
determining a gradient for each of the remaining plurality of pixels based on the associated partial derivative with respect to the x direction, based on the associated partial derivative with respect to the y direction and based on the associated partial derivative with respect to the z direction.
4. The method as claimed in
5. The method as claimed in
6. The method as claimed in
7. The method as claimed in
8. The method as claimed in
9. The method as claimed in
10. The method as claimed in
11. The method as claimed in
12. A system of shading for rendering volumetric data, comprising:
a device for imaging a body part to derive scalar values associated with a plurality of pixels;
a processor that, for each of the plurality of pixels, determines a partial derivative in an x direction in a screen-space, determines a partial derivative in a y direction in the screen-space, determines a partial derivative with respect to a z direction in an object-space and determines a gradient based in part on the partial derivative in the x direction, on the partial derivative in the y direction and on the partial derivative with respect to the z direction in the object-space;
wherein the processor performs direct rendering of volumetric data based in part on the gradient; and
a display that displays the rendered volumetric data.
13. The system as claimed in
14. The system as claimed in
15. The system as claimed in
16. The system as claimed in
17. The system as claimed in
18. The system as claimed in
19. A method of rendering volumetric data having a plurality of pixels, comprising:
direct rendering of the volumetric data, including the steps of:
obtaining a sample of data from a volume;
classifying the sample;
computing a gradient for each of the plurality of pixels, the gradient being based at least in part on a partial derivative with respect to an x direction in a screen space, on a partial derivative with respect to a y direction in the screen-space and on a partial derivative with respect to a z direction in an object-space;
normalizing the gradient;
computing diffuse and specular components; and
computing output colors.
20. The method as claimed in
21. The method as claimed in
22. A shading method for rendering volumetric data, the method comprising:
performing direct rendering including the steps of:
selecting a first pixel in screen-space coordinates, the first pixel having a corresponding scalar value obtained from the volumetric data;
determining a difference between the scalar value associated with the first pixel and a scalar value associated with a neighboring pixel in a x direction in screen space;
determining a difference between the scalar value associated with the first pixel and a scalar value associated with a neighboring pixel in a y direction in screen space;
determining a difference between the scalar value associated with the first pixel and a scalar value associated with a neighboring pixel in a z direction in object space;
determining a gradient associated with the first pixel based on the differences; and
using the gradient to compute a shading characteristic of the first pixel.
This application claims the benefit of U.S. Provisional Application No. 60/600,428, filed Aug. 9, 2004, which is incorporated herein by reference.
Techniques for rendering volumetric data sets are known in the art. These techniques have been described in detail. See, for example, Marc Levoy, A Hybrid Ray Tracer for Rendering Polygon and Volume Data, IEEE Computer Graphics and Applications, pages 33-40, March 1990; and Marc Levoy, Efficient Ray Tracing of Volume Data, ACM Transactions on Graphics, 9(3):245-261, July 1990, both of which are incorporated herein by reference.
Fast rendering of volumetric data sets using the texturing capabilities of graphics hardware has also been described in detail. See, for example, B. Cabral, N. Cam and J. Foran, “Accelerated Volume Rendering and Tomographic Reconstruction Using Texture Mapping Hardware”, ACM Symposium on Volume Visualization, 1994; C. Rezk-Salama, K. Engel, M. Bauer, G. Greiner, T. Ertl, “Interactive Volume Rendering on Standard PC Graphics Hardware Using Multi-Textures and Multi-Stage Rasterization”, In Euographics/SIGGRAPH Workshop on Graphics Hardware, 2000, pages 109-118, 147, Addison-Wesley Publishing Company, Inc., 2000, which are incorporated by reference. Shading models for volumetric data are also discussed in N. Max, “Optical Models for Direct Volume Rendering”, IEEE Transactions on Visualization and Computer Graphics 1995, pages 99-108, which is incorporated herein by reference.
Shading is a well-known technique to enhance depth-perception and accentuate details contained within the data that can be performed in real-time. Shading volumetric data requires gradient information for each voxel. Gradients can either be pre-computed and stored along with the volume data or computed on-the-fly. Pre-computing gradients significantly increases the amount of memory required, thereby making this approach impractical for large volume data sets. Clinical routines currently generate data sets exceeding half a gigabyte, and the amount of information being acquired is increasing rapidly. New systems are capable of generating data sets that are more than a gigabyte in size. Furthermore, gradients are typically quantized to either 8 or 16-bits to constrain the amount of memory required, which decreases the quality of the volumetric renderings. On the other hand, although computing gradients on-the-fly provides better quality, and neither requires pre-processing nor imposes any significant additional memory demands, it suffers from significantly slower performance. This reduction in performance is due to the additional computations and memory accesses during rendering; for example, a forward difference gradient requires accessing three neighboring voxels, while a central difference gradient requires accessing six neighboring voxels.
It is therefore desirable to provide methods and systems that perform faster on-the-fly gradient estimation for shaded rendering of volumetric data.
This invention provides a fast method to enhance image quality in texture-based volume rendering by shading large volumes without pre-computing gradient information. Gradients are computed in screen-space, instead of in object-space, to reduce the overhead of acquiring neighboring data by computing multiple pixels at a time. This invention is well-suited for modem GPUs (graphics processing units), as they process multiple pixels at a time (e.g., a 2×2 pixel “quad”).
In accordance with one aspect of the present invention, a shading method for rendering volumetric data is provided. The method includes the steps of determining a partial derivative with respect to the x component of a screen coordinate of a pixel, determining a partial derivative with respect to a y component of the screen coordinate of the pixel, and determining a gradient from the partial derivatives in screen-space. In accordance with another aspect of the present invention, the method includes the step of using the gradient to determine a shading characteristic of the pixel based on the gradient.
In accordance with a further aspect of the present invention, the method includes performing the steps of determining the partial derivatives in a graphics processing unit. In accordance with a preferred embodiment, the partial derivatives are calculated using a fragment function ddx in the graphics processing unit and a fragment function ddy in the graphics processing unit.
In accordance with a further aspect of the present invention, the method of rendering volumetric data is applicable on a variety of imaging systems. In accordance with these aspects, systems of the present invention include an imaging device, a processor and a display.
In accordance with another aspect of the present invention, a method of rendering volumetric data includes the following steps: obtaining a sample of data from a volume; classifying the sample; computing a screen-space gradient; normalizing the screen-space gradient; computing diffuse and specular components; and computing output colors. The results obtained from these steps are eventually displayed.
The data processed by this invention is a three-dimensional array of data. Referring to
When performing full body scans with a magnetic imaging machine, it is common to generate approximately 3000 slices of data. These slices may be approximately 0.4 mm apart.
The acquisition of this type of data is well known in the art and is produced by systems such as Magnetic Resonance (MR) imaging systems, Computed Tomography (CT) scanning systems, Positron Emission Tomography (PET), and Ultrasound scanning. Such imaging systems 70 are shown in
Image Synthesis from Volumetric Data.
Methods to generate a two-dimensional image from volumetric data can roughly be classified into indirect and direct volume rendering techniques. Indirect methods generate and render an intermediate representation of the volume data, while direct methods display the voxel data by evaluating an optical model that describes how the volume emits, reflects, scatters, absorbs and occludes light. See, for example, N. Max, “Optical Models for Direct Volume Rendering”, IEEE Transactions on Visualization and Computer Graphics 1995, pages 99-108.
The voxel values are mapped to physical quantities describing light interaction at the respective points in 3 D-space. During image synthesis the light propagation is computed by integrating light interaction effects along viewing rays based on the optical model. The corresponding integral is known as the volume rendering integral.
The data in a slice as shown in
Shading of Volumetric Data.
Shading provides visual cues to aid in the visualization and interpretation of volumetric data. Shading can accentuate small-scale surface structures and texture (roughness), as well as additional depth cues. Shading takes into account the effect of external light sources by computing the interaction of light at each point where it illuminates the surface, taking into consideration the properties of the surface at each point illuminated by a ray. The properties consist of material properties such as color, opacity and shininess, as well as the local shape of the surface.
The most computationally expensive part of the shading equation is computing the local shape, which is typically represented by the local gradient. The most frequently used techniques for computing gradients for a volume are the forward and central differences methods. See, for example, Abramowitz, M. and Stegun, I. A. (Eds.), Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th printing, New York: Dover, p. 877, 1972, which is incorporated herein by reference. Previously, gradients were computed on object-space data. Also, these gradients are frequently computed in a pre-processing step and stored in an adjunct volume or combined with the source data in another volume. However, the additional memory required to store gradient information makes this approach ill-suited for large volume data. Consequently, gradients are computed on-the-fly during rendering. On-the-fly computation of gradient information requires evaluating the neighborhood of each voxel being rendered, which for a forward difference involves evaluating three neighbors, while a central difference involves evaluating six neighbors. The present invention reduces the cost of accessing and evaluating neighboring information, and therefore considerably increases performance.
In one embodiment of the present invention, the computation of the gradient information can be moved from a central processing unit (CPU) to a fragment program on graphics processing unit (GPU). There are a few advantages to this approach. First, the GPU can typically perform such vector operations faster than a CPU—GPUs are single instruction multiple data (SIMD) parallel processors. Second, the effective cost of the gradient computation is reduced due to the GPU pipeline, where the same fragment program is typically executed in on neighboring pixels in parallel by multiple pixel pipelines. That is, one fragment pipline do not impose any additional delay because it executes in parallel with, and independent of, other fragment pipelines. Third, moving the computation to a fragment program can reduce the amount of data transferred from the CPU to the GPU via the relatively slow AGP (Accelerated Graphics Port) or PCI Express (data is transferred using AGP or the newer PCI Express interface from main memory into local GPU memory or vice versa).
Screen-Space Partial Derivatives.
This invention uses partial derivatives computed in screen-space for the gradients used in shading calculations. This approach requires access to information for rays traversing neighboring pixels having the same depth with respect to the camera; that is, a scalar physical property p (such as density) of the volume at pixels with the same depth as the ray at the current pixel position has to be evaluated. In fact, any method for computing screen-space partial derivatives (such as forward differences, central differences, a Sobel operator, etc.) can be employed; however, in the following, the focus is on forward differences. In particular, in addition to the value of the current pixel at the screen-space position (x, y), denoted by p(x, y) with a given depth with respect to the camera, a forward difference gradient requires the values of the pixels to the right (p(x+1, y)) and above (p(x, y+1)) the current pixel. A 2 D forward difference screen-space gradient is then computed as:
As discussed in more detail below, the present invention's use of finding the gradient in screen-space is particularly well-suited for GPUs. However, it should be clear that significant speed increases can be found without necessarily requiring a GPU. For example, certain volumetric rendering algorithms involve casting groups of rays in neighboring pixel locations through the volume in parallel. During the computation, the volume densities (i.e., the scalar parameter p(x, y)) of these neighboring rays may be cached in registers, and hence fetching the densities can be quickly performed for the gradient calculations.
In order to produce a 3 D vector, a third component is required. This component may be, for example, an arbitrary constant, a forward difference computed in object-space along the ray direction, or a central difference computed in object-space along the ray direction. The most computationally efficient approach is to use a constant (whose value is proportional to brightness), while higher-quality results can be obtained from object-space central or forward differences.
Finally, it has been determined that the third component may be computed as a value that is proportional to the brightness and that is scaled according to the zoom value used to view the volumetric data. This has the benefit of assuring that the apparent brightness remains constant, regardless of the zoom value, since at higher and higher zoom values rays through neighboring pixels in screen space get ever closer in object space. Thus the offsets for the lookups of neighbors in screen space get smaller with higher zoom factors. A constant for the third component (the object-space partial derivative) tends to become relatively greater with respect to the first and second components (the screen-space partial derivatives), resulting in a brightness that appears to increase with the zoom. Thus compensation factors are introduced which scale each screen-space partial derivatives with a constant that is dependent on the zoom-factor. These constants are computed by projecting the distance between the samples used for the object-space partial derivative into screen-space and determining the number of pixels covered by this distance along the screen-space x- and y-directions.
If a central differences method is used, then the partial derivative in the x direction is determined by the difference between the scalar value associated with pixel X and the scalar value associated with pixel X1. Also, the partial derivative in the y direction is determined by the difference between the scalar value associated with pixel Y and the scalar value associated with pixel Y1. As can be seen, this requires access to values outside a Quad, and therefore cannot be used on the nVidia graphics processor.
As previously discussed, the partial derivative in the z direction is preferably determined using object-space data.
As an example, the following C code provides a utility function to compute compensation factor to make shading independent on zoom-factor. The code takes as an input distance between samples along viewing direction and outputs scaling factors for screen-space partial derivatives:
GPU-Based Screen-Space Derivative Shading.
nVIDIA's CineFX architecture facilitates the fragment functions ddx and ddy to approximate the partial derivatives of a scalar value with respect to screen-space coordinates. Specifically, ddx(a) approximates the partial derivative of a with respect to the x-component of the screen coordinate, while ddy(a) approximates the partial derivative of a with respect to the y-component of the screen coordinate. These partial derivatives are computed very efficiently by the graphics hardware because several neighboring pixels are processed in parallel. The values of rays at neighboring positions are cached internally so that it is more efficient than accessing them using conventional direct texture access. Screen-space partial derivatives can exploit architectural features of GPUs.
By applying the ddx and ddy functions to the values obtained from sampling the volume at neighboring pixels, one can obtain information about the change of data values inside the volume in screen-space. This gradient information can be used for the Blinn-Phong shading model to evaluate the local illumination. Referring to
The first step 50 obtains a texel value from the 3 D texture representing the volume using the current texture coordinate (which is an input to the fragment program) as the index to the 3 D texture. Tri-linear interpolation is automatically performed by the hardware during this step.
Classification is performed in the second step 52. In accordance with one aspect of the present invention, classification is performed using a dependent texture lookup with the texel value as the index to the transfer function.
In the third step 54, a gradient is determined in the screen-space. This means that neighboring density information is determined dependent on x- and y-coordinates of the current pixel, rather than the x-, y- and z-coordinates of the current sampling position inside the volume. As previously mentioned, the gradient can be calculated in different ways. In accordance with one aspect of the present invention, the gradient can be calculated by determining a partial derivative in a screen-space with respect to a plurality of pixels in screen space, each having different screen coordinates. Then a gradient is determined from the partial derivatives. The number of pixels used depends on the method used to calculate the gradient, ie whether a forward difference method or another method is used.
In accordance with one aspect of the present invention, the gradient is determined by determining the partial derivative of a first pixel having a first screen coordinate and by determining a partial derivative in the screen-space with respect to a second pixel having a second screen coordinate. The gradient is then determined based on the results. Once again, the number of pixels used is dependent on the method used to calculate the gradient.
In accordance with a further aspect of the present invention, the step of determining the partial derivative with respect to a pixel includes the steps of determining a partial derivative with respect to a x-coordinate of the pixel and determining a partial derivative with respect to a y-coordinate of the pixel. Once again, the number of pixels used to determine the partial derivative is dependent on the method used to calculate the gradient.
In accordance with another aspect of the present invention, the partial derivative of the x-component of a pixel is obtained using the ddx function with the texel obtained from the first step. The partial derivative of the y-component of the gradient is obtained using the ddy function. The z-component of the gradient can be a constant, the object-space derivative in the z direction using the difference between the texel obtained from the first step and a second texel value obtained from a second texture coordinate, or a value scaled according to the zoom.
Thus, as illustrated in
The gradient is normalized in the fourth step 56. The gradient is then used in the fifth step 58 to compute the diffuse and specular components. Finally, the output color is computed in step 60 using the classified color (from step 52), diffuse and specular parameters (from step 58) and material parameters (which are passed into the fragment program as constants).
This process is repeated for each sampling position in a dataset to render volumetric data. For example, shading characteristics can be determined using this process.
The present invention is not limited to merely the specular and diffuse components of shading. Screen-based partial derivatives can be used for other types of shading as well, such as opacity gradient modulation, where the opacity (alpha component) is modulated by the magnitude of the gradient to focus on the edges within the volume.
The following software, using NVIDIA's Cg programming language, can be used in accordance with one aspect of the present invention. The scaling factor below is obtained from the code above so that the brightness of the displayed image does not change with the zoom value. In the code below, the scaling is performed on the x and y value such that the x and y values are decreased as a zoom out operation occurs and such that the x and y values are increased as a zoom in operation occurs. Of course, the z component can also be scaled according to the zoom operation (with different code) to achieve the same result.
In this program, the volume is bound to a 3 D sampler (Volume), and the transfer function is bound to a ID sampler (Transfer function). First, a density value is fetched from the volume texture using the texture coordinates passed in to texture stage 0 (TexCoord0). Then, classification is performed using a dependent texture lookup into the transfer function. Screen-space partial derivatives of the volume density are evaluated using DDX and DDY commands, and scaled based upon the zoom value as previously discussed. An optional object-space partial derivative is evaluated for the z-component using a texture coordinate that is offset by the samping distance along the view-space z-direction (TexCoord1). Then the gradient is normalized and diffuse and specular shading parameters for a Blinn-Phong shading model are computed. Finally, the output color is computed and returned from the fragment program.
To further optimize the fragment code, the diffuse and specular shading coefficients can be pre-computed and stored in a look up table (lookup-texture). The following code illustrates the computation of the look up table for given Blinn-Phong shading parameters (ambientparam, diffuseparam, specularparam, shininessparam), a given light direction (lightDir), a given halfway light direction (halfway) and a lookup table size (1size).
Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It will be apparent to those skilled in the art that various modifications and variations can be made to the method of the present invention without departing from the spirit and scope of the invention. Thus, it is intended that the present invention include modifications and variations that are within the scope of the appended claims and their equivalents.