I dont understand how exactly stencil buffer works in openGL. One aspect that confuses me is why do we use a bit operator there in glStencilFunc. Some text say that is is used to achieve multiple stencil planes but I am not sure how that can be done. Can anybody please explain the whole procedure of stencil buffer calculation during the rendering cycle. Thank you.
Okay, this does sound like something you can do with stenciling. Stenciling is generally used in conjunction with Multipass Rendering.
Multipass Rendering basically refers to a rendering loop where multiple renders a performed per frame. Each pass may use a different configuration of OpenGL (e.g. with different stenciling settings) and render the same or different artifacts. The pseudo code would look something like this:
In your case, you will need a rendering pass for each layer of rectangles so that you can prepare the stencil buffer before each pass.
Effectively, you want to restrict the pixels that can be drawn in each rendering pass to the set of pixels that were drawn in the previous rendering pass. To do this, you are going to need to do three things:
Record the pixels you draw
This is where
The third argument is the important one, it determines what happens when a pixel is drawn (normally), to understand the function better, read the reference docs.
Only draw pixels that were also drawn in the previous rendering pass
This is a job for
You need to call it before every rendering pass to say 'only draw if the stencil buffer says I drew last time round'. Since the stencil buffer is being incremented each time, the value to test against needs to be incremented too. For example, the first three rendering passes should call
Just ignore the third parameter for now, read the reference documentation if want to know what it is.
EDIT: The third parameter is used when you want to do bit-wise tests against the stencil buffer value. Most implementations allow 8 bits in the stencil buffer (otherwise known as 8 planes). The third parameter allows you to mask out bits that you do not want to include in the test (by ANDing it with the second parameter before doing the test). Since I have used 1 as the third parameter, ANDing it has no effect. Also, since we are treating the stencil buffer as holding integers in this example, bit-wise tests don't make sense. By using these planes, though, we could potentially hold 8 stencils in the buffer at one time and choose which combination of stencils we want to use in each rendering pass.
Remove the record of which pixels have been drawn
Each frame you don't want to continue adding to the existing stencil buffer values just like you don't want to leave the previously rendered image in the frame buffer. You clear it in the same way you would clear the frame buffer too:
Or, do both at once: