64 LOW-LEVEL RENDERING CHAPTER 3 You can approximate a smooth unique vertex normal by averaging the normals of the adjacent triangles, possibly weighted by the triangle areas: n = i a i n i , (3.2) where a i is the area of the triangle associated with n i , and n again needs to be normal- ized. This gives a smooth look to the surface. A more involved approach would be to fit a smooth surface to the neighborhood of a vertex and to evaluate the surface normal at the vertex. A unique normal vector for each mesh vertex is useful for smooth objects, but fails if we actually want to represent a sharp crease. The solution is to define the vertex as many times as there are different orientations around it. Similarly vertices need to be replicated if other properties, such as material attributes, should change abruptly between triangles. The simplest approach for determining the color of a primitive is to choose the coloring at one vertex and use it for the whole triangle; this is called flat shading. More pleasing results can be obtained by interpolation, however. There are two key approaches for such inter- polation: one could either first interpolate the normal vectors and then separ ately shade each pixel within a triangle (Phong shading [Pho75]), or alternatively calculate the shad- ing at the vertices and interpolate colors across the triangle (Gouraud shading [Gou71]). Phong shading produces visually better results than Gouraud shading. However, Gouraud shading is much cheaper to calculate, for two reasons. First, linear interpolation of colors is less expensive than interpolation of orientations. Second, assuming triangles typically cover several pixels, the lig hting equations have to be evaluated less often: only at the ver- tices. For this reason OpenGL only supports Gouraud shading, and all lighting happens at vertices. However, we will see in Section 3.4.1 that one can modulate both the colors and the apparent orientations within the triangles using texture and bump mapping. If the surface is transformed by matrix M, each vector that is normal to the original surface must be transformed by M −T , the inverse transpose of M, so that it remains perpendicular to the transfor med surface (see Section 2.2.4 for proof). If M only consists of rotations and translations, the length of the normal vectors does not change. By default OpenGL does not rescale the normals after the transformation; therefore, if more complex trans- formations are used, the user needs to ask the system to renormalize the normals. Too long normals will make the surface appear too bright, and too short normals make it too dark. 3.2.3 REFLECTION MODELS AND MATERIALS Lighting and reflection models can be classified into global and local ones. A global light- ing model accounts for shadows caused by other objects or self-shadowing due to other parts of the same object, as well as light reflected from other surfaces, requiring com- plicated analysis of spatial relationships and perhaps access to all of the scene descrip- tion. Real-time graphics engines such as OpenGL have traditionally used much simpler SECTION 3.2 LIGHTING 65 local lighting models. Local models ignore the effects of other surfaces of the scene, and only require that you know the position and normal vector at a single surface point, various local material properties, and the light sources that potentially illuminate the surface point. However, e ven a local lighting model can have a complicated reflection modeltodeter- mine how much and what kind of light is reflected toward the camera. Some materials, such as hair, silk, velvet, or brushed metal, reflect different amounts and colors of light to different directions, defying simple analytic reflection models and requir ing sampling and tabulating the reflectance function into a bidirectional reflectance distribution function (BRDF). For some other materials it is possible to define a function that approximates the actual reflectance behavior quite closely. OpenGL uses a combination of several simple reflectance models that for the most part are not very accurate models of true materials, but are a reasonable approximation for some materials such as plastics or paper. Off-line photorealistic rendering systems used for special effects, advertisements, and even feature-length movies use much more realis- tic, but computationally more involved lighting and reflectance models. The main com- ponents of the OpenGL lighting model, ambient, diffuse, and specular reflectance models are illustrated in Figure 3.6 and described in the following text. We also discuss materials that emit light, e.g., neon signs. Ambient reflectance Ambient reflectance is the simplest reflectance model. The idea is that the ambient light has been reflected and scattered around the scene so many times that it permeates the whole scene without any directional preference. Assuming we have ambient light I a with red, green, blue, and alpha components, and a surface material k a with the same compo- nents, the light projecting to the camera is simply I = k a I a , where the matching compo- nents are multiplied (red of I is the red of k a times red of I a , and so forth). This simple equation uses very little information, just the color of the ambient light and the ambient material reflectance coefficient. In particular it does not use any information Figure 3.6: A densely triangulated sphere with ambient, diffuse, and diffuse + specular shading. 66 LOW-LEVEL RENDERING CHAPTER 3 that relates to the direction of the light, the surface normal vector, or the viewing direction. Therefore it does not encode any information about surface shape, and the image of a sphere on the left in Figure 3.6 appears as a flat disk. Since ambient lighting loses all shape information, it should be used as little as possible. However, the other reflectance models typically do not illuminate the side of the object opposing the light, and using ambient lighting allows at least the object outlines to be drawn in their own colors, instead of being just black. Diffuse reflectance Diffuse reflectance takes into account the direction of the incoming lig ht with respect to the surface normal to calculate the amount of light reflected toward the camera. Since it assumes that the material reflects all the incoming light uniformly into every direction away from the surface, we do not need the direction to the camer a. With this assumption, the diffuse reflectance rule follows from geomet ry. In Figure 3.7 we have two bundles of rays coming to the surface, one bundle perpendicular to the surface (dark) and one in an angle θ (gray). Denoting a unit area by the width of the ray bundle in the image, the dar k bundle illuminates the unit area, reflecting then all the light out to every direction that can actually see the surface point (the dark arc). However, the gr ay bundle comes in at an angle, and thus illuminates a larger surface area (larger by factor 1 cos θ ), and therefore reflects out only a factor cos θ of the incoming light per unit area (the gray arc). Therefore, for diffuse reflectance we get I = k d I d cos θ. By denoting the surface unit normal by n and the unit direction vector to the light source by l, we see that cos θ = n · l. Finally, only Figure 3.7: Geometry of diffuse reflectance. Rays coming perpendicular (dark) to the surface illu- minate the surface more and also reflect more light away (dark arc). Rays coming in an angle θ (gray) illuminate an area larger by factor 1 cosθ , thus a unit area reflects only cosθ times less light (gray arc). SECTION 3.2 LIGHTING 67 light directions that can see the surface point are considered, so the negative values are clamped to zero, which we denote by parentheses with a plus sign as a subscript ( · ) + , yielding I = k d I d n · l + . No real surface material is completely diffuse, but many materials such as dust, chalk, or rough paper can be approximated fairly well with the diffuse reflectance model. Even then the approximation breaks down at grazing angles, i.e., when θ is small. You can test this by taking a piece of paper and looking at a bright light source almost along the surface of the paper; the paper reflects the light like a very dull mirror. Specular reflectance The specular reflection accounts for the highlights that you see on shiny objects, like the rightmost sphere in Figure 3.6. Whereas in ambient or diffuse reflectance the direc- tion to the v iewer does not matter, that direction is important in specular reflectance. In Figure 3.8, most of the light coming from l is reflected to the mirror reflection direction r, that is, to the direction of the light source reflected about the normal vector n.The larger the angle θ between the viewer direction v and r, the less light is reflected toward the viewer. One way to approximate this drop-off is using cos θ. The half-vector h (half way between l and v) is slightly less expensive to calculate than the reflection vector r, and since h makes the angle θ/2 with the normal n, OpenGL uses h · n to calculate the specular drop-off term. Notice that as opposed to the diffuse reflectance, the cosine term has no physical or l h n v r /2 Figure3.8: Geometry of specular reflectance. Light coming from a light source l hits a surface with normal vector n, and most of the light is reflected to the direction r. The larger the angle θ between the viewer direction v and the mirror reflection direction r, the less light is reflected toward the viewer. Note that the angle between the half-vector h (splits evenly l and v) and the normal n is θ/2 . 68 LOW-LEVEL RENDERING CHAPTER 3 geometrical significance; it is simply a heuristic model to reduce reflectance to directions other than r. Since lighting is done in camera coordinates, and the camera looks toward the negative z axis, the viewing direction v can be approximated by (0, 0, 1). Finally, the cosine term is raised to a shininess power n spec . A large n spec , e.g., 128, attenuates the reflectance very quickly, and produces a small and sharp highlight, while a small n spec , e.g., 2, produces a fairly wide highlight. Putting it all together, we get the specular component from I = k s I s n · h n spec + . Emission The simplest material property is the emissive coefficient. Emission simply adds light to the material without requiring any external light sources. An emissive material does not illuminate any other surfaces. You could use this to model the surface of a light source such as a light bulb or a television screen, or to model fluorescent and phosphorescent materials. 3.2.4 LIGHTS OpenGL defines several simple light sources: the global ambient light, point lights, direc- tional lights, and spot lights. These lights work for sources that are very far away or have a small surface area. Accurate modeling of area light sources is much more expensive, but is sometimes done by discretizing the area source into a set of point light sources. Lights have RGBA colors where the channels can have “overbright” values exceeding 1.0. The global ambient light only has the ambient color component, but the other lights have a separate color for each of ambient, diffuse, and specular components. In this section we cover those light types, as well as the lig ht attenuation function. Global ambient light The simplest of the lights is the global ambient light. There can only be one of those and it is simply defined by an RGBA value, it is not associated with a position, and it affects all surfaces. Point lights Even a large area light source can be accurately modeled as a point light if the objects illuminated by the source are sufficiently far away from the source. A point light is located atagiven(x, y, z, 1) position, and emits light to every direction, as shown on the left in Figure 3.9. Like the rest of the light types, it contains a separate color for each of the ambient, diffuse, and specular components. If a light attenuation function is defined, it affects the intensity of the light reaching the surface. The light direction is calculated as the vector difference from the light position to the vertex being illuminated. SECTION 3.2 LIGHTING 69 u u 2u Figure 3.9: Various light sources: point light, directional light, spot light, and directional attenuation of spot lights. Directional lights A directional light can be thought of as a point light infinitely far away. For most practical purposes sunlight on earth can be modeled as a directional light. To define the direction, place the light source into direction (x, y, z, 0). The light will then shine toward the oppo- site direction, (−x, −y, −z, 0). Directional light is cheaper to evaluate than point light as the light direction is constant for all the vertices (see the second image in Figure 3.9), and the intensity of the light is never attenuated. Spot lights A spot light is a mixture between a point light and a directional light, and aims to model the behavior of a typical desk lamp. Like the point light, a spot light has a position, and its intensity can be attenuated as a function of distance. However, it also has a preferred direction toward which it shines brightest. No light escapes to directions farther than the cut-off angle θ from the preferred direction (third image in Figure 3.9), and the light that is not cut off is attenuated for directions other than the spot light direction using a similar exponentiated cosine function as already familiar from specular reflectance. That is, if l is the direction from the light to the surface, s is the spot light direction, and n spot is the spot direction fall-off exponent, the directional fall-off becomes spot = (l · s) n spot , unless the angle between l and s is greater than the cut-off angle, in which case spot = 0. The last image in Figure 3.9 illustrates the directional attenuation function, which cuts to zero for angles larger than θ, and gets more concentrated toward the spot light direction s as the exponent n spot grows. Light attenuation A faraway light source casts fewer photons of light to a given surface area than if the same light source were nearby. OpenGL models attenuation of lig ht due to distance w ith att = 1/(k c + k l d + k q d 2 ),wherek c stands for the constant coefficient, and k l and k q are the linear and quadr atic coefficients, while d is the distance from the light source to the surface. 70 LOW-LEVEL RENDERING CHAPTER 3 Real point light source energy is attenuated by the square of the distance from the light source. However, OpenGL does not typically represent the true light energy, but a com- pressed representation of light, and quadratic attenuation typically produces too harsh changes in illumination. Also, most indoor lights are area lights, not point lights, and have a more complicated and softer attenuation behavior that can be better matched by adding the linear and constant terms into the equation. The main outdoor light source, the sun, is so far away that all the visible points in practice get the same amount of light and therefore the relative light attenuation can be ignored. Purely quadr atic attenuation may make sense in a situation where the only light source really is a relatively weak point light source, such as a single candle, and a very dramatic attenuation is desired. 3.2.5 FULL LIGHTING EQUATION Combining Sections 3.2.1–3.2.4, the full OpenGL lighting equation can be formulated as Equation (3.3). The resulting light intensity (I) consists of the emissive term of the material (k e ), ambient term of the material (k a ) illuminated by the global ambient light (I a ), and then the contributions due to each active light source are summed up. For each light i, the ambient term (k a I ai ) is added, and the distance attenuation att i and spot light fall-off (spot i ) are combined with the diffuse (k d I di (n · l i ) + ) and specular ( k s I si f i (n · h) n spec + ) components. For a directional light att i = 1, and for directional and point lights spot i = 1. Finally, f i = 0 turns off the calculation of the specular term if the direction to the light is perpendicular to the surface normal (n · l i = 0), otherwise f i = 1. I = k e + k a I a + i k a I ai + att i spot i k d I di (n · l i ) + + k s I si f i (n · h) n spec + (3.3) It is also possible to compute double-sided lighting, in which case the lig hting equations are evaluated both for the outside normal n and the inside normal −n. This can be useful if you want to draw a thin object, such as a sheet of paper, without having to separately model both the front and the back side. On desktop OpenGL one can even give different materials to the different sides, but mobile APIs simplify this and use the same material on both sides. 3.3 CULLING AND CLIPPING Not all primitives end up being visible. For example, almost half of the triangles of typical scenes face away from the camera and may thus be discarded in the rendering pipeline. Others fall outside of the view fr ustum, or overlap it only partially. In this section we cover back-face culling and clipping of primitives. SECTION 3.3 CULLING AND CLIPPING 71 3.3.1 BACK-FACE CULLING With real solid objects it is impossible to directly see the back side of an object: by definition it faces away from the observer, and it is always occluded by another part of the same object. Since rendering back-facing triangles would not contribute to the final image, it makes sense to save time by not rendering them. On average half of the trian- gles are back-facing, and the time savings due to skipping them can be substantial. With back-face culling, the system may be able to avoid the cost of lighting the vertices, and in any case it avoids rasterizing the triangle. There are several ways to cull back-facing triangles. One possibility is to calculate the true normal vector to the triangle, and compare that to the direction to the camera. Another approach, often faster and more stable, is to project the vertices to the image plane, cal- culate the signed area of the projected triangle, and cull the triangle if the area turns out to be negative. One can e ven try back-projecting the camera into object space and doing the check there. In order to determine which is the front and which the back side of a triangle, a winding convention has to be used. By default, triangles are defined so that when viewed from outside, the vertices are given in a counterclockwise order. Similarly, by default, it is the back faces that are culled. However, the user may override both of these conventions, that is, explicitly set the front face to have clockwise or counterclockwise vertex ordering, or specify that the front face should be culled instead of the back face. One reason to do so is if the user first models one-half of a symmetrical object, and then obtains the other half by mirroring the first half. In such a case the user should toggle the winding direction for the second half to ensure correct culling and shading of the triang les on the mirrored half. Back-facing triangles are not culled by default in OpenGL, so the culling has to be explic- itly enabled. A reason for not using back-face culling would be if one would like to model thin objects with double-sided triangles. 3.3.2 CLIPPING AND VIEW-FRUSTUM CULLING The camera sees only a finite region of space—the view frustum. The sides of the view ing cone are defined by the extent of the window in the image plane through which the camera views the scene, and the cone is capped at the near and far distances, making it a trun- cated pyramid, a frustum. Primitives that fully fall outside of the frustum will not affect the scene and can therefore be ig n ored. Determining that complete objects lie outside of the viewing frustum and skipping them completely may be done sometimes with an easy check within the application. Some engines may also perform an automatic conservative viewport culling by calculating a bounding box of a vertex array and testing for an inter- section with the view frustum, and rejecting the whole array if the bounding box is fully outside the frustum. For example, most M3G implementations include this optimization (see Section 5.3). 72 LOW-LEVEL RENDERING CHAPTER 3 Clipping to the view frustum If a primitive intersects the view frustum it may need to be clipped. Here we describe clipping as specified in OpenGL. There are always six clipping planes, corresponding to the left, right, top, and bottom of theviewport, and the near (front) and far (back) clipping planes. As described before, in clip coordinates (after the multiplication by the projection matrix but before homogeneous division by w) the clipping planes correspond to one of the x, y,orz coordinates being either −w or w. When a triangle is clipped by a plane, there are several possible outcomes. If the triangle does not intersect the plane, the triangle is either completely accepted or completely rejected, depending on which side of the plane it is. If it does intersect the plane, the clipping may shave off two vertices and an edge, resulting in a smaller triangle, or shave off just one vertex and yield a smaller quadrangle. Thus each clip may grow the number of vertices by one, and clipping a triangle against six planes may lead up to a nine-vertex polygon (see Figure 3.10). Some systems may be able to just evaluate the parts of the primitives within the viewport without doing real clipping. Clipping lines and points is simpler. Clipping a line may shorten the line but the result is still a line. Clipping a point either erases or keeps the point, and the clipping is done based on the point center, even for wide points. Artifacts caused by clipping Real clipping may cause some artifacts. Ideally, if you render an image in four pieces, one quarter at a time, into smaller images, not forgetting to set up the projection matrix appropriately, you should get the same pixels as when you render the whole image in one go. However, if clipping the primitives introduces new vertices at image boundaries, and unless the colors and possibly other properties are interpolated using exactly the same algorithm as used in rasterization, some pixels at the screen boundaries will appear dif- ferent in the smaller images. The middle image in Figure 3.10 illustrates some problems when clipping wide lines and points. Clipping a wide line may omit a part of the line; in this case the areas that are gray Viewport Guard Band Figure 3.10: Left: Each clip plane may grow the number of vertices by one. Middle: For wide points and lines clipping may produce artifacts. Right: Guard band clipping combines clipping and scissoring. SECTION 3.4 RASTERIZATION 73 but inside the viewport. Since points are clipped based on their center, an animated wide point approaching the viewport boundary may suddenly vanish once the center exits the viewport, instead of gracefully sliding off the screen. Guard band clipping Some implementations may avoid some of these problems by implementing clipping implicitly as part of the rasterization, for example by using guard bands [BSD + 89], or by e valuating only those pixels of the primitives that fall on the screen using 2D homo- geneous coordinates [OG97]. The rightmost image in Figure 3.10 illustrates guard band clipping, which can both avoid some clipping artifacts and accelerate clipping. There the medium gray triangles can be trivially rejected as they are completely outside of the view- port. The light triangles are completely within the guard band, and they are simply ras- terized and the pixels outside of the viewport are ignored. Only the dark triangle which enters the viewport and exits the guard band needs to be clipped. Clipping to an arbitrary plane Finally, the user can define arbitrary clip planes (supported in OpenGL ES 1.1). Each plane is defined by the coefficients [abcd] T of the equation ax + by + cz + d = 0, defined in object coordinates. If you have a plane with a normal vector [N x N y N z ] T toward the half-space you want to keep, going through the point [p x p y p z ] T , the coefficients become ⎡ ⎢ ⎢ ⎢ ⎣ a b c d ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ N x N y N z −(N x p x + N y p y + N z p z ) ⎤ ⎥ ⎥ ⎥ ⎦ . (3.4) Since the user clip plane is defined in object coordinates, the system transforms the coef- ficients into eye coordinates by M −T abcd T , as also shown in Figure 3.1. User clip planes have been used for a variety of visual effects, such as reflections from water—here the algorithm is to mirror the world geometry to be below the water, and use the clip plane to only render the parts that are below the waterline. 3.4 RASTERIZATION Before rasterization, vertices are collected into primitives in a stage called primitive assem- bly. Rasterization then decides which pixels of the frame buffer the primitives cover, and which colors and depth values those pixels are assigned. This is done by interpolating var- ious values associated with the vertices, such as the colors due to illumination, the depth values, and the texture coordinates. . light source. However, OpenGL does not typically represent the true light energy, but a com- pressed representation of light, and quadratic attenuation typically produces too harsh changes in illumination image in one go. However, if clipping the primitives introduces new vertices at image boundaries, and unless the colors and possibly other properties are interpolated using exactly the same algorithm. clipped. Here we describe clipping as specified in OpenGL. There are always six clipping planes, corresponding to the left, right, top, and bottom of theviewport, and the near (front) and far (back)