3D Graphics with OpenGL ES and M3G- P32 pdf

294 BASIC M3G CONCEPTS CHAPTER 13 Performance tip: On hardware-accelerated devices, it is a good idea to clear the color buffer and depth buffer completely and in one go, even if the whole screen will be redrawn. Various hardware optimizations can only be enabled when starting from a clean table, and some devices can clear the screen by just flipping a bit. Even with soft- ware implementations, clearing everything is typically faster than clearing a slightly smaller viewport. Note that the viewport need not lie inside the render target; it can even be larger than the render target. The view that you render is automatically scaled to fit the viewport. For example, if you have a viewport of 1024 by 1024 pixels on a QVGA screen, you will only see about 7% of the rendered image (the nonvisible parts are not really rendered, of course, so there is no performance penalty); see the code example in Section 13.1.4. The maximum size allowed for the viewport does not depend on the type of rendering target, but only on the implementation. All implementations are required to support viewports up to 256 by 256 pixels, but in practice the upper bound is 1024 by 1024 or higher. The exact limit can be quer ied from Graphics3D.getProperties. Pitfall: Contrary to OpenGL ES, there is no separate function for setting thescissor rect- angle (see Section 3.5). Instead, the scissor rectangle is implicitly defined as the inter- section of the viewport and the Graphics clipping rectangle. A concept closely related to the viewport is the depth range,setby setDepthRange(float near, float far), where near and far are in the range [0, 1]. Similar to the viewport, the depth range also defines a mapping from nor malized device coordinates to screen coordinates, only this time the screen coordinates are depth values that lie in the [0, 1] range. Section 2.6 gives insight on the depth range and how it can be used to make better use of the depth buffer resolution or to speed up your application. 13.1.3 RENDERING As we move toward more ambitious goals than merely clearing the screen, the next step is to render some 3D content. For simplicity, let us just assume that we have mag- ically come into possession of a complete 3D scene that is all set up for rendering. In case of M3G, this means that we have a World object, which is the root node of the scene graph and includes by reference all the cameras, lights, and polygon meshes that we need. To render a full-screen view of the world, all we need to do within the bindTarget-releaseTarget block is this: g3d.render(myWorld); This takes care of clearing the depth buffer and color buffer, setting up the camera and lights, and finally rendering everything that there is to render. This is called retained-mode rendering, because all the information necessary for rendering is retained by the World SECTION 13.1 GRAPHICS3D 295 and its descendants in the scene graph. In the immediate mode, you would first clear the screen, then set up the camera and lights, and finally draw your meshes one by one inaloop. The retained mode and immediate mode are designed so that you can easily mix and match them in the same application. Although the retained mode has less overhead on the Java side, and is generally recommended, it may sometimes be more convenient to handle overlays, particle effects, or the player character, for instance, separately from the rest of the scene. To ease the transition from retained mode to immediate mode at the end of the frame, the camera and lights of the World are automatically set up as the current camera and lights in Graphics3D, overwriting the previous settings. The projection matrix (see Chapter 2) is defined in a Camera object,whichinturnis attached to Graphics3D using setCamera (Camera camera, Transform trans- form). The latter parameter specifies the t ransformation from camera space, also known as eye space or view space, into world space. The Camera class is described in detail in Section 14.3.1. For now, it suffices to say that it allows you to set an arbitrary 4 × 4 matrix, but also provides convenient methods for defining the typical perspective and parallel projections. The following example defines a perspective projection with a 60 ◦ vertical field of view and the same aspect ratio as the Canvas that we are rendering to: Camera camera = new Camera(); float width = myCanvas.getWidth(); float height = myCanvas.getHeight(); camera.setPerspective(60.0f, width/height, 10.0f, 500.0f); g3d.setCamera(camera, null); Note that we call setCamera with the transform parameter set to null.Asageneral principle in M3G, a null transformation is treated as identity, which in this case implies that the camer a is sitting at the world-space origin, looking toward the negative Z axis with Y pointing up. Light sources are set up similarly to the camera, using addLight(Light light, Trans- form transform). The transform parameter again specifies the transformation from local coordinates to world space. Lighting is discussed in Section 14.3.2, but for the sake of illustration, let us set up a single directional white light that shines in the direction at which our camera is pointing: Light light = new Light(); g3d.addLight(light, null); Now that the camera and lights are all set up, we can proceed with rendering. There are three different render methods in immediate mode, one having a higher level of abs- traction than the other two. The high-level method render(Node node, Transform transform) draws an individual object or scene graph branch. You can go as far as ren- dering an entire World with it, as long as the camer a and lights are properly set up in 296 BASIC M3G CONCEPTS CHAPTER 13 Graphics3D. For instance, viewing myWorld with the camera that we just placed at the world space orig in is as simple as this: g3d.render(myWorld, null); Of course, the typical way of using this method is to draw individual meshes rather than entire scenes, but that decision is up to you. The low-level render methods, on the other hand, are restricted to drawing a single triangle mesh. The mesh is defined by a vertex buffer, an index buffer, and an appearance. As with the camera and lights, a transformation from model space to world space must be given as the final parameter: g3d.render(myVertices, myIndices, myAppearance, myTransform); The other render variant is similar, but takes in an integer scope mask as an additional parameter. The scope mask is bitwise-ANDed with the corresponding mask of the current camera, and the mesh is rendered if and only if the result is non-zero. The same applies for lights. The scope mask is discussed further in Chapter 15, as it is more useful in retained mode than in immediate mode. 13.1.4 STATIC PROPERTIES We mentioned in Section 12.2 that there is a static getter for retrieving implementation- specific information, such as whether antialiasing is supported. This special getter is defined in Graphics3D, and is called getProperties. It returns a java.util. Hashtable that contains Integer and Boolean values keyed by Strings. The static properties, along with some helpful notes, are listed in Table 13.1. To illustrate the use of static properties, let us create a viewpor t that is as large as the implementation can support, and use it to zoom in on a high-resolution rendering of myWorld: Hashtable properties = Graphics3D.getProperties(); maxViewport = ((Integer)properties.get("maxViewportDimension")). intValue(); g3d.bindTarget(graphics, true, hints); int topLeftX = — (maxViewport — graphics.getClipWidth())/2; int topLeftY = — (maxViewport — graphics.getClipHeight())/2; g3d.setViewport(topLeftX, topLeftY, maxViewport, maxViewport); g3d.render(myWorld); g3d.releaseTarget(); We first query for maxViewportDimension from the Hashtable.Thevalueis returnedasajava.lang.Object, which we need to cast into an Integer and then convert into a primitive int before we can use it in computations. Later on, at the paint method, we set the viewport to its maximum size, so that our Canvas lies at its center. Assuming a QVGA screen and a 1024-pixel-square viewport, we would have a zoom factor of about 14. The zoomed-in view can be easily panned by adjusting the top-left X and Y. SECTION 13.2 IMAGE2D 297 Table 13.1: The system properties contained in the Hashtable returned by Graphics3D.getProperties. There may be other properties, as well, but they are not standardized. Key (String) Value Notes supportAntialiasing Boolean true on some hardware-accelerated devices supportTrueColor Boolean false on all devices that we know of supportDithering Boolean false on all devices that we know of supportMipmapping Boolean false on surprisingly many devices supportPerspectiveCorrection Boolean true on all devices, but quality varies supportLocalCameraLighting Boolean false on almost all devices maxLights Integer ≥ 8 typically 8 maxViewportWidth Integer ≥ 256 typically 256 or 1024; M3G 1.1 only maxViewportHeight Integer ≥ 256 typically 256 or 1024; M3G 1.1 only maxViewportDimension Integer ≥ 256 typically 256 or 1024 maxTextureDimension Integer ≥ 256 typically 256 or 1024 maxSpriteCropDimension Integer ≥ 256 typically 256 or 1024 maxTransformsPerVertex Integer ≥ 2 typically 2, 3, or 4 numTextureUnits Integer ≥ 1 typically 2 13.2 Image2D There are a few cases where M3G deals with 2D image data. Texturing, sprites, and background images need images as sources, and rendering to any of them is also supported. Image2D, as the name suggests, stores a 2D array of image data. It is similar in many respects to the javax.microedition.lcdui.Image class, but the important dif- ference is that Image2D objects are fully managed by M3G. This lets M3G implementa- tions achieve better performance, as there is no need to synchronize with the 2D drawing functions in MIDP. Similarly to the MIDP Image,anImage2D object can be either mutable or immutable. To create an immutable image, you must supply the image data in the constructor: Image2D(int format, int width, int height, byte[] image) The format parameter specifies the type of the image data: it can be one of ALPHA, LUMINANCE, LUMINANCE_ALPHA, RGB, and RGBA .Thewidth and height parame- ters determine the size of the image, and the image array contains data for a total of width × height pixels. The layout of each pixel is determined by format: each image com- ponent takes one byte and the components are interleaved. For example, the data for 298 BASIC M3G CONCEPTS CHAPTER 13 a LUMINANCE_ALPHA image would consist of two bytes giving the luminance and alpha of the first pixel, followed by two bytes giving the luminance and alpha of the second pixel, and so on. The pixels are ordered top-down and left to right, i.e., the first w idth pixels provide the topmost row of the image starting from the left. Upon calling the constructor, the data is copied into internal memory allocated by M3G, allowing you to discard or reuse the source array. Note that while the image is input upside-down compared to OpenGL ES (Section 9.2.2), the t texture coordinate is similarly reversed, so the net effect is that you can use the same texture images and coordinates on both OpenGL ES and M3G. Unfortunately, there is no support in M3G for packed image formats, such as RGB565. This is partially because OpenGL ES does not give any guarantees regarding the internal color depth of a texture image, but also because the image formats were intentionally kept few and simple. In retrospect, being able to input the image data in a packed format would have been useful in its own right, regardless of what happens when the image is sent to OpenGL ES. As a form of image compression, you can also create a paletted image: Image2D(int format, int width, int height, byte[] image, byte[] palette) Here, the only difference is that the image arr ay contains one-byte indices into the palette array, which stores up to 256 color values. The layout of the color values is again as indi- cated by format. There is no guarantee that the implementation will internally maintain the image in the paletted format, though. Pitfall: The amount of memory that an Image2D consumes is hard to predict. Depending on the device, non-palletized RGB and RGBA images may be stored at 16 or 32 bits per pixel, while palletized images are sometimes expanded from 8 bpp to 16 or 32 bpp. Some implementations always generate the mipmap pyramid, consum- ing 33% extra memory. Some devices need to store two copies of each image: one in the GL driver, the other on the M3G side. Finally, all or part of this memory may be allocated from somewhere other than the Java heap. This means that you can run out of memory even if the Java heap has plenty of free space! You can try to detect this case by using smaller images. As for remedies, specific texture formats may be more space-efficient than others, but you should refer to the developer pages of the device manufacturers for details. A third constructor lets you copy the data from a MIDP Image: Image2D(int format, java.lang.Object image) Note that the destination format is explicitly specified. The source format is either RGB or RGBA, for mutable and immutable MIDP images, respectively. Upon copying the data, M3G automatically converts it from the source format into the destination format. SECTION 13.2 IMAGE2D 299 As a general rule, the conversion happens by copying the respective components of the source image and setting any missing components to 1.0 (or 0xFF for 8-bit colors). A cou- ple of special cases deserve to be mentioned. When converting an RGB or RGBA source image into LUMINANCE or LUMINANCE_ALPHA, the luminance channel is obtained by converting the RGB values into grayscale. A similar conversion is done when convert- ing an RGB image into ALPHA. This lets you read an alpha mask from a regular PNG or JPEG image through Image.createImage, or create one with the 2D drawing functions of MIDP, for example. Often the most convenient way to create an Image2D is to load it from a file. You can do that with the Loader, as discussed in Section 13.5. All implementations are required to support the M3G and PNG formats, but JPEG is often supported as well. 4 Loading an image file yields a new Image2D whose format matches that of the image stored in the file. JPEG can do both color and grayscale, yielding the internal formats RGB and LUMINANCE, respectively, but has no concept of transparency or alpha. PNG supports all of the Image2D formats except for ALPHA. It has a palletized format, too, but unfor- tunately the on-device PNG loaders tend to expand such data into raw RGB or RGBA before it ever reaches the Image2D. M3G files obviously support all the available for- mats, including those with a palette. Pitfall: The various forms of transparency supported by PNG are hard to get right. For example, the M3G loader in some early Nokia models (e.g., the 6630), does not support any form of PNG transparency, whereas some later models (e.g., the 6680) support the alpha channel but not color-keying. Possible workarounds include using Image.createImage or switching from PNG files to M3G files. These issues have been resolved in M3G 1.1; see Section 12.3. Finally, you can create a mutable Image2D: Image2D(int format, int width, int height) The image is initialized to opaque white by default. It can be subsequently modified by using set(int x, int y, int width, int height, byte[] pixels). This method copies a rectangle of width by height pixels into the image, starting at the pixel at (x, y) and proceeding to the right and down. The origin for the Image2D is in its top left corner. A mutable Image2D can also be bound to Graphics3D as a rendering target. The image can still be used like an immutable Image2D. This lets you, for example, render dynamic reflections or create feedback effects. 4 JPEG support is in fact required by the Mobile Service Architecture (MSA) specification, also known as JSR 248. MSA is an umbrella JSR that aims to unify the Java ME platform. 300 BASIC M3G CONCEPTS CHAPTER 13 Table 13.2: The available Image2D formats and their capabilities. The shaded cells show the capabilities of most devices, as these cases are not dictated by the specification. Mipmapping is entirely optional, and palletized images may be silently expanded into the corresponding raw formats, typically RGB. Most devices support mipmapping and palletized images otherwise, but will not generate mipmaps for palletized textures, nor load a palletized PNG without expanding it. There are some devices that can do better, though. Finally, note that JPEG does not support the alpha channel, and that PNG does not support images with only an alpha channel. Load from M3G Load from PNG Load from JPEG Copy from Image Render Target Back- ground Mutable Texture Mipmap Sprite ALPHA ✓✖✓ ✖✓✓✖✖✓ LUMINANCE ✓✖✓ ✖✓✓✓✓✓ LUM_ALPHA ✓✖✓ ✓✖✓✓✓✖✓ RGB ✓✓✓ ✓✓✓✓✓✓✓ RGBA ✓✓✓ ✓✓✓✓✓✖✓ Palette ✖✖ ✖✓✓✖✖✖✓✖✖ ✖ ✖ ✓ ✓ ✓ ✓ ✓ Performance tip: Beware that updating an Image2D, whether done by rendering or through the setter, can be a very costly operation. For example, the internal format and layout of the image may not be the same as in the set method, requiring heavy conversion and pixel reordering. If your frame rate or memory usage on a particular device is not what you would expect, try using immutable images only. While Image2D is a general-purpose class as such, there are various restrictions on what kind of images can be used for a specific purpose. For example, textures must have power- of-two dimensions, and render targets can only be in RGB or RGBA formats. Table 13.2 summarizes the capabilities and restrictions of the different Image2D formats. 13.3 MATRICES AND TRANSFORMATIONS One of the most frequently asked questions about M3G is the difference between Transform and Transformable. The short answer is that Transform is a sim- ple container for a 4 × 4 matrix with no inherent meaning, essentially a float array wrapped into an object, whereas Transformable stores such a matrix in a compo- nentized, animatable form, and for a particular purpose: constructing the modelview matrix or the texture matrix. The rest of this section provides the long answer. 13.3.1 Transform Transform stores an arbitrary 4 × 4 matrix and defines a set of basic utility functions for operating on such matrices. You can initialize a Transform toidentity,copyitin SECTION 13.3 MATRICES AND TRANSFORMATIONS 301 from another Transform,orcopyitfromafloat[] in row-major order (note that this is different from OpenGL ES, which uses the unintuitive column-major ordering). setIdentity resets a Transform back to its default state, facilitating object reuse. Creating a matrix To give an example, the following code fragment creates a matrix with a uniform scaling component [ 222 ] and a translation component [ 345 ] . In other words, a vector multiplied by this matrix is first scaled by a factor of two, then moved by three units along the x axis, four units along y, and five units along z: Transform myTransform = new Transform(); myTransform.set(new float[] { 2f, 0f, 0f, 3f, 0f, 2f, 0f, 4f, 0f, 0f, 2f, 5f, 0f, 0f, 0f, 1f }); Matrix operations Once you have created a Transform, you can start applying some basic arithmetic functions to it: You cantranspose the matrix (M  = M T ), invert it (M  = M −1 ), or multiply it with another matrix (M  = MA). Note that each of these operations overwrites the pre-existing value of the Transform with the result (M  ). The matrix multiplication functions come in several flavors: void postMultiply(Transform transform) void postScale(float sx, float sy, float sz) void postTranslate(float tx, float ty, float tz) void postRotate(float angle, float ax, float ay, float az) void postRotateQuat(float qx, float qy, float qz, float qw) The post prefix indicates that the matrix is multiplied from the right by the given matrix (e.g., M  = MA); pre would mean multiplying from the left (e.g., M  = AM), but there are no such methods in Transform. Going through the list of methods above, the first three probably need no deeper explanation. The rotation method comes in two varieties: postRotateQuat uses a quaternion to represent the rotation (see Section 2.3.1), whereas postRotate uses the axis-angle format: looking along the positive rotation axis  ax ay az  , the rotation is angle degrees clockwise. To make things more concrete, let us use postScale and postTranslate to con- struct the same matrix that we typed in manually in the previous example: Transform myTransform = new Transform(); myTransform.postTranslate(3f, 4f, 5f); myTransform.postScale(2f, 2f, 2f); 302 BASIC M3G CONCEPTS CHAPTER 13 Transforming vertices As in OpenGL, you should think that the matrix operations apply to vertices in the reverse order that they are written. If you apply the transformation TS toavertexv, the vertex is first scaled and then translated: T (Sv). Let us write out the matrices and confirm that the above code fragment does indeed yield the correct result: M  = ITS= ⎡ ⎢ ⎢ ⎢ ⎣ 1000 0100 0010 0001 ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 1003 0104 0015 0001 ⎤ ⎥ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎢ ⎣ 2000 0200 0020 0001 ⎤ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎢ ⎣ 2003 0204 0025 0001 ⎤ ⎥ ⎥ ⎥ ⎦ One of the most obvious things to do with a transformation matrix is to transform an array of vectors with it. The Transform class defines two convenience methods for this purpose. The first, transform(float[] vectors) multiplies each 4-element vector in the vectors array by this matrix and overwrites the original vectors with the results (v  = Mv,wherev is a column vector). The other transform variant is a bit more complicated: void transform(VertexArray in, float[] out, boolean w) Here,wetakein2Dor3DvectorsinaVertexArray, set the fourth component to zero or one depending on the w parameter, and write the transformed 4-element vectors to the out array. The input array remains unmodified. The transform methods are provided mostly for convenience, as they play no role in rendering or any other function of the API. Nonetheless, if you have a large number of vec- tors that you need to multiply with a matrix for whatever purpose, these built-in methods are likely to perform better than doing the same thing in Java code. The VertexArray variant also serves a more peculiar purpose: it is the only way to read back vertices from a VertexArray on many devices, as the necessary VertexArray.get methods were only added in M3G 1.1. Other use cases Now that you know how to set up Transform objects and use them to transform ver- tices, let us look at what else you can use them for. First of all, in Graphics3D you need them to specify the local-to-world transformations of the immediate-mode camera, lights, and meshes. In both immediate mode and retained mode, you need a Transform to set up an oblique or otherwise special projection in Camera, or any kind of projec- tion for texture coordinates in Texture2D. Finally, you can (but do not have to) use a Transform in the local-to-parent transformation of a Node. Each of these cases will come up later on in this book. SECTION 13.3 MATRICES AND TRANSFORMATIONS 303 13.3.2 Transformable Transformable is an abstract base class for the scene graph objects Node and Texture2D.Conceptually,itisa4 × 4 matrix representing a node transformation or a texture coordinate transformation. The matrix is made up of four components that can be manipulated separately: translation T, orientation R, scale S,andageneric4 × 4 mat rix M. During rendering, and otherwise when necessary, M3G multiplies the components together to yield the composite transformation: C = TRSM (13.1) A homogeneous vector p =  xyzw  T , representing a vertex coordinate or texture coordinate, is then transformed into p  =  x  y  z  w   T by: p  = Cp (13.2) The components are kept separate so that they can be controlled and animated indepen- dent of each other and independent of their previous values. For example, it makes no difference whether you first adjust S and then T, or vice versa; the only thing that matters is what values the components have when C needs to be recomputed. Contrast this with the corresponding operations in Transform, which are in fact matrix multiplications and thus very much order-dependent. Note that for node transformations, the bottom row of the M component is restricted to [ 0001 ] —in other words, projections are not allowed in the scene graph. Texture matrices do not have this limitation, so projective texture mapping is fully supported (see Section 3.4.3). Methods The following four methods in Transformable allow you to set the transformation components: void setTranslation(float tx, float ty, float tz) void setOrientation(float angle, float ax, float ay, float az) void setScale(float sx, float sy, float sz) void setTransform(Transform transform) The complementary methods, translate, preRotate, postRotate, and scale, each modify the current value of the respective component by applying an additional translation, rotation, or scaling. The user-provided rotation can be applied to the left . cases where M3G deals with 2D image data. Texturing, sprites, and background images need images as sources, and rendering to any of them is also supported. Image2D, as the name suggests, stores. Graphics3 D, and is called getProperties. It returns a java.util. Hashtable that contains Integer and Boolean values keyed by Strings. The static properties, along with some helpful notes, are. compared to OpenGL ES (Section 9.2.2), the t texture coordinate is similarly reversed, so the net effect is that you can use the same texture images and coordinates on both OpenGL ES and M3G. Unfortunately,