x CONTENTS CHAPTER 11. EGL 241 11.1 API Overview 242 11.2 Configuration 244 11.3 Surfaces 248 11.4 Contexts 252 11.5 Extensions 253 11.6 Rendering into Textures 254 11.7 Writing High-Performance EGL Code 255 11.8 Mixing OpenGL ES and 2D Rendering 257 11.8.1 Method 1: Window Surface is in Control 257 11.8.2 Method 2: Pbuffer Surfaces and Bitmaps 258 11.8.3 Method 3: Pixmap Surfaces 258 11.9 Optimizing Power Usage 259 11.9.1 Power Management Implementations 259 11.9.2 Optimizing the Active Mode 261 11.9.3 Optimizing the Idle Mode 262 11.9.4 Measuring Power Usage 262 11.10 Example on EGL Configuration Selection 264 PART III M3G CHAPTER 12. INTRODUCING M3G 269 12.1 Overview 270 12.1.1 Mobile Java 270 12.1.2 Features and Structure 272 12.1.3 Hello, World 276 12.2 Design Principles and Conventions 277 12.2.1 High Abstraction Level 278 12.2.2 No Events or Callbacks 279 12.2.3 Robust Arithmetic 280 12.2.4 Consistent Methods 281 12.2.5 Parameter Passing 282 12.2.6 Numeric Values 283 12.2.7 Enumerations 284 12.2.8 Error Handling 284 12.3 M3G 1.1 285 12.3.1 Pure 3D Rendering 285 12.3.2 Rotation Interpolation 285 12.3.3 PNG and JPEG Loading 286 12.3.4 New Getters 287 12.3.5 Other Changes 288 CONTENTS xi CHAPTER 13. BASIC M3G CONCEPTS 289 13.1 Graphics3D 290 13.1.1 Render Targets 290 13.1.2 Viewport 293 13.1.3 Rendering 294 13.1.4 Static Properties 296 13.2 Image2D 297 13.3 Matrices and Transformations 300 13.3.1 Transform 300 13.3.2 Transformable 303 13.4 Object3D 306 13.4.1 Animating 306 13.4.2 Iterating and Cloning 306 13.4.3 Tags and Annotations 308 13.5 Importing Content 311 13.5.1 Loader 311 13.5.2 The File Format 313 CHAPTER 14. LOW-LEVEL MODELING IN M3G 319 14.1 Building meshes 319 14.1.1 VertexArray 319 14.1.2 VertexBuffer 320 14.1.3 IndexBuffer and Rendering Primitives 323 14.1.4 Example 325 14.2 Adding Color and Light: Appearance 326 14.2.1 PolygonMode 327 14.2.2 Material 328 14.2.3 Texture2D 329 14.2.4 Fog 332 14.2.5 CompositingMode 333 14.3 Lights and Camera 337 14.3.1 Camera 337 14.3.2 Light 339 14.4 2D Primitives 343 14.4.1 Background 343 14.4.2 Sprite3D 346 CHAPTER 15. THE M3G SCENE GRAPH 349 15.1 Scene Graph Basics: Node, Group, and World 349 15.2 Mesh Objects 351 15.3 Transforming Objects 354 15.3.1 Camera, Light, and Viewing Transformations 355 15.3.2 Node Alignment 356 xii CONTENTS 15.4 Layering and Multi-Pass Effects 360 15.5 Picking 362 15.6 Optimizing Performance 364 15.6.1 Visibility Optimization 365 15.6.2 Scope Masks 365 CHAPTER 16. ANIMATION IN M3G 367 16.1 Keyframe Animation: KeyframeSequence 367 16.2 Animation Targets: AnimationTrack 372 16.3 Timing and Speed: AnimationController 374 16.4 Animation Execution 377 16.5 Advanced Animation 378 16.5.1 Deformable Meshes 378 16.5.2 Animation Blending 385 16.5.3 Creating Discontinuities 387 16.5.4 Dynamic Animation 388 PART IV APPENDIX A FIXED-POINT MATHEMATICS 393 A.1 Fixed-Point Methods in C 395 A.1.1 Basic Operations 395 A.1.2 Shared Exponents 397 A.1.3 Trigonometric Operations 399 A.2 Fixed-Point Methods in Assembly Language 400 A.3 Fixed-Point Methods in Java 405 B JAVA PERFORMANCE TUNING 407 B.1 Virtual Machines 408 B.2 Bytecode Optimization 409 B.3 Garbage Collection 410 B.4 Memory Accesses 411 B.5 Method Calls 413 C GLOSSARY 415 Bibliography 419 Index 425 Preface The mobile phone is by far the most widely available device with rendering capabilities in the world, and it is very likely that this will continue to be the case. However, this ubiquitous tool may not continue to be centered around its phone function for much longer, as it evolves more and more into a multifaceted device, which you might want to call a mobile Gizmo (see Bruce Sterling’s keynote at SIGGRAPH 2004). Inevitably, graphics is becoming a core part of such a Gizmo. The pivotal role of graphics in the future of the Gizmo, and the fact that these devices are spread out (quite evenly, compared to other rendering platforms) over the entire globe, makes the mobile phone an incredibly exciting platform on which to develop graphics. Over the past few years, I have done quite a lot of research on mobile graphics and energy- efficient graphics hardware targeting these platforms. I believe that the authors of this book and I share the vision of omnipresent three-dimensional graphics on all mobile devices. Compared to the contributions made through my research, the authors provide within these pages more than a small stepping stone. In my opinion, this book is an escalator, which takes the field to new levels. This is especially true because their text ensures that the topic is easily accessible to everyone with some background in computer science. Further, this book is unique in that it provides a single resource covering both OpenGL ES and M3G. These open APIs have been specifically developed for mobile devices, and many in the community, including myself, expect that these will be the most widely utilized APIs for the foreseeable future. The foundations of this book are clear, and the authors are extremely knowledgeable about the subject, partly due to the enormous amounts of time and effort they have invested in standardization organizations, such as the Khronos Group and Java commu- nity, which are committed to making both the OpenGL ES and M3G standards faster, more robust, and easier to use. Undoubtedly, the authors of this book will continue to help develop even better versions of these APIs as the field progresses. I am certain that the future of mobile graphics will be more than bright, and with this book in your hand, you, the reader, will be able to create vibrant applications with three-dimensional xiii xiv PREFACE graphics on mobile devices. Hopefully, your mobile graphics applications will be like nothing the world has ever seen before. Please, do surprise me. Tomas Akenine-M ¨ oller Lund University Sweden About the Authors Kari Pulli contributed to both OpenGL ES and M3G from the very beginning, and was among the most active technical contributors to each API. Kari, originally Principal Scientist and later Research Fellow, headed Nokia’s graphics research, standardization, and technology strategy and implementation, and was Nokia’s contact person for both standards. Tomi Aarnio, Senior Research Engineer, mostly concentrated on the M3G standard. He was the specification editor of all versions of M3G, and headed the implementation project of both its Reference Implementation and the engine that is shipping on Nokia phones. Ville Miettinen was active and influential on the definition of the first versions of both of these graphics standards. At the time he acted as the CTO of Hybrid Graphics, and later as a specialist of next-generation mobile graphics platforms at NVIDIA. Nowadays, he is a private consultant. Kimmo Roimela, Senior Research Engineer at Nokia, also concentrated on the M3G stan- dardization and implementation. He was the main architect of the M3G’s animation model and an associate editor of the M3G specification. He was also the lead programmer of the Nokia M3G implementation. Jani Vaarala, Graphics Architect at Nokia, was very active in the definition of OpenGL ES standard. He also headed the team that implemented and integrated Nokia’s first OpenGL ES and EGL solution. xv Acknowledgments The creation and adoption of OpenGL ES and M3G was possible because of the hard work of many people and companies. When we use the term “we” in this book, we mean not just the authors but everybody w ho participated in the OpenGL ES working group or M3G expert group, and in some cases in both of them. Below we m ention some of the most active contributors, the full list can be found from the API specifications. Neil Trevett initiated the creation of OpenGL ES and chaired the OpenGL ES working group from the beginning until OpenGL ES 2.0. Tom O lson was an active contributor from the beginning and became the next chair of the OpenGL ES working group. David Blythe was the original specification editor for OpenGL ES. He also adapted the OpenGL sample implementation for OpenGL ES. Aaftab (Affie) Munshi became the editor after David left the Khronos Group to become the head architect of Direct 3D at Microsoft. Jon Leech, the OpenGL ARB secretary and EGL specification editor contributed a lot to all aspects of OpenGL ES. He is also the editor of the OpenGL ES 1.1 normative specification. Tom McReynolds, Robert Simpson, Petri Kero, Gary King, Graham Connor, and Remi Arnaud were important contributors for OpenGL ES, and Claude Knaus created the first OpenGL ES conformance tests. Jyri Huopaniemi chaired the first M3G (JSR 184) expert group. Sean Ellis was one of the most active contributors to the M3G specification, and an associate specification editor, authoring the M3G file format. Mark Patel, Mark Tarlton, Doug Twilleager, Paul Beardow, Michael Steliaros, and Chris Grimm were among the most active members of the M3G expert group. Mark Callow, Jacob Str ¨ om, and Ed Plowman have been very active contributors to both OpenGL ES and M3G APIs. We would like to thank the following people who read at least parts of the book and provided many comments, making the book better than it would have otherwise been: Timo Aila, Tomas Akenine-M ¨ oller, Oliver Bimber, Suresh Chitturi, Sean Ellis, Michael Frydrych, Jiang Gao, Radek Grzeszczuk, Timo Haanp ¨ a ¨ a, Kari Kangas, Laszlo Kishonti, Chris Knox, Sami Ky ¨ ostil ¨ a, Jon Leech, Mika Pesonen, Vidya Setlur, Robert Simpson, Dominic Symes, Yaki Tebeka, Juha Uola, Gareth Vaughan, and Yingen Xiong. xvi 1 CHAPTER INTRODUCTION Mobile phones are the new vehicle for bringing interactive graphics technologies to consumers. Graphics that in the 1980s was only seen in industrial flight simulators and at the turn of the millennium in desktop PCs and game consoles is now in the hands of billions of people. This book is about the technology underpinnings of mobile three- dimensional graphics, the newest and most rapidly advancing area of computer graphics. Computer graphics has been around since the 1960s. Its application areas range from user interfaces to video gaming, scientific visualization, special effects in movies, and even full-length animated films. In the field of computer graphics, it is the subset of three- dimensional (3D) graphics that produces the most life-like v isuals, the “wow” effects, and the eye-candy. Since the late 1990s, almost all computer games, and more recently even operating systems such as OS X and Windows Vista, have come to rely heavily on real-time 3D graphics. This has created an enormous drive for graphics hardware devel- opment. Dedicated graphics hardware is ubiquitous on desktop and laptop computers, and is rapidly becoming common on high-end mobile phones. Low-cost software-based implementations bring 3D graphics to mass-market consumer phones as well. Computer graphics is nowadays an integral part of the phone user experience: graphics is the face of the device. Mobile phones, also know n as cellular or cell phones, have recently become universal communication and computation devices. In countries such as the UK there are more mobile phone subscriptions than there are people. At the same time, the capabilities of the devices are improving. According to Moore’s law [Moo65], the transistor density on 1 2 INTRODUCTION CHAPTER 1 integrated circuits roughly doubles every one or two years; today’s high-end mobile phone has more computational power than a late 1990s home PC. The display resolutions of mobiles will soon reach and surpass that of conventional broadcast television, with much better color fidelity. Together, these advances have resulted in a truly mobile computer. As a side effect, real-time, interactive 3D graphics has become feasible and increasingly desirable for the masses. 1.1 ABOUT THIS BOOK This book is about writing real-time 3D graphics applications for mobile devices. We assume the reader has some background in mathematics, programming, and computer graphics, but not necessarily in mobile devices. The 3D graphics capabilities of mobile devices are exposed through two standardized application programming interfaces (APIs): OpenGL ES, typically accessed through C or C++, and M3G, for mobile Java. We introduce the latter standard in terms of the former. As OpenGL ES is utilized as the fundamental building block in many real-world M3G implementations, expressing this relationship explicitly is highly useful for describing the inner workings of M3G. The two APIs are equally suited to programming embedded devices other than mobile phones, from car navigation systems to display screens of microwave ovens. However, most of such platforms are closed—parties other than the de vice manufacturer cannot develop and install new applications on them. By contrast, most mobile phones are open: third parties such as professional software developers, students, and individual enthusi- asts can program, install, and distribute their own applications. Having a programmable mobile phone at hand to try out the techniques described in this book is actually a great idea. However, the details of mobile application development vary considerably across platforms, so we defer those details to each platform’s developer documentation. This book consists of three parts and several appendices. Part I gives an introduction to the 3D graphics concepts that are needed to understand OpenGL ES and M3G, which are then covered in Parts II andIII, respectively. The use of each API is demonstrated with hands-on code examples. The appendices provide additional information and optimization tips for both C/C++ and Java developers as well as a glossary of acronyms and terms used in this book. There is also a companion web site, www.graphicsformasses.com, hosting code examples, errata, and links to other online resources. A more comprehensive treatment of 3D graphics, such as Real-Time Rendering by Tomas Akenine-M ¨ oller and Eric Haines [AMH02], is recommended for readers new to computer graphics. The “OpenGL Red Book” [SWN05] is a traditional OpenGL beginner’s guide, while a book by McReynolds and Blythe [MB05] collects more advanced OpenGL tips in one place. Those unfamiliar with programming in mobile Java may find Beginning J2ME: From Novice to Professional by Sing Li and Jonathan Knudsen [LK05] useful. SECTION 1.2 GRAPHICS ON HANDHELD DEVICES 3 1.1.1 TYPOGRAPHIC CONVENTIONS Alongside the basic text, there are specific tips for achieving good performance and avoiding common pitfalls. These hints are called performance tips and pitfalls, respectively. An example of each follows: Performance tip: Enabling the optimization flag in the compiler makes your appli- cation run faster. Pitfall: Premature optimization is the root of all evil. Code snippets and class, token, and function names are shown in typewriter typeface like this: glPointSize( 32 ); glEnable( GL_POINT_SPRITE_OES ); glTexEnvi( GL_POINT_SPRITE_OES, GL_COORD_REPLACE_OES, GL_TRUE ); glDrawArrays( GL_POINTS, 0, 1 ); When API functions are introduced, they are marked like this: void function(int parameter). Any later references to the function or parameter in the text are also similarly emphasized. 1.2 GRAPHICS ON HANDHELD DEVICES The very first mobile phones were heavy bricks with separate handsets; a few examples can be seen in Figure 1.1. They were designed to be lugged around rather than carried in Figure 1.1: The evolution of mobile phones from the early car phones on the left to the multimedia computer on the right spans roughly two decades. From the left: Mobira Talkman, Nokia R72, Mobira Cityman, Nokia 3410 (the first GSM phone with a 3D graphics engine), Nokia 6630 (the first phone to support both OpenGL ES and M3G), and Nokia N93 (the first phone with hardware acceleration for both APIs). Images Copyright c 2007 Nokia Corporation. . definition of OpenGL ES standard. He also headed the team that implemented and integrated Nokia’s first OpenGL ES and EGL solution. xv Acknowledgments The creation and adoption of OpenGL ES and M3G. programming, and computer graphics, but not necessarily in mobile devices. The 3D graphics capabilities of mobile devices are exposed through two standardized application programming interfaces (APIs): OpenGL. three parts and several appendices. Part I gives an introduction to the 3D graphics concepts that are needed to understand OpenGL ES and M3G, which are then covered in Parts II andIII, respectively.