Learn ARKit, Apple’s New Augmented Reality Development Framework With ARKit, you can create immersive, engaging experiences by mixing virtual 2D and 3D content with the live camera feed of the world around you. If getting started with this new framework sounds difficult and timeconsuming, don’t worry, we’ve got you covered. ARKit by Tutorials helps you learn ARKit the quick and easy way: by following fun and easytoread tutorials. Who This Book Is For This book is for intermediate to advanced developers who want to create ARKit apps and learn about the intricacies of developing for Augmented Reality on iOS. Topics Covered in ARKit by Tutorials Introduction to ARKit: Get a basic introduction to Augmented Reality and the ARKit framework concepts. Session Management: Discover what’s required to manage an AR session properly; this includes error handling and tracking issues. Adding Objects: Learn how to place 3D Objects into augmented space. Materials Lighting: Explore the concepts of Physical Based Rendering, textures and light estimation. Creating 3D Assets: Get creative with the SceneKit editor and make your own 3D assets right inside Xcode. Facebased Apps: Find out what it takes to develop and ship FaceBased AR apps. And much, much more
ARKit by Tutorials ARKit by Tutorials By Chris Language, Namrata Bandekar, Antonio Bello & Tammy Coron Copyright ©2019 Razeware LLC Notice of Rights All rights reserved No part of this book or corresponding materials (such as text, images, or source code) may be reproduced or distributed by any means without prior written permission of the copyright owner Notice of Liability This book and all corresponding materials (such as source code) are provided on an “as is” basis, without warranty of any kind, express of implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in action of contract, tort or otherwise, arising from, out of or in connection with the software or the use of other dealing in the software Trademarks All trademarks and registered trademarks appearing in this book are the property of their own respective owners raywenderlich.com ARKit by Tutorials Table of Contents: Overview About the Cover 13 What You Need 18 Book License 19 Book Source Code & Forums 20 Section I: Getting Started with ARKit! 22 Chapter 1: Hello ARKit! 23 Chapter 2: Your First ARKit App 35 Chapter 3: Basic Session Management 61 Chapter 4: Adding 3D Objects & Textures 75 Chapter 5: Detecting Surfaces 100 Chapter 6: Adding Physics 120 Section II: Portal Apps 142 Chapter 7: Building a Portal 143 Chapter 8: Adding Objects to Your World 155 Chapter 9: Geometry, Textures & Lighting 171 Section III: Interactive AR Apps 192 Chapter 10: Detecting Placeholders 194 Chapter 11: Beginning User Interaction 213 Chapter 12: Advanced User Interaction 241 Chapter 13: Location Tracking & Beacons 268 Section IV: Face-Based AR 298 raywenderlich.com ARKit by Tutorials Chapter 14: Getting Started with Face-Based AR 299 Chapter 15: Tracking the User’s Face 311 Chapter 16: Creating Assets for Face-Based AR 326 Chapter 17: Working with Blend Shapes 345 Chapter 18: Recording Your ARKit Experience with ReplayKit 354 Section V: Game Physics 368 Chapter 19: Vehicle Physics, Part I 369 Chapter 20: Vehicle Physics, Part II 386 Section VI: Persistent & Shared Experiences 400 Chapter 21: World Mapping & Persistent AR Experiences 401 Chapter 22: Collaborative AR Experiences 420 Section VII: Image Tracking, Object Detection & AR Quick Look 437 Chapter 23: Introducing AR Quick Look 438 Chapter 24: Introducing USDZ 452 Chapter 25: Detecting Images & 3D Objects 470 Section VIII: Introduction to RealityKit 494 Chapter 26: Introducing RealityKit & Reality Composer 495 Conclusion 511 raywenderlich.com ARKit by Tutorials Table of Contents: Extended About the Cover 13 About the Authors 15 About the Editors 16 About the Artists 17 What You Need 18 Book License 19 Book Source Code & Forums 20 Section I: Getting Started with ARKit! 22 Chapter 1: Hello ARKit! 23 Augmented reality Introducing ARKit What can ARKit for you? Limitations of ARKit Technology behind ARKit Challenge Key points 24 26 27 30 32 32 34 Chapter 2: Your First ARKit App 35 Getting started 36 Key points 60 Chapter 3: Basic Session Management 61 Getting started Starting an AR session Controlling an AR session Updating the status message Handling AR session errors Handling AR session state changes raywenderlich.com 62 64 67 68 68 70 ARKit by Tutorials Challenge 72 Key points 74 Chapter 4: Adding 3D Objects & Textures 75 Getting started Adding 3D objects Creating the dice scene Shaders, materials and textures Physically based rendering (PBR) Texturing the 3D Objects Loading 3D objects Placing 3D objects Challenge Key points 76 76 78 79 81 89 91 93 97 99 Chapter 5: Detecting Surfaces 100 Getting started Adding game states Anchors Detecting surfaces Creating new planes Updating planes Creating a focus node AR Coaching Overlay View Challenge Key points 101 102 104 106 106 109 110 115 118 119 Chapter 6: Adding Physics 120 Getting started Adding physics Plane physics Randomizing rotation Force raywenderlich.com 121 123 127 129 129 ARKit by Tutorials Lights and shadows Wrapping things up Hit testing Challenge Key points 131 134 138 139 140 Section II: Portal Apps 142 Chapter 7: Building a Portal 143 The portal app Getting started Setting up ARKit Plane detection and rendering Key points 144 144 145 148 154 Chapter 8: Adding Objects to Your World 155 Getting started Hit testing Adding crosshairs Adding a state machine Key points 156 160 162 166 170 Chapter 9: Geometry, Textures & Lighting 171 Getting started Building the portal Adding the doorway Placing lights Key points Where to go from here? 172 174 186 190 191 191 Section III: Interactive AR Apps 192 Chapter 10: Detecting Placeholders 194 Introducing Razeware Mobile Kiosk 195 RazeAd 195 raywenderlich.com ARKit by Tutorials Getting started Plane detection vs object detection Detecting a rectangle Creating the billboard Displaying placemarks Adding SceneKit nodes Handling interruptions Key points 196 197 197 202 206 208 210 211 Chapter 11: Beginning User Interaction 213 Getting started Detecting a QR code Showing an image Showing an image carousel Playing a video Removing the video player Using the data encoded in the QR code Key points Where to go from here? 214 219 222 225 230 235 236 239 240 Chapter 12: Advanced User Interaction 241 The updated starter project Interacting with the billboard Going fullscreen Detecting reference images Key points 242 245 254 259 267 Chapter 13: Location Tracking & Beacons 268 Getting started Locating the user Geofencing Beacons Testing raywenderlich.com 269 270 276 286 294 ARKit by Tutorials Key points 297 Where to go from here? 297 Section IV: Face-Based AR 298 Chapter 14: Getting Started with Face-Based AR 299 ARKit for face-based AR The starter project Starting a face tracking session Handling errors and interruptions Key points 300 301 305 307 310 Chapter 15: Tracking the User’s Face 311 Detecting the user’s face Working with face geometry Handling updates Adding additional masks Connecting the switch button Setting up a basic environment Adjust the lighting Key points 312 313 316 318 322 323 323 325 Chapter 16: Creating Assets for Face-Based AR 326 Creating 3D Content Creating the Glasses class Adding the glasses option Creating Pig Creating the Pig class Key points 327 329 331 333 342 344 Chapter 17: Working with Blend Shapes 345 Animojis and blend shapes 346 Animating the brow 347 Animating the eyes 349 raywenderlich.com ARKit by Tutorials Animating the mouth 352 Key points 353 Chapter 18: Recording Your ARKit Experience with ReplayKit 354 Getting started with ReplayKit Adding ReplayKit to FaceCase Enhancing the user's experience Key points Where to go from here? 355 357 362 367 367 Section V: Game Physics 368 Chapter 19: Vehicle Physics, Part I 369 Getting started Building the monster truck Loading the monster truck Adding vehicle physics Key points 370 373 379 380 385 Chapter 20: Vehicle Physics, Part II 386 Getting started Creating an infinite ground plane Adding engine force Adding steering Limiting the speed Resetting the game Challenge Key points 387 387 390 393 396 397 398 399 Section VI: Persistent & Shared Experiences 400 Chapter 21: World Mapping & Persistent AR Experiences 401 Getting started 402 Placing anchors 403 raywenderlich.com 10 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Here’s a breakdown of all the cool features: • Rendering: RealityKit offers a powerful new physical-based renderer built on top of Metal, which has been fully optimized for all Apple devices • Animation: It has built-in support for Skeletal animation and Transform-based animation So you can animate a zombie or you can just move, scale and rotate things around with various easing functions • Physics With a powerful physics engine, you’ll be able to throw anything at it, pun intended! You’ll be able to adjust real-world physics properties like mass, drag and restitution, allowing you to fine-tune collisions • Audio: With spacial audio understanding and automatic listener configuration, you’ll be able to attach sound effects to 3D objects The sounds are tracked and will sound realistic based on their position in the real world • ECS: From a coding perspective, RealityKit enforces the Entity Component Design Pattern to build objects within the world • Synchronization: The framework has built-in support for Networking, designed for collaborative experiences It even offers automatic synchronization of entities between multiple clients Enough talk, let’s dive into some code next Creating a RealityKit project Time to create your first RealityKit project Start up Xcode, let’s create a new Augmented Reality App project from scratch raywenderlich.com 497 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Click Next to continue Set the Project Name to PigFlip and fill in the rest of the information Behold, Xcode now has a newly added RealityKit option added under the Content Technologies: Click Next and save the project somewhere safe Xcode will now continue to generate the project for you, with the end result looking something like this: Before doing anything else, build and run the project to give it a quick try before we take a look at what Xcode generated raywenderlich.com 498 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Wow, out of the box your app is already doing quite a bit It’s requesting access to the camera, it’s detecting horizontal surfaces and it’s got proper environmental lighting and reflections And where did that smooth looking cube come from? You didn’t have to write one single line of code for all that Nice! Well, you’ll be even more impressed when you take a look at what’s inside the project Reviewing the project At first glance, you’ll notice the usual suspects, but there are a few new things too Let’s take a look: • AppDelegate.swift: Your apps starting point • ViewController.swift: Your app controller with the code behind Note that it’s importing the RealityKit framework There’s an IBOutlet to an ARView and then two lines of code that loads a box and appends a boxAnchor to the scene More on this a little later • Experience.rcproject: This is a Reality Composer project, which is essentially a 3D scene that contains the box and the box anchor used in the previous step More on this a little later as well raywenderlich.com 499 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer • Main.storyboard: Your app’s main storyboard Note that it’s using an ARView that’s already connected to the arView outlet inside the view controller • Assets.xcassets: This contains all of your project assets like images and app icons • LaunchScreen.storyboard: The UI the user will see while your app is launching • Info.plist: The app's basic configuration settings Note that there's already Camera Usage Description property added, you just need to go change it to something more appropriate for your app RealityKit API components There are a few main components that form parts of the RealityKit API that you should know about raywenderlich.com 500 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Here's an example of a typical structure containing all the important elements: • ARView: The ARView sits at the core of any RealityKit experience It's responsible for doing all of the heavy lifting It even comes with full gesture support, allowing you to attach gestures to entities It's also responsible for the post-processing camera effects, very similar to the effects seen in AR Quick Look • Scene: This can be seen as the container for all your entities • Anchor: RealityKit exposes all the available ARKit's anchors (Plane, Face, Body, Image and Object) as first-class citizens Anchors form the local root for entity structures Note that content attached to an Anchor will stay hidden until the anchor is successfully identified and connected to the real world • Entity: All the virtual content in a scene can be seen as Entities — the basic building blocks of your experience A tree-like hierarchical structure is established by parenting entities to other entities • Components: Entities consists of different types of Components that gives entities specific features and functionality, like a Model, Collision and Physics Now that you know something about RealityKit, let's take a look at Reality Composer raywenderlich.com 501 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer What is Reality Composer? Reality Composer is an iOS and macOS-based app that you can download and install from the App Store It enables the creation of simple interactive AR-based experiences with an intuitive WYSIWYG (What You See Is What You Get) design that's self-explanatory and super easy to use You can even build your scenes directly in AR when you're using the app on an iPad or an iPhone Here's an example of what it looks like on an iPad: Reviewing Reality Composer Reality Composer comes with its very own Content Library with all sorts of objects that will get you building your scenes quickly! Let's add some custom content next raywenderlich.com 502 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Back in your Xcode project, select Experience.rcproject and then click on the Open in Reality Composer button on the right Reality Composer will launch and it will automatically load the Experience.rcproject file Let's a quick review of the interface: • Top Tool Bar: At the top, you'll see the main toolbar that gives you quick access to various tabs and options You can hover over each option to see a short description of what that option does • Scenes Panel: Select the Scenes button at the top-left to open the Scenes panel on the left Here you'll be able to manage your scenes • Properties Panel: Select the Properties button at the top-right to open the Properties panel on the right Here you'll find all sorts of properties of the active selected element raywenderlich.com 503 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer • 3D Viewport: In the middle, you'll see the main 3D viewport where you get to add your virtual content You can easily pan, zoom and rotate around the scene with your mouse and mouse wheel Creating a new Reality scene Let's add a new scene, then load in some custom content With the Scenes panel open, select the plus button at the top to create a new scene You're now presented with all of the available Anchor types Select Horizontal then select the Choose button to continue The new scene will now activate once a horizontal surface has been detected raywenderlich.com 504 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer With the scene still active, open the Properties panel on the top-right Rename the scene from Scene to Pig You'll also notice that you can change the Anchor type and the scene's physics properties at this point Importing custom content Reality Composer comes with its own content library, nicely classed into various types of activities that you can pick from Click the Objects button at the middle-top to open the content library Feel free to browse through the available content Note that some of the available objects are not just basic models but intelligent shape generators that offer a massive amount of variants raywenderlich.com 505 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer For now, let's import a custom object from a file Click the Import button on the topright, then find the Pig.usdz model under the starter/resources/Pig folder Note that you can only import USDZ content into Reality Composer Click the Import button to continue Note: USDZ is Apple's latest file format for augmented reality Fun fact; it was created by a partnership between Apple and Pixar The Pig model is automatically imported and placed into your active scene Click on the Pig model to select it, then under the Properties panel, rename the object to Pig raywenderlich.com 506 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Adding behaviors Behaviors will bring your content to life by adding Triggers and Actions to your content Click the Behaviors button at the top-right to open the Behaviors panel With the Pig object still selected, select the plus button to add a new behavior, then choose Tap & Flip raywenderlich.com 507 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer This will add a new behavior consisting of a Tap Trigger with an Emphasise Action Sequence Click the play button at the top to test your scene Tap on the pig and see what happens Excellent, the pig flips once tapped So satisfying! Moving on Select File > Save to store your changes Then, close Reality Composer and go back to your Xcode project Note: You can also export your scene as a reality file at this point in time, which is 100% compatible with AR Quick Look Code generation Reality Composer is tightly integrated into Xcode When you build your project, Xcode will inspect all the associated Reality files within the project and it will generate Swift code The generated code provides strongly-typed access to all the content within the Reality file It also provides direct access to be able to invoke triggers for custom actions within your code In this case, Xcode generated an Experience.swift file with strongly-typed access to the Pig scene and Pig object within the Reality file raywenderlich.com 508 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer Loading Reality scenes Once back in Xcode, you should now notice your newly added scene The only thing left to is modify the code to load and display your new scene Open the ViewController.swift file and replace all of the code inside viewDidLoad(), right after calling super.viewDidLoad(), with: // let pigAnchor = try! Experience.loadPig() // arView.scene.anchors.append(pigAnchor) Let's break it down: Thanks to the generated code and the way you named your scene and objects within it, you've got strongly-typed access to the scene There's also a few helper functions like the useful loadPig() function This simply loads the Anchor entity into pigAnchor Finally, the anchor is appended to the scene Now, when the scene successfully identifies a valid matching anchor, it will make all of the content attached to that anchor visible Build and run your project and test it all out raywenderlich.com 509 ARKit by Tutorials Chapter 26: Introducing RealityKit & Reality Composer The app starts up and, once it detects a horizontal surface, the pig is placed on top of it When you tap on the pig, it does a little flip So satisfying There's nothing like flipping pigs! Note: As always, you can find a copy of the completed project under the final/ PigFlip folder Key points You've reached the end of this chapter, let's recap some of the key takeaways: • You now know about Apple's latest and greatest RealityKit framework, designed for AR first to help reduce the complexities AR developers have to face on a daily basis • You also got a basic introduction to Reality Composer, a brand-new app available for macOS and iOS to build basic interactive AR experiences with a simple intuitive design Now anyone can create AR experiences really quick • Reality Composer is tightly integrated into Xcode, which will generate companion code that gives you strongly-typed access to your virtual scenes and content Where to go from here? Now that you've been introduced to RealityKit and Reality Composer, know that you've only seen the tip of the iceberg Don't stop here, there's a fleet of information about RealityKit and Reality Composer from WWDC 2019 waiting for you It's highly recommended that you go check out the following: • Information about Reality Composer and RealityKit - https://apple.co/2OYxIsG • WWDC video's on RealityKit & Reality Composer - https://apple.co/2GP6iyI raywenderlich.com 510 C Conclusion Thank you again for purchasing ARKit by Tutorials If you have any questions or comments as you continue to develop for ARKit, please stop by our forums at http:// forums.raywenderlich.com Your continued support is what makes the tutorials, books, videos and other things we at raywenderlich.com possible We truly appreciate it Best of luck in all your development and AR-making adventures, • Chris Language (Author), Namrata Bandekar (Author), Antonio Bello (Author), Tammy Coron (Author/FPE), Joey deVilla (Tech Editor), Matthew Morey (Tech Editor), Jerry Beers (Tech Editor), Morten Faarkrog (Tech Editor) The ARKit by Tutorials team raywenderlich.com 511 .. .ARKit by Tutorials ARKit by Tutorials By Chris Language, Namrata Bandekar, Antonio Bello & Tammy Coron Copyright ©2019 Razeware LLC Notice of Rights... anytime raywenderlich.com 20 ARKit by Tutorials Book Source Code & Forums Visit our ARKit by Tutorials store page here: • https://store.raywenderlich.com/products /arkit- by- tutorials And if you purchased... raywenderlich.com 18 L Book License By purchasing ARKit by Tutorials, you have the following license: • You are allowed to use and/or modify the source code in ARKit by Tutorials in as many apps as