At the company’s annual WWDC developer conference today, Apple revealed ARKit 3, its latest set of developer tools for creating AR applications on iOS. ARKit 3 now offers real-time body tracking of people in the scene as well as occlusion, allowing AR objects to be convincingly placed in front of and behind those people. Apple also introduced Reality Composer and RealityKit to make it easier for developers to build augmented reality apps.

Today during the opening keynote of WWDC in San Jose, Apple revealed ARKit 3. First introduced in 2017, ARKit is a suite of tools for building AR applications on iOS.

From the beginning, ARKit has offered computer vision tracking which allows modern iOS devices to track their location in space, as well as detect flat planes like the ground or a flat table which could be used to place virtual objects into the scene. With ARKit 3, the system now supports motion capture and occlusion of people.

Human Occlusion & Body Tracking

Using computer vision, ARKit 3 understands the position of people in the scene. Knowing where the person is allows the system to correctly composite virtual objects with regard to real people in the scene, rendering those objects in front of or behind the person depending upon which is closer to the camera. In prior versions of ARKit, virtual objects would always show ‘on top’ of anyone in the scene, no matter how close they were to the camera. This would break the illusion of augmented reality by showing conflicting depth cues.

Similar tech is used for real-time body tracking in ARKit 3. By knowing where people are in the scene and how their body is moving, ARKit 3 tracks a virtual version of that person’s body which can in turn be used as input for the AR app. Body tracking could be used to translate a person’s movements into the animation of an avatar, or for interacting with objects in the scene, etc.

From the footage Apple showed of their body tracking tech, it looks pretty coarse at this stage. Even with minor camera movement the avatar’s feet don’t remain particularly still while the rest of the body is moving, and small leg motions aren’t well tracked. When waving, the avatar can be seen to tip forward in response to the motion even though the user doesn’t. In the demo footage, the user keeps their arms completely out to the sides and never moves them across their torso (which would present a more challenging motion capture scenario).

For now this could surely be useful for something simple like an app which lets kids puppeteer characters and record a story with AR avatars. But hopefully we’ll see it improve over time and become more accurate to enable more uses. It’s likely that this was a simple ‘hello world’ sort of demo using raw tracking information; a more complex avatar rig could smartly incorporate both motion input and physics to create a more realistic, procedurally generated animation.

SEE ALSO
Report: Apple Developing AR/VR Headset with 8K Resolution Per-eye Slated for 2020

Both human occlusion and body tracking will be important for the future of AR, especially with head-worn devices which will be ‘always on’ and need to constantly deal with occlusions to remain immersive throughout the day. This is an active area of R&D for many companies, and Apple is very likely deploying these features now to continue honing them before the expected debut of their upcoming AR headset.

Apple didn’t go into detail but listed a handful of other improvements in ARKit 3:

  • Simultaneous front and back camera
  • Motion capture
  • Faster reference image loading
  • Auto-detect image size
  • Visual coherence
  • More robust 3D object detection
  • People occlusion
  • Video recording in AR Quick Look
  • Apple Pay in AR Quick Look
  • Multiple-face tracking
  • Collaborative session
  • Audio support in AR Quick Look
  • Detect upt to 100 images
  • HDR environment textures
  • Multiple-model support in AR Quick Look
  • AR Coaching UI

RealityKit

Image courtesy Apple

With ARKit 3, Apple also introduced RealityKit which is designed to make it easier for developers to build augmented reality apps on iOS.

Building AR apps requires a strong understanding of 3D app development, tools, and workflows—something that a big portion of iOS developers (who are usually building ‘flat’ apps) aren’t likely to have much experience with. This makes it less likely for developers to jump into something new like AR, and Apple is clearly trying to help smooth that transition.

From Apple’s description, RealityKit almost sounds like a miniature game engine, including “photo-realistic rendering, camera effects, animations, physics and more.” Rather than asking iOS developers to learn game engine tools like Unity or Unreal Engine, it seems that RealityKit will be an option that Apple hopes will be easier and more familiar to its developers.

SEE ALSO
Apple CEO on AR Headsets: 'We don't want to be first, we want to be the best'

With RealityKit, Apple is also promising top notch rendering. While we doubt it’ll qualify as “photo-realistic,” the company is tuning rendering the allow virtual objects to blend as convincingly as possible into the real world through the camera of an iOS device by layering effects onto virtual objects as if they were really captured through the camera.

“RealityKit seamlessly blends virtual content with the real world using realistic physically-based materials, environment reflections, grounding shadows, camera noise, motion blur, and more, making virtual content nearly indistinguishable from reality,” Apple writes.

RealityKit, which uses a Swift API, also supports the creation of shared AR experiences on iOS by offering a network solution out of the box.

Reality Composer

Image courtesy Apple

Just like RealityKit, Reality Composer aims to make things easier for developers not experienced with game engines, workflows, and assets. Apple says that Reality Composer offers a library of existing 3D models and animations with drag and drop ease, allowing the creation of simple AR experiences that can integrate into apps using Xcode or be exported to AR Quick Look (which allows built-in iOS apps like Safari, Messages, Mail, and more, to quickly visualize 3D objects at scale using augmented reality).

In addition to the built in object library, Reality Composer also allows importing 3D files in the USDZ format, and offers a spatial audio solution.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Ted Joseph

    With almost all of Apples products, Apple waits then comes out with something spectacular. iPhone, MacBook Pro, AirPods, HomePod, Apple TV. I have all of it. Cant wait to see their AR product! I have a feeling it will be powered by iPhone which will wireless send data to the glasses. If they can achieve this at a lightweight reading or sunglass frame, they will conquer AR!!!

    • jj

      or not… out of all those things, only the iphone was something new and even that was a combination of a cell phone (previously invented) and an mp3 player(also previously invented). so they really just sell fadware at high prices to chums who are ego driven and want to flex….

      • Wayne Singh

        While I agree, what I think they do is take existing tech and make it easy to use for the average person and market the hell out of it. For example, I would always recommend an iPhone to my parents over Android simply because it works and I don’t have to do as much family tech support. For myself I run Android because i like the flexibility and customisation I can do on Android that I cant on iOS.

      • ann

        Engaging in a job which enables you to truly enjoy your life, It is really the things you need to look for. Travel and leisure as well as do the job worldwide from Italy, Rome, Berlin to Paris. Simple Internet based work, simply get your laptop and you are ready to make money. Now I am able to earn 135 US dollars just doing work for couple of minutes. Stress free work from the most respected and incredible on-line sites. Don’t overlook and try this out ==> gigankle.dreht-auf.de

        • BarryG

          This ad approved by the Nigerian scam committee. Please have credit card, bank account and SSI ready. By clicking on the link, you agree to have spyware installed on your browser. Thank you.

  • Nejham Mosquera

    The have the software, now they just need to build the AR headse.

  • oompah

    With a cash rich company like Apple
    I only expect a set of glasses with:
    1. 100-120 deg glasses
    2. Optical waveguide like hololens
    3. foveated rendering
    4. Eye tracking
    5. Glasses so light that they look like ordinary spectacles (for that they can have these connected to the pocket based cellphone)

    6. Shall I say a specialized chip for ray tracing , but that’d be one too many hah ha

    The present AR/VR glasses are so awful they always create headache after a few hours

    • kontis

      Their rumored AR prototype glasses had terrible FOV.

    • BarryG

      Apple has some large fraction of a trillion in the bank. All that will not create an as yet impossible technology. Waveguides will not do this for example.

  • ShiftyInc

    This company has been so out of touch as of late.

    • Downvote King

      What examples do you see of Apple being out of touch?

      • nejihiashi88

        1-a monitor stand sold for 1000 dollars .
        2-outdated software features that were introduced before 3 years.
        3-closed system that is not practical.
        4-no innovations always late with new technologies.

        • Downvote King

          The monitor stand was pretty funny, although I think it’s been mostly missed that the monitor it’s made for is also $6000, both are extremely high end premium products. Still funny though. It’ll be interesting to see how the Pro line fares now that it’s been clearly bumped into the stratosphere of prestige customers. The rest of the points seem the same as Apple has always been to me.

          They’ve never really innovated, just taken proven product categories and refined/simplified the product for ease-of-use, production quality, fit and finish, etc. A closed system is integral to this approach. Like a video game console. It’s more important for them that their products are solid, reliable and work well relative to their competitors than blaze new ground, although as they’re such a big, premium brand they get more press coverage so a lot of the features in their products are often mistaken as being cutting edge by the public, when in reality they are generally just attempting to make a better, more cohesive product.

  • That’sright

    AR on a phone will never be great.

  • Body tracking is overly interesting… imagine the possibilities of doing cheap mocap for indie devs just by using a phone

    • kontis

      Many indie devs use kinect for mocap or multiple ps3eyes and it costs 3x less than iPhone with this functionality.

      Those who want something more reliable invest in vive + trackers. Currently the best quality/price mocap on the market.

  • naomi

    Make /per month Up to $5011 by working for Cardinal Health from home. There are plenty of advantages of working on the net from home like you are your own boss. There is no one saying to you what to do,use your laptop computer with internet access, carry it to seaside beach or perhaps a park or mountain, and do your work in peace with nature, potential to earn money is boundless all depending on you. You can fulfill all your wishes by working on line ,which has limits when you work 9am-5pm in a office job. You want to spend more time with your family ,or even go on a family trip? On job you must seek permission from your boss. Online earning, simply take your laptop with you on your family vacation.You have the luxury to be financially FREE. Here’s the most convenient way to start ===> urethaneboundless.gsoz.com

  • jenniffer

    One year past I made the decision to drop my prior work and I am too joyful at this point…. I started off working online, for a business I located online, for just a few hour or so everyday, and I obtain more than I did on my office job… My pay-check for prior month was 9,000 US dollars… The most advantageous thing about this is the additional free moments I managed to get for my little kids…and that the single requirement for this job is basic typing and also accessibility to internet… I am capable to put in quality time with my close relatives and buddies and take care of my little kids as well as going on family holiday with them very frequently. Don’t skip this chance and make sure to take action fast. Try it out, what it’s about… folkking.sms.cr