What I Want From visionOS 3, visionOS 4, and visionOS 5

Four and a half years ago I complained about the limitations of RealityKit 1.0 for iOS. No concave models. No opacity control. No videos. No shaders. Some of those issues were (arguably) resolved the following week at WWDC 2020, and others took another few years to be addressed. I didn’t predict what was in Apple’s iOS development pipeline years in advance by way of insider knowledge or witchcraft – I was an indie & contract iOS developer with knowledge of where things got annoying, cumbersome, or impossible when building with ARKit & RealityKit. Building things with a nascent framework and investigating its rough edges give you a good idea of its strengths, weaknesses, and bits that are probably littered with TODOs in the source code.

The same holds true now. I build stuff for visionOS. I experiment and tinker and explore. And I hit walls. Generally, I’ll hit walls where other devs (internal or external to Apple) also hit walls. Those hurdles and pitfalls represent opportunities for Apple to expand what’s possible to build for visionOS.

Let’s expand upon visionOS 1 (above) and visionOS 2

I’ve only released one personal visionOS app thus far and I have another two in the works I aim to release later this year. But I’ve already built multiple visionOS client projects and multiple visionOS personal research experiments. I’ve also already abandoned a handful of visionOS proofs of concept along the way. Let’s dive into what I’m looking forward to in visionOS 3, visionOS 4, and visionOS 5 that will make building spatial computing apps faster, easier, and more powerful.

In my flimsy estimation of decreasing level of likelihood:

Feature Requests for Future Versions of visionOS

3D, Volumetric MapKit

This is a no-brainer, an absolute must; it’s just a matter of when. Apple has had 3D versions of select cities in the Maps app for over a decade via Flyover. Viewing a 2D MapKit map on a flat plane in visionOS 2 just doesn’t cut it. Since I like maps, I find it insulting. I don’t expect immediate availability of a global, three-dimensional map that accurately represents the entire world’s built environments, natural features, and terrain when MapKit goes 3D – I expect the typical roll-out over a few years as Apple adds regional 3D maps in dribs and drabs.

I don’t have much of an argument here beyond petulant indignation and the staunch belief this should have been available, even in some minimal form, in visionOS 1.

If what we’re trying to build is spatial computing, how we’ve been navigating our space for over 8,000 years (i.e. maps) should have a first-party option to be represented spatially.

I hope to see some developments here in visionOS 3.

Spatial Interaction and/or a New Robust Multiplayer Framework

More Spatial Interaction between devices, please

Way back in 2020 Apple announced Nearby Interaction. The ability for devices to know precise locations of other devices sounds very exciting and handy, especially for spatial computing. Since that announcement I may have used Nearby Interaction once, when I was in the discovery phase with a prospective client. I don’t have an AirTag, nor a modern car, nor a HomePod – all three of those benefit slightly from the presence of an ultra-wideband chip like U1 or U2. Did Apple really create a separate line of chips and bundle over a billion of them in devices over the past 5 years to…slightly improve AirDrop?

I’m actually unsure how the Vision Pro is automatically placing a “Connect” button above MacBooks to launch the Mac Virtual Display feature – my guess is they’re largely leaning on visually recognizing MacBook keyboards, similar to visionOS 2’s keyboard breakthrough feature. But I bring up this singular instance of the Vision Pro understanding where another physical Apple device is to say: This is awesome.

Having my Vision Pro accurately understand where my iPhone, iPad, Apple Watch, and Apple TV are in three dimensional space (without hacky workarounds) would be a capability I would love to explore. And that’s an incomplete list of Apple products – perhaps the products that would be even more useful, if Apple does attack the smart home space in the coming months and years with new products, would control and augment various characteristics of my home.

And all that is before we get to the limitless possibilities opened up when software on my Vision Pro can recognize visionOS devices being worn by other people (although that scenario is vanishingly rare today in the world at large, I expect that to be increasingly possible over the coming years for normal people – and sooner in industrial or business settings).

So is the answer for Apple to drop U2 chips (or U3 or whatever) into all of its products and expand Nearby Interaction into a new Spatial Interaction framework? Maybe. I think they had the beginnings of the “real” answer over 6 years ago when they released sample code of a multiuser AR experience that utilized ARSCNView, and then iterated on that with collaborative sessions in 2019. I think the answer is either a brand new, privacy-and-spatial-computing-focused multidevice/multiplayer framework or for Apple to tack on a straightforward collaborative session feature to RealityView.

I deem adding such a capability as critical to the short-term health of visionOS as a platform, and I hope to see some changes here by visionOS 4.

RoomPlan

RoomPlan is too powerful, too expensive, and too promising to not come to visionOS.

I expect some flavor of RoomPlan on visionOS, paired with forthcoming smart home products, to drive a smaller-yet-still-significant chunk of personal, non-social adoption down the road. But I am less convinced it will be a straight “port” of the iOS version of RoomPlan. I don’t think Apple is willing to give developers access to accurate, semantic 3D representations of people’s homes on a device we wear on our heads.

I don’t blame them. There are already (and have been for years) many companies big and small that are eager to pillage the datasets accessible through something that tracks our eyes, hands, and surroundings.

I don’t know if the visionOS spin on RoomPlan will be super locked-down, severely watered-down, or something else. But I do expect visionOS 3 to provide more capabilities that expand upon what’s possible using today’s room tracking, scene understanding, and world tracking.

ARKit Multitasking or ARKit Multi-App Concurrency

It wasn’t until iOS 4 was released in 2010 that iPhones got multitasking. I personally can’t recall everyday details from last week, much less the day-to-day experience of using my iPhone 3G in 2008 before multitasking arrived.

But as a visionOS & iOS dev focused on ARKit and RealityKit – developing for visionOS 2 feels like pre-multitasking days to me. The looming, preeminent issue is whether multiple apps will ever be able to access ARKit functionality at the same time.

This can get tricky, but generally speaking an app needs to request specific ARKit functionality. ARKit functionality in visionOS includes, amongst a few others:

• Plane detection
• World tracking
• Hand tracking
• Scene reconstruction
• Image tracking
• Object tracking
• Room tracking

So I can build some crazy, intense app that uses every single one of those features – but since that app uses ARKit it is sandboxed and effectively no other apps can run simultaneously as my ARKit app. Even if I use a single ARKit feature, like hand tracking, I cannot run any other app as long as I have my hand tracking app foregrounded.

I understand there are privacy concerns that led to this decision; if an ARKit session that’s gathering data about my home is shared between apps, who’s to say a nefarious app won’t find a way to build up a database of the insides of people’s homes.

I understand there are severe user experience concerns; it’s my uninformed guess that ARKit sessions are expensive to run, and now the OS needs to understand how to share ARKit session data between multiple apps? What if one app mishandles data in a way that causes the ARKit session to crash – and now my app crashes too or underperforms or appears to fail because of some other developer’s buggy code? And what happens when two apps using scene reconstruction simultaneously draw a model in the same spot, causing a visual mishmash and other unintended consequences?

But I maintain that a visionOS that sandboxes ARKit apps is a seriously hamstrung OS. There must eventually be a way to run my app that only uses hand tracking alongside another app that only uses world tracking. To expect users and developers to be satisfied with absolute ARKit monotasking on visionOS is akin to saying iPadOS, iOS, and macOS users would have been happy to only ever run one app at a time. No picture in picture. No split view. No alt-tabbing. No windows.

No way.

I don’t expect the answer is to just “share” an ARKit session across multiple apps. That doesn’t make sense and I would expect that to fail for innumerable reasons.

So while I don’t have a good answer or mental model for precisely how ARKit multisessioning would be implemented, off the top of my head: How about similar to how you can’t play two videos with audio at the same time on iOS, there are similar single-instance resources for visionOS ARKit apps. Only one app can use hand tracking at a time. Only one app can use world tracking at a time. But if I have five different ARKit apps that are each using a different ARKit capability – they can all run simultaneously in the Shared Space. If I launch a second app that uses hand tracking, the first app will gracefully pause or recede (similar to how the YouTube iOS app pauses when in Picture in Picture mode and you start to play another video with audio) and the latest app to use hand tracking will take precedence. I think of this as ARKit multi-app concurrency (if this ever materializes, it definitely won’t be called that).

This could lead to some confusion and eventually some new norms being formed; or perhaps there’s a glaring reason why this wouldn’t work or is a terrible idea too.

But I believe ARKit multitasking, or something that yields similar results, is essential for the long-term viability of visionOS as a platform. I know there are privacy concerns, I know there are hardware concerns, and there are definitely UX concerns – but I believe this is the single most important thing visionOS needs if its spatial computers are to ever fulfill the promise of their unique strengths, without throwing us into an ad-filled hellscape where all devs and all apps have unfettered access to all cameras and all sensors.

I would be surprised if this shows up before visionOS 5…if it ever does.

Postprocessing

The very first setting I changed on my first Apple Watch was to furiously turn off the Breathe reminders

This one’s easy. Apple added postprocessing effects to RealityKit in 2021. I didn’t actually dive in until I explored postprocessing last year while prototyping a real-time spatial map. It’s pretty cool.

I also haven’t seen many/any apps use RealityKit postprocessing in the wild. It’s definitely a nice-to-have from a visual perspective, and there must be some groundwork already laid in the iOS code that drives postprocessing…but we already got a super watered-down visionOS version of this by way of the colorMultiply SurroundingsEffect.

Given Apple’s focus on calm, placid experiences and the existence of a “High Motion” warning for visionOS apps – I can’t imagine adding API functionality that could drastically filter, animate, or transform a user’s passthrough video will be high on Apple’s list.

But who knows! It would certainly be fun to play around with.

PersonaKit

Would she rather be a dragon?

I vacillate on whether this is a framework that is absolutely coming in a couple years, or if this is fundamentally verboten.

I believe if visionOS becomes a significant spatial computing platform in several years, Personas will be a vital contributor to that ascension. A 5-minute Spatial FaceTime between Vision Pro users – where Personas SharePlay an app and discuss a video, website, or meme together – may make you a believer too.

For visionOS to reach a promising position by 2030, which path for Personas would be more valuable (in terms of sales, market share, privacy, defensibility, platform lock-in, trust, mind share, whatever):

1. A true-to-life, faithful reflection of what we actually look like
2. An avatar we have the option to meticulously craft, tweak, edit, and update with a plethora of options

In the world of the 2010s and beyond, where warehouses full of thousands of iPhones (and one human assigned to a couple hundred iPhones at a time) generate countless fake downloads, reviews, comments, and other ephemera…tying a device’s usage to a single verified, authentic person is enticing. At least to some of us.

In the world of the 2020s and beyond where pictures, videos, audio, worlds, and “people” are generated and manipulated and weaponized for a full gamut of reasons – a realistic Persona we can trust may be downright wonderful.

There is a definite argument for trusting that when I interact with a Persona I know who the person on the other side is, what they look like, and what they sound like.

And then there is the reality of the world we live in. Photoshop. Insta. Facetune. Deepfakes. AI agents. People have been altering their appearance in minute and massive ways since…forever.

If a theoretical (public) PersonaKit were released with access to skin tones, blend shapes, facial landmarks, hair options, hand APIs, and more in visionOS (i.e. a lot of what we had over seven years ago with ARFaceAnchor): all bets are off. Expect an abundance of lizard people, anime waifus, celebrity and fictional character knockoffs, orcs, and bad-virtual-plastic-surgery Personas in your next SharePlay session.

Which option is “better”? The boring option or the anarchic option?

I don’t know.

But I think rather than going solely with one of those extremes:

• In the short term, Apple keeps things relatively tame. There will be the “true reflection” Personas and some Animoji-esque options. It will be crystal clear when you’re speaking with an “accurate” Persona versus a cartoon caricature.
• In the longer term, perhaps Apple adds some version of PersonaKit that offers various customization options. You could then have different modes, similar to iOS Focus modes. You set your Persona mode to use your realistic Persona for FaceTime and Zoom, your Animoji-esque T-Rex Animality for social apps, and an elaborate custom ghillie-suited sniper Persona (that looks nothing like the actual you) for gaming.

Right now, I have low expectations we will see a Persona modification framework anytime soon.

Unless Apple buys (the AR-related parts of) Snap.