Skip to main content

The New Generation of Spatial Apps

With Apple's release of the next-gen XR OS, visionOS, and its new spatial-computing architecture (Unified Rendering app model), the long-standing bottlenecks of traditional XR apps have been fundamentally addressed.

The new generation of XR apps - spatial apps - are not cut off from the 2D apps on desktop and mobile platforms. Instead, they extend the advantages of those 2D apps, adding spatial capabilities that bring optional enhancements on spatial-computing platforms and free them from flat 2D windows.

Comparing Three App Types​

info

In the comparisons below, green background means advantage and red background means disadvantage.

Multitasking​

  • Spatial apps

    • Can run and use multiple apps that include 3D content at the same time.
    • Can quickly switch between the currently focused apps.
    • Can keep auxiliary apps open while using a primary app.

  • Desktop/Mobile apps

    • Can run and use multiple apps at the same time.
    • Can quickly switch between the currently focused apps.
    • Can keep auxiliary apps open while using a primary app.

  • Traditional XR apps

    • Only one app with 3D content can run at a time.
    • Switching requires quitting the current app and launching the next, which is costly.
    • Cannot keep auxiliary apps (especially 3D apps) open while using a primary app.

Relationship with the OS​

  • Spatial apps

    • An app can own the entire 3D space or only a local part of it such as its own scene container (similar to a bounding box), while the OS handles the rest.
    • Because the OS supplies most functionality and a unified experience, the app can focus on its unique value, so design and development costs stay low and package size is small.

  • Desktop/Mobile apps

    • Each app only handles part of what's shown on the screen (the content within its own window), while the OS handles the rest.
    • Because the OS supplies most functionality and a unified experience, the app can focus on its unique value, so design and development costs stay low and package size is small.

  • Traditional XR apps

    • The app takes over the entire 3D space, rendering all visuals and features (excluding the OS's temporary flat overlays, like the global toolbar).
    • The app must satisfy nearly all user needs on its own, so design and development costs are high, package size is large, and experiences are inconsistent across apps.

3D Capability​

  • Spatial apps

  • Desktop/Mobile apps

    • Both 2D and 3D content in the app must stay in a single plane (3D content is projected onto that plane).

  • Traditional XR apps

    • 2D and 3D content in the app can be shown anywhere in the app's exclusive 3D space.

Interaction Capability​

  • Spatial apps

    • Leverage environment sensing, head tracking, hand tracking, and eye tracking to deliver truly natural interaction.
    • Need no controllers. Regardless of GUI type (2D or 3D) or distance, these apps default to using the most efficient, low-effort, and natural eye-hand interaction. The OS handles interaction implementation and visual effects, reducing app costs and keeping the user experience consistent.

  • Desktop/Mobile apps

    • Interaction has evolved from mouse and keyboard to multi-touch, gradually approaching natural interaction in the physical world.

  • Traditional XR apps

    • Leverage environment sensing, head tracking, hand tracking or XR controllers to allow further progress toward natural interaction beyond multi-touch on screens.
    • For distant 2D GUIs, users must aim a pointer ray with a controller or hand, which feels less natural and is less efficient than than a mouse.

Development Approach​

  • Spatial apps

    • Built on 2D GUI frameworks enhanced with spatial features.
    • Fit almost every business domain and app type (except a few heavy 3D niches) while building on the existing product/GUI design patterns and keep developing from there.
    • APIs are highly abstract, concrete, intuitive, and have a low learning curve.
    • Large potential base of developers ready to start building and strong integration with the open-source ecosystem.

  • Desktop/Mobile apps

    • Based on 2D GUI frameworks.
    • Fit almost every business domain and app type (except some 3D games) with existing product/GUI design patterns.
    • APIs are highly abstract, concrete, intuitive, and have a low learning curve.
    • Large developer base and thriving open-source ecosystem.

  • Traditional XR apps

    • Based on 3D game engines.
    • Few existing product/GUI design patterns for non-gaming apps.
    • APIs come from computer graphics, they are relatively low level, less intuitive, and have a steep learning curve.
    • The developer base and ecosystem are smaller, skewing toward paid tools with fewer high-quality open-source options.

Interaction Implementation​

UI Implementation​

  • Spatial apps

    • The GUI is built from UI components (including 3D containers).
    • The OS and framework render those components each frame; the app does not care about drawing.
    • Position and size of these UI elements are automatically decided based on their layout relationships (like nesting hierarchy or order) and attributes. Developers use them like building blocks - highly concrete and intuitive - allow developers to work entirely in code without the assistance of visual editors.

  • Desktop/Mobile apps

    • The GUI is built from UI components.
    • The OS and framework render those components each frame; the app does not care about drawing.
    • Layout relationships and attributes determine position and size; developers think in building blocks, code-centric, no visual editor needed.

  • Traditional XR apps

    • The GUI is built from graphics objects.
    • The app itself renders those objects each frame.
    • Position and size come from coordinates. Developers usually need visual editors to help make the rendering and using of these objects more clear and intuitive.

Problems with Traditional XR Apps​

Before visionOS, apps on XR platforms were completely different from apps on desktop and mobile:

Desktop apps

Traditional XR apps

From the comparison above, we can see that besides the three main advantages mentioned in the previous section, traditional XR apps have downsides both in terms of user experience and development, losing many benefits of desktop/mobile apps. Traditional XR development could not reuse existing app code, tools, and programming mindsets from mainstream platforms. This makes it harder to support diverse use cases and business domains, leads to fewer developers, and raises both the entry barrier and development costs.