What kind of OS is 'visionOS' installed in Apple's mixed reality headset 'Vision Pro'?

The MR headset ' Vision Pro ' announced at Apple's developer conference ' WWDC23 ' held on June 6, 2023 (Japan time) will be equipped with an OS named ' visionOS '. Apple has released an overview of visionOS about what this visionOS is.

visionOS-Apple Developer
https://developer.apple.com/visionos/

◆ VisionOS Spatial Computing
Vision Pro is a device that gives you an experience called Augmented Reality (AR) or Mixed Reality (MR). Basically, you can see the environment around you as it is through the visor, and icons, windows, and content are displayed on it. You can see how it feels specifically by looking at the following article that summarizes the contents of the presentation at WWDC23.

Apple announces AR headset 'Vision Pro', freed from the limitation of display, allowing you to operate your favorite applications with your eyes, hands and voice wherever you like-GIGAZINE

visionOS is an OS with spatial computing in mind that displays windows and contents in space in three dimensions.

・Spaces
A space is the concept of a space in which windows and content are displayed. Applications are basically launched in a shared space, and multiple windows are displayed side by side just like on macOS. Apps can display content using windows and volumes (described below), and users can rearrange them wherever they like. For a more immersive experience, an application can also open its own full space and display just that application's content. Within full space you can view windows and volumes, create borderless 3D content, open portals to other worlds, and be fully immersed in your environment.

・Windows
One or more windows can be created in the visionOS app. These are built with SwiftUI , and in addition to traditional views and controls, adding 3D content adds depth to the user experience (UX).

・Volume
Volume is what gives your app a 3D feel and adds depth. 3D volumes are constructed in SwiftUI using RealityKit or Unity to create a UX that can be viewed from any angle in a shared space or the full space of your app.

◆ Extension of the framework
The following development environments and frameworks have been extended or newly designed for visionOS spatial computing.

・Xcode
VisionOS is developed using Xcode, an integrated development environment provided by Apple.

・Swift UI
'SwiftUI is the best way to build new visionOS apps or bring existing iPadOS or iOS apps to the platform,' Apple said. SwiftUI works with RealityKit and UIKit to support the building of sharp, responsive and voluminous application UIs for VisionOS.

・Reality Kit
RealityKit, Apple's 3D rendering engine, enables applications to present 3D content, animations, and visual effects. With RealityKit, it is possible to do various things such as 'making shadows by automatically adjusting lighting conditions', 'opening portals to other worlds', and 'building overwhelming visual effects'. RealityKit also uses MaterialX to export materials.

・ARKit
ARKit by default provides core system functionality for apps in shared spaces. And when your app moves to full space and asks for permission, you can take advantage of powerful ARKit APIs such as plane estimation, scene reconstruction, image coordinate fixing, ambient tracking, and hand tracking. Therefore, it will be possible to ``sprinkle water on the wall'' and ``bounce the ball on the floor''.

・Accessibility
Vision Pro is equipped with gaze tracking and voice recognition functions, and it is possible to operate the device with eyes, voice, or both. In addition, hand recognition by an external camera allows you to use your index finger or wrist as a mouse. They were designed with accessibility in mind, just like the iPhone and Mac, so you can develop accessible apps for visionOS and make Vision Pro a great experience for everyone. said Apple.

・Reality Composer Pro
Reality Composer is a tool for building, testing, tuning and simulating AR experiences for iPhone and iPad. Based on this Reality Composer, Reality Composer Pro is newly designed to preview 3D content for visionOS apps. Reality Composer Pro is available in Xcode and integrates with XCode's build process to preview and optimize visionOS assets. It is also useful for importing and organizing 3D models, sounds, etc.

・Unity
By using the game development platform Unity, it is now possible to port games developed in Unity to visionOS, and to develop new apps and games for visionOS in Unity. Not only the AR experience, but also the 'pass-through function' that displays the surrounding environment as it is, and the 'Dynamically Foveated Rendering' that tracks the user's line of sight and draws only the central part of the field of view at high resolution can be applied to games. By combining application rendering with RealityKit, content created with Unity can run on visionOS without any problems.

The visionOS SDK, a development kit for visionOS, will be released in late June 2023, along with new Xcode, visionOS simulator, Reality Composer Pro, detailed documentation, sample code and design guidance.