Exploring the Use of Smartphones as Input Devices for the Mixed Reality Environment

Nowadays, researches on mixed reality (MR) have made a lot of exploration in the aspects of user experience, hardware devices, interaction technologies, application systems, etc. However, more research is still needed to explore how to improve the experience in the shared MR Environment and design appropriate collaborative interaction models. The traditional VR handle can realize tool simulation and 3D interaction, while smartphones can realize 2D interactions such as fast 2D gestures, text input and handwriting. Using the mobile phone as the handle of the MR headset can realize the complementary advantages of the two. In this study, we design a cross-device collaborative system sharing a hybrid reality environment. In the MR Environment, the smartphone will realize the functions of the controller for remote selection and manipulation of objects, and its advantages in 2D interaction can be well applied to some special tasks. We design the user interface with three levels of immersion: high, medium and low. High immersion: We provide a hybrid user interface combining a HoloLens2 and a smartphone. Medium immersion: We provide a 3D user interface represented by HoloLens2. Low immersion: We provide a multi-touch user interface represented by tablet. User studies show that the hybrid user interfaces can bring users satisfactory immersion and interactive experience, but it also needs to design accurate and efficient input methods according to the interactive tasks.