EXPLORING THE ROLE OF IOT IN HUMAN-CENTRIC MIXED REALITY ENVIRONMENT: APPLICATIONS AND DESIGN CONSIDERATIONS

With advances in hardware and mobile computing capabilities, Mixed Reality (MR), which encompasses technologies from Augmented Reality (AR) to Virtual Reality (VR), has evolved from a laboratory concept into a mainstream product, as seen with popular releases like the Oculus Quest and Vision Pro. MR combines elements of both physical and virtual environments, enabling interactions and visualizations that transcend the limitations of the physical world. In this thesis, we aim to move beyond the conventional approach of treating MR as a standalone display medium. Specifically, we leverage advancements in IoT (internet of things) technology to fully integrate them into the interaction design.

On a broader level, our work in this thesis focuses on novel IoT design and integration, exploring how they can expand interaction possibilities in MR. This thesis is based on our four published conference papers (MechARspace, ColabAR, InstruMentAR, and LearnIoTVR), which explore the symbiosis between IoT and MR in different dimensions. In particular, we design IoT-enabled MR experiences to 1) promote social presence during remote collaboration, 2) enrich playfulness during the interaction, 3) streamline the in-situ creation of instrument tutorials, and 4) foster STEM education at minimum cost. Throughout our work, we focus on making novel interactions that improve user experiences while addressing real-world challenges. Additionally, we focus on designing with cost-effectiveness and usability in mind, aiming to lower entry barriers. Our goal is to allow end-users to adopt and engage with our system effortlessly.

Takeaways from the research series include 1) The usage of IoT modules, with their sensing and actuating capabilities, significantly enriches interaction modalities compared to traditional display-only MR interactions, 2) IoT, with its universal connectivity, enables spontaneous interactions commonly encountered in daily life, and 3) Mixed Reality, with its capability to generate immersive 3D experiences, can create a realistic digital replica of real-world IoT environments in a flexible and cost-effective way, supporting a wide range of applications.