Next Generation XR Systems-Large Language Models Meet Augmented and Virtual Reality

Muhamamd Zeshan Afzal,Sk Aziz Ali,Didier Stricker,Peter Eisert,Anna Hilsmann,Daniel Perez-Marcos, Marco Bianchi,Sonia Crottaz-Herbette, Roberto De Ioris,Eleni Mangina, Mirco Sanguineti, Ander Salaberria,Oier Lopez de Lacalle,Aitor Garcia-Pablos,Montse Cuadros

IEEE Computer Graphics and Applications（2025）SCI 4区

DFKI

Cited 0|Views5

Abstract

Extended reality (XR) is evolving rapidly, offering new paradigms for human-computer interaction. This position paper argues that integrating large language models (LLMs) with XR systems represents a fundamental shift toward more intelligent, context-aware, and adaptive mixed-reality experiences. We propose a structured framework built on three key pillars: first, perception and situational awareness, second, knowledge modeling and reasoning, and third, visualization and interaction. We believe leveraging LLMs within XR environments enables enhanced situational awareness, real-time knowledge retrieval, and dynamic user interaction, surpassing traditional XR capabilities. We highlight the potential of this integration in neurorehabilitation, safety training, and architectural design while underscoring ethical considerations, such as privacy, transparency, and inclusivity. This vision aims to spark discussion and drive research toward more intelligent, human-centric XR systems.

Translated text

求助PDF

上传PDF

Bibtex

AI Read Science

AI Summary

AI Summary is the key point extracted automatically understanding the full text of the paper, including the background, methods, results, conclusions, icons and other key content, so that you can get the outline of the paper at a glance.

Example

Background

Key content

Introduction

Methods

Results

Related work

Fund

Key content

Pretraining has recently greatly promoted the development of natural language processing (NLP)
We show that M6 outperforms the baselines in multimodal downstream tasks, and the large M6 with 10 parameters can reach a better performance
We propose a method called M6 that is able to process information of multiple modalities and perform both single-modal and cross-modal understanding and generation
The model is scaled to large model with 10 billion parameters with sophisticated deployment, and the 10 -parameter M6-large is the largest pretrained model in Chinese
Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities We will continue the pretraining of extremely large models by increasing data to explore the limit of its performance

Upload PDF to Generate Summary

Must-Reading Tree

Example

Generate MRT to find the research sequence of this paper

Data Disclaimer

The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn

Chat Paper

【要点】：本文提出将大型语言模型（LLMs）与扩展现实（XR）系统结合，以实现更加智能、情境感知和自适应的混合现实体验，并探讨了其在神经康复、安全培训和建筑设计等领域的潜在应用及伦理考量。

【方法】：文章构建了一个基于三个关键支柱的结构化框架，分别是感知与情境认知、知识建模与推理、可视化与交互。

【实验】：未具体描述实验过程，但提出研究愿景，并呼吁针对更加智能、以人为本的XR系统开展讨论和研究；未提及使用的数据集名称。