MinD-3D: Reconstruct High-quality 3D objects in Human Brain
CoRR(2023)
摘要
In this paper, we introduce Recon3DMind, a groundbreaking task focused on
reconstructing 3D visuals from Functional Magnetic Resonance Imaging (fMRI)
signals. This represents a major step forward in cognitive neuroscience and
computer vision. To support this task, we present the fMRI-Shape dataset,
utilizing 360-degree view videos of 3D objects for comprehensive fMRI signal
capture. Containing 55 categories of common objects from daily life, this
dataset will bolster future research endeavors. We also propose MinD-3D, a
novel and effective three-stage framework that decodes and reconstructs the
brain's 3D visual information from fMRI signals. This method starts by
extracting and aggregating features from fMRI frames using a neuro-fusion
encoder, then employs a feature bridge diffusion model to generate
corresponding visual features, and ultimately recovers the 3D object through a
generative transformer decoder. Our experiments demonstrate that this method
effectively extracts features that are valid and highly correlated with visual
regions of interest (ROIs) in fMRI signals. Notably, it not only reconstructs
3D objects with high semantic relevance and spatial similarity but also
significantly deepens our understanding of the human brain's 3D visual
processing capabilities. Project page at: https://jianxgao.github.io/MinD-3D.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要