Four-Dimensional Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNNs)

Yu Zhao, Xiang Li, Heng Huang, Wei Zhang, Shijie Zhao, Milad Makkie, Mo Zhang, Quanzheng Li, Tianming Liu

Research output: Contribution to journalArticlepeer-review

23 Scopus citations


Since the human brain functional mechanism has been enabled for investigation by the functional magnetic resonance imaging (fMRI) technology, simultaneous modeling of both the spatial and temporal patterns of brain functional networks from 4-D fMRI data has been a fundamental but still challenging research topic for neuroimaging and medical image analysis fields. Currently, general linear model (GLM), independent component analysis (ICA), sparse dictionary learning, and recently deep learning models, are major methods for fMRI data analysis in either spatial or temporal domains, but there are few joint spatial-Temporal methods proposed, as far as we know. As a result, the 4-D nature of fMRI data has not been effectively investigated due to this methodological gap. The recent success of deep learning applications for functional brain decoding and encoding greatly inspired us in this paper to propose a novel framework called spatio-Temporal convolutional neural network (ST-CNN) to extract both spatial and temporal characteristics from targeted networks jointly and automatically identify of functional networks. The identification of default mode network (DMN) from fMRI data was used for evaluation of the proposed framework. Results show that only training the framework on one fMRI data set is sufficiently generalizable to identify the DMN from different data sets of different cognitive tasks and resting state. Further investigation of the results shows that the joint-learning scheme can capture the intrinsic relationship between the spatial and temporal characteristics of DMN and thus it ensures the accurate identification of DMN from independent data sets. The ST-CNN model brings new tools and insights for fMRI analysis in cognitive and clinical neuroscience studies.

Original languageEnglish (US)
Article number8713897
Pages (from-to)451-460
Number of pages10
JournalIEEE Transactions on Cognitive and Developmental Systems
Issue number3
StatePublished - Sep 2020
Externally publishedYes


  • Deep learning
  • functional brain networks
  • functional magnetic resonance imaging (fMRI)

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Four-Dimensional Modeling of fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNNs)'. Together they form a unique fingerprint.

Cite this