1.Introduction

Schizophrenia is a psychiatric disorder with characteristic symptoms such as delusions, hallucinations, disorganized speech, grossly disorganized or catatonic behavior, and negative symptoms as described in DSM-V [1]. The average age of onset is from late teens to early 30s, and tends to be earlier in males (from late teens to early 20s), than in females (from late 20s to early 30s) [2-3]. Up to 2023, there are an estimated 24 million people in the world who are diagnosed with schizophrenia [4-8].

Traditional ways of diagnosis rely on identifying two or more symptoms that last for at least one month and eliminating possibilities of other medical conditions such as substance-induced psychosis or depressive and bipolar disorder. The current diagnostic process poses several issues preventing patients from receiving adequate help as early as possible. When lasting, observable symptoms have already arisen, the disorder is often in an advanced stage, making treatment difficult or ineffective. Additionally, clinical diagnosis is prone to error due to the similarity of many schizophrenia symptoms with those of other disorders, and it is also prone to discrimination, especially that against females, blacks, and individuals with learning disabilities. Therefore, we aim to use machine learning to perform schizophrenia diagnosis using neurocognitive and behavioral measures involved in traditional diagnosis to provide earlier and faster diagnosis and build the foundation for risk factor assessment. Behaviorally, schizophrenia affects the normal functioning of many cognitive abilities, particularly working memory. Working memory falls under short-term memory and is a psychological and cognitive construct that provides temporary storage for information needed to perform complex cognitive tasks such as reasoning, learning, and language comprehension.

In recent years, magnetic resonance imaging (MRI) techniques have been applied to the diagnosis of schizophrenia, with past research revealing the relevance of total brain volume decrease, ventricular enlargement, reduction in hippocampal and thalamic volumes, enlargement in globus pallidus volume, and changes to the neocortical temporal lobe regions [10]. In addition, abnormalities in cortical thickness, and gray and white matter distribution [10-11], as well as changes in connectivity among brain networks and systems, are associated with schizophrenia [12].

To analyze brain imaging data that reflect brain structural changes associated with disorders, researchers have trained machine learning algorithms to analyze MR images of schizophrenia patients versus controls. Examples of commonly used machine learning algorithms include Random Forest (RF), Support Vector Machine (SVM), Ridge, Lasso, Gradient boosting, and logistic regression. Other studies have also used deep learning and convolutional neural networks. This study uses an open dataset and applies linear SVM, RF, multilayer perceptron (MLP), and k-Nearest Neighbor (kNN) to classify participants as schizophrenic or healthy based on demographic, cognitive, brain anatomical, and functional MRI data, and find important features in schizophrenia pathology. Our model expands from preexisting models in that it uses demographic and clinical data of patients that would be incorporated in clinical diagnosis currently but have not been in previous classifiers. We also aim to compare the performance of four classifiers when trained with different combinations of data. This will tell us whether structural or functional data contribute more to accurate diagnosis for each classifier.

2.Materials and Methods

2.1.Participants

We used an open dataset named “Working Memory in Healthy and Schizophrenic Individuals” on openneuro.com [14]. Participants in the study were recruited at Washington University School of Medicine in St. Louis. The open dataset consists of 99 participants (40 female, 59 male), including 23 participants diagnosed with Schizophrenia based on semi-structured interview and Structured Clinical Interview for DSM-IV Axis I Disorders [15]. One participant (sub-03) was removed from the analyses due to erroneous and incomplete measures. Demographic information on study participants can be found in Table S1, and more details on participation eligibility criteria can be found in [16].

2.2.Clinical and Cognitive Measures

A series of clinical and cognitive measures were collected from each participant. The clinical measures were based on the Scale for the Assessment of Positive Symptoms (SAPS) and Scale for the Assessment of Negative Symptoms (SANS), whose sub-scores were grouped to yield measures on three composite scores: positive symptom domain, negative symptom domain, and disorganization symptom domain. Details on the computation of these composite scores can be found in the Supplemental Materials of Repovš and Barch (2012). Cognitive abilities were assessed using subscales from the Wechsler Adult Intelligence Scale, Wechsler Abbreviated Scale of Intelligence, Wechsler Memory Scale–Third Edition [17], N-back task [18], Continuous Performance Task, California Verbal Learning Test [19], Trails B [20], category and verbal fluency tasks [21], Wisconsin Card Sort [22]. These sub-scores were grouped and combined into four composite scores, reflecting participants’ intelligence, working memory, episodic memory, and executive function. Summary statistics of these clinical and cognitive composite scores are shown in Table S1.

2.3.N-back Working Memory Task

All participants performed three N-back tasks, in separate runs of functional MRI scanning, with different levels of demand on their working memory load. The order of these tasks was counterbalanced across participants. In the 0-back task, participants responded to whether the letter shown on the screen was the same as a pre-specified letter (“A” and “X”), where there was little working memory load. In the 1-back task with intermediate working memory load, participants responded to whether the current letter was the same as the immediately preceding letter. In the 2-back task, participants responded to whether the current letter was the same as the letter shown 2 trials before, and thus had the highest working memory load. However, in the OpenNeuro dataset, all trials were coded as “non-target” perhaps due to some coding error. We were therefore not able to determine the actual type of individual experimental trials, i.e., whether the participant should have responded or not. As such, both the behavioral and fMRI analyses were conducted at the condition level, e.g., mean accuracy and fMRI activity during each N-back task for each participant.

2.4.MRI Scanning

Magnetic resonance imaging (MRI) is divided into the subtypes structural MRI (sMRI) and functional MRI (fMRI), with sMRI mainly used to record structural brain information while subjects are not completing a specific cognitive task and fMRI mainly used to record brain regional activity while subjects complete cognitive activities [23]. sMRI is based on translating local differences in water content in the brain into a gray scale through which the outline of the patient’s brain can be created. Structural magnetic resonance images provide information regarding the shape, size, and some regional information about the brain. From the resulting images, abnormal tissues with changes in tissue density or composition as well as overall and regional brain structural changes can be detected and analyzed, and this information used in disease diagnosis [24].

fMRI works by measuring small changes in the magnetic field strength caused by paramagnetic deoxyhemoglobin in the blood vessels. Changes in the concentration of deoxyhemoglobin reflect regional consumption of oxygen, which is an indication of local metabolic and neural activity. fMRI has been widely applied to clinical trials such as disease detection and diagnosis, surgical planning, monitoring treatment outcomes, and searching for biomarkers in pharmacologic and training programs.

2.4.1.Scanning Metrics. Scanning was completed at Washington University Medical School using a 3T Tim TRIO Scanner. Structural images were acquired at a spatial resolution (voxel size) of 1 mm × 1 mm × 1 mm. Functional images of the working memory task were acquired in 3 runs of 17.1 min of scanning (13.1 min of task performance) with a repetition time (TR) of 2.5 seconds and a spatial resolution (voxel size) of 4 mm × 4 mm × 4 mm. Each participant completed three functional runs of the working memory tasks. More details can be found in the published study [16].

2.5.sMRI Preprocessing and Analysis

2.5.1.Preprocessing. sMRI data preprocessing was run on MATLAB R2023a with the toolboxes SPM12 and CAT12. The preprocessing process included brain extraction, spatial normalization (to a template brain), and tissue segmentation (gray matter, white matter, and cerebrospinal fluid).

2.5.2.Structural Regions of interest (SROIs). We conducted a literature review of schizophrenia-related or -diagnostic features in brain anatomy. Typical brain anatomical changes include ventricle enlargement, gyrus volume reduction, and gray matter volume and density reduction distributed over several regions, including amygdala, thalamus, hippocampus, and insula [11, 16, 25-47]. We used Neuromorphometrics parcellation of the brain to extract relevant volumetric features (see Table S2 in Supplementary Materials) [48].

2.6.fMRI Preprocessing and Analysis

2.6.1.Preprocessing. fMRI preprocessing was done by the original authors using fMRIPrep 21.0.1 [49]. fMRI preprocessing procedures include realigning functional images to a reference image to correct for head motion across time, co-registering functional images to the participant’s own structural MRI image for better spatial alignment, resampling the functional images into a standard template brain space (MNI 152) for ease of comparison, and spatially smoothing the signal with a Gaussian kernel of 6mm full-width half-maximum to improve signal-to-noise ratio. More details on fMRI preprocessing can be found in the fMRIPrep-generated methods description from “Working Memory in Healthy and Schizophrenic Individuals”.

2.6.2.First-level analysis. To separate behavior-related signals from noise, we conducted a first-level analysis [50] using the SPM12 toolbox in MATLAB. A general linear model (GLM) with a regressor that indicates the expected hemodynamic response of individual experimental trials, by convolving the task duration with the canonical hemodynamic response function [50], was fitted to the preprocessed BOLD signal time series. Nuisance regressors of head motion and signal in the white matter and cerebrospinal fluid were also included in the GLM to remove their artifacts. The GLMs yield a regression coefficient (beta) that reflects how closely the observed BOLD signal mirrored the expected activity, thereby providing us a measure of neural activity. We obtained the neural activity estimates for each of the 3 working memory tasks for each participant.

2.6.3.Functional Regions of Interest. To identify the brain regions implicated in working memory, we referred to a meta-analysis of 46 existing fMRI studies on working memory in younger adults (age range comparable to our sample) [46]. The meta-analysis identified multiple regions in the dorsal frontal cortex, inferior parietal lobule, and precuneus (see Figure 2 and Table 3 of [46]). A term search of “working memory” on Neurosynth, an automated online meta-analysis, yielded very similar activation patterns across the brain [51]. We parcellated the brain using the Automated Anatomical Labeling 3 (AAL3; [43]) atlas and selected relevant parcels (see Table S3 for functional regions of interest and Table 1 below for all features used in classification, including behavioral, sMRI, and fMRI data.)

Table 1. Behavioral, structural, and functional features used in classification

* If the left or right side is unspecified, then the whole structure is considered.

2.7.Classification Setup

Using the scikit-learn package in Python [52], we implemented four machine learning classification methods: linear SVM, RF, MLP, and kNN. All algorithms were trained to perform a binary classification task: Control vs. Schizophrenic. Linear SVM segregates a given dataset into multiple categories with a single straight line. RF is a random composition of various decision trees through which the composition fits various sub-samples of a dataset and uses averaging to improve predictive accuracy and control over-fitting. kNN classifies new data points based on the classification of the k nearest data points to the given data points. We compared the predictive accuracy and feature importance for each classifier.

2.7.1.Feature Preprocessing. Data cleaning was done on all subjects before building the classifiers. Missing data were imputed with median values. Global standardization of features was done across all participants.

2.7.2.Cross-validation. We employed 100-fold cross-validation. All classification results reported below are the averaged performance from 100 iterations.

2.7.3.Train-Test Split. For each iteration, the dataset was randomly split into a training set and a test set, with the requirement that each test set contained 5 CON and 5 SCZ, leaving 70 CON + 18 SCZ for classifier training. All numeric columns in the training set were standardized (mean=0, sd=1), and the same transformation was applied to the test set.

2.7.4.Data Augmentation. Data augmentation was performed to match the number of SCZ and CON subjects in the training set to resolve class imbalance, which could lead to classifier bias. We made a total of 53 copies of existing SCZ subjects in the training set and added Gaussian noise (mean=0, sd=0.2) to all of their numeric features, such that the final training set contained 71 observations of each class.

3.Results

3.1.Behavioral Analysis

We first conducted behavioral analysis on the demographics, psychopathology, and neurocognitive measures of the patients. This is to rule out any confounds such as educational level and learning abilities that might affect their working memory performance and to ascertain whether the control and the schizophrenic group are similar in terms of age, gender, and racial makeup. We found that there are no statistically significant differences in these measures between the two groups. Additionally, psychopathology and neurocognitive measures validate clinical diagnosis of the subjects because there are statistical differences in performance scores between the two groups. More details can be found in Table 2.

Table 2. Behavioral Analysis of CON vs. SCZ individuals

Through our behavioral analysis, we found that demographics were statistically similar for both groups and thus did not pose as a confound. Psychopathology tests confirmed the average accuracy of clinical schizophrenia diagnosis. Schizophrenia patients also showed working memory impairment, though there was no statistical difference between executive function between the two groups.

3.2.N-back Task Performance

We compared the letter n-back task target performance between the two groups and found that there are statistically significant differences for the n-back 1 and 2 tasks but not for the n-back 0 task. This makes sense because the n-back 0 task does not use working memory whereas the second 2 tasks do. This shows that working memory is indeed impaired in schizophrenic individuals. See more details in Table 3 below.

3.3.Classifier Accuracy

We trained all four classifiers with the following three combinations of data types and recorded the average accuracy of 100 iterations: a) behavioral + structural; b) behavioral + functional; c) behavioral + structural + functional.

Behavioral data included psychopathology and neurocognitive measures because two sample t tests have shown that there are statistical differences between the two group’s scores in these areas, as well as subject performance in the letter n-back tasks. Gender was also considered as a factor in classifier training since there is a slight gender imbalance in our samples and because it is considered in clinical diagnosis as well. Structural data included anatomical metrics of regions of interest recorded in structural MRI images of the subjects. These metrics were purely structural such as volume, thickness, matter distribution, and shape because these were recorded when subjects were at rest. When accompanied by psychopathology and neurocognitive measures, working memory task performance was an accurate indicator of each subject’s cognitive abilities, especially, their working memory functioning.

The purpose of comparing classifier accuracy across multiple types of training data was to investigate which type(s) of data contributed more to classifier decisions and thus can be used to provide more accurate diagnosis. We found that combinations including functional data usually yielded higher accuracies across multiple classifiers.

We also compared diagnostic accuracy across the four classifiers to examine which one works best when trained with the data available. This suggests to future researchers which algorithm(s) might work better. We found that linear SVM has the best performance on average and achieved our highest accuracy of 0.75 when trained with behavioral and functional data. More information regarding our classifier performance can be found in Table 3 and Figure 1.

Table 3. Comparison of Algorithmic Accuracy based on Structural vs. Functional Data (100 iterations)

Figure 1. Accuracy by classifier

We plotted feature importance to identify the features contributing the most to algorithmic decisions for three classifiers–linear SVM, random forest, and kNN under different data type conditions. We found that there are many overlaps in the top ten features of importance for the three classifiers, which means that the brain regions associated with working memory functioning and other schizophrenia symptoms are largely universal across different algorithms. The result informs physicians of features that should be paid extra attention to during diagnosis. More information about feature importance by classifier can be found in Figure 2.

Figure 2. Selected features of importance by classifier including all data types.

4.Discussion

In this study, we demonstrated the possibility of using objective, general-purpose measures of behavior and the brain to diagnose schizophrenia. These behavioral and brain measures are relatively easy to acquire and therefore the implementation may be fairly accessible to the general public. As such, this set of procedures has the potential to aid the diagnosis of schizophrenia early on, when the patients and family experience little symptoms and would not have thought about consulting with psychiatrists.

This study is novel in its comprehensive nature: we both incorporated additional information in training classifiers and compared performance across multiple classifiers. Past research has explored the possibility of using either functional or structural MRI data to detect schizophrenia [53-56], but little research has also included patient background information as would be in a clinical setting. Neither has a lot of research been done to compare the effectiveness of using functional versus structural MRI images in diagnosis, but we found that functional data leads to better classifier performance while the performance based on functional versus both functional and structural data are more similar. This also indicates that functional data such as fMRI while completing working memory tasks are valuable in clinical diagnosis as well and physicians can consider weighing them more than structural data. The focus on working memory performance itself is relatively novel in the field of machine learning diagnosis of schizophrenia. Additionally, we have only noted one research comparing classifier performance in prediction of hospitalized patients with schizophrenia or other mental health disorders [57]. While covering many classifiers in their research, all of their subjects are either diagnosed with schizophrenia or other mental health disorders. We improved in that area by selecting subjects with or without schizophrenia, which more closely resembles the population in clinical settings.

The results from this study are limited in its small sample size and the number of neurocognitive measures and regions of interest used to train our algorithms. Therefore, we suggest three possible directions for future research. First, we suggest the use of larger sample sizes, when possible, for better classifier training. Since the number of subjects from the public dataset was relatively small, and there was a heavy imbalance of schizophrenic to healthy individuals due to the nature of the previous study, we trained our classifiers using augmented data. However, it will be desirable to use more real subject data both of schizophrenic and healthy individuals for classifier training. Secondly, future studies can try additional behavioral measures, working memory tasks, or other neurocognitive measures to see whether those yield higher diagnosis accuracy. Our results suggest that classification based on neurocognitive measures and fMRI data seems to give higher accuracy, which can help inform future research in choosing their features. Finally, future studies can find a collection of behavioral and neural measures that can be applied to the diagnosis of multiple mental health, neurodegenerative, or psychiatric disorders, such that the utility of the procedures is not limited to schizophrenia.

5.Conclusion

In this study we trained Linear SVM, RF, MLP, and kNN classifiers to diagnose patients as schizophrenic or healthy based on neurocognitive, structural, and functional data. We trained the four classifiers using combinations of afore-mentioned neurocognitive measures and structural or functional MRI data to compare which combination yields higher accuracy. To eliminate confounds in these data, we first conducted a behavioral analysis which yielded that there are no statistical differences in demographic measures between the control and schizophrenic group and the latter does show psychopathology and neurocognitive scores corroborating their clinical diagnosis. In terms of classifier performance, we found that behavioral and functional measures might yield a higher accuracy if linear SVM is used, but that the best combination differs from classifier to classifier. Overall, we found a higher than chance diagnostic accuracy for all classifiers used, with the highest being 75% accuracy using linear SVM based on neurocognitive and functional MRI data. This suggests that algorithms can be effectively used in schizophrenia diagnosis in addition to traditional methods though the technology is not mature enough to replace the traditional methods.

6.Acknowledgements

I would like to thank Dr. Huang from the Department of Psychology and Neuroscience at Duke University for guiding my research, revising codes, and providing editing feedback to this manuscript. This research would not be possible without your support. I would also like to acknowledge OpenNeuro for providing public online datasets on which this research is based.

7.Supplementary materials

7.1.Demographic, clinical, and cognitive measure statistical analysis

We calculated the mean and standard deviation of demographic, clinical and cognitive measures of the control and schizophrenic group to compare the differences between them. Details about these statistics can be found in Table S1.

Table S1. Demographic, clinical, and cognitive measures

Of the 101 participants (99 presented in public dataset) aged 18 or older from whom the original authors collected resting state data, there were 12 who were excluded for poor quality imaging data (4 SCZ, 8 CON), and 15 who were excluded because they did not have N-back data (6 SCZ, 9 CON). Data for subject-03, specifically, had to be discarded in our statistical calculations due to outliers in columns sans17, sans22, sans25, d4prime, LFLUNOVS, and CFLUNOVA, and no data for the remaining neurocognitive and working memory assessments. These outliers are likely due to excessive movement and failure to complete the whole protocol.

7.2.Regions of Interest

We selected structural and functional regions of interest as recorded in Table S2 and Table S3. Both categories were selected based on literature review and all regions of interest were used to train classifiers. Structural regions are given in terms of their neuromorphometrics atlas label. Functional regions are defined based on the Automated Anatomical Labelling atlas 3 (AAL3).

Table S2. Structure MRI Regions of Interest and Correlating AAL3 Labels

Table S3. Functional Regions of Interest and CorrespondingAAL3 labels