深度學習為腦成像分析提供了一種強有力的方法(LeVillage 等，2015; Cole 等，2017; Farooq 等，2017; Kamnitsas 等，2017; Shen 等，2017; Mohsen 等，2018)。深度卷積神經網絡(CNN)(Simonyan and Zisserman 2014; He et al。2016; Huang et al。2017; Krizhevsky et al。2017)具有更好的表現能力，可以自動提取低層到高層的空間特征(Gu et al。2018; Abrol et al。2021) ，通常優于需要手工制作特征的傳統機器學習方法。受到醫學影像分析中深度學習的巨大成功的啟發(Shen 等，2017; Kermany 等，2018; Sevakula 等，2018; Ding 等，2018; Zhu 等，2020) ，神經影像學研究人員更加關注深度學習。一些研究使用3D CNN 來分析3D 腦圖像，因為大腦是3D 的(Valliani 和 Soni 2017; Oh 等，2019; Bashyam 等，2020; Thomas 等，2020)。三維模型可以保存大腦的空間信息，并在個體水平上捕捉特征。然而，它們有太多的參數需要訓練，導致收斂困難。由于包含空值的體積計算無效，計算效率很低。此外，模型的良好調整需要大量的數據，但大腦成像數據的收集是昂貴的和耗時的。
In recent years, transfer learning has emerged as a crucial method for solving the insufficient training data problem by transferring knowledge from a source domain to a target domain (Pan and Yang 2009; Tan etal. 2018). The models are usually pretrained on a large-scale source dataset (e.g., ImageNet) (Deng etal. 2009) and then fine-tuned on the small target dataset in deep learning (Tan etal. 2018). The wealth of knowledge learned from the source dataset is implicitly encoded in huge parameters, making it possible for the target task to achieve better performance with limited samples. For brain imaging analysis, the surface-based cortical shape morphometry is closely related to sex, age, and neuropsychiatric disorders (Yuan etal. 2015; Bedford etal. 2020; Gharehgazlou etal. 2021). Many medical imaging studies use large-scale natural image datasets as the source domain and achieve better performance (Kermany etal. 2018; Sevakula etal. 2018; Ting etal. 2018; Zhu etal. 2020), indicating the feasibility of transfer learning from natural images to brain imaging data. However, most of the current pretrained models are designed for 2D planar images and cannot be directly applied to 3D brain magnetic resonance imaging (MRI). How to bridge the 3D brain images and 2D pretrained models remains unsolved.
(資料圖片僅供參考)
近年來，通過將知識從源領域轉移到目標領域，遷移學習已經成為解決訓練數據不足問題的關鍵方法(Pan and Yang 2009; Tan et al。2018)。模型通常在大規模源數據集(例如 ImageNet)上進行預訓練(Deng et al。2009) ，然后在深度學習中對小目標數據集進行微調(Tan et al。2018)。從源數據集中學到的大量知識隱式地編碼在巨大的參數中，這使得目標任務可以在有限的樣本下實現更好的性能。對于腦成像分析，基于表面的皮層形態測量與性別，年齡和神經精神障礙密切相關(Yuan 等，2015; Bedford 等，2020; Gharehgazlou 等，2021)。許多醫學成像研究使用大規模的自然圖像數據集作為源域，并獲得更好的性能(Kermany 等，2018; Sevakula 等，2018; Ding 等，2018; Zhu 等，2020) ，表明將學習從自然圖像轉移到腦成像數據的可行性。然而，目前大多數預先訓練的模型是為二維平面圖像設計的，不能直接應用于三維腦磁共振成像(MRI)。如何將三維腦圖像和二維預訓練模型連接起來仍然是一個未解決的問題。
Most MRI studies based on CNN models use Euclidean distance as the metric for brain imaging analysis, ignoring the fact that the human brain has complex structures with folded sulcus and gyri (Fischl etal. 2008; Zhang etal. 2020). Nevertheless, the Euclidean distance is not the best metric of the brain images due to the non-Euclidean geometry of the complicated folding of the cerebral cortex (Seong etal. 2018). Treating brain images as ordinary images and applying Euclidean distance-based deep models directly to the brain images, which lack the neurobiological basis, may lead to the signal mixture of different brain regions and destruction of topological structure (Glasser etal. 2013). On the contrary, the distance along the brain surface is more consistent with neurobiology and cerebral cortex geometry (Fischl 2012; Glasser etal. 2013; Honnorat etal. 2015). For the aforementioned reasons, it is more reasonable to adopt the distance along the surface of the cerebral cortex instead of Euclidean distance in CNN models to obtain results with neurobiological significance.
大多數基于 CNN 模型的 MRI 研究使用歐幾里得度量作為大腦成像分析的指標，忽略了人類大腦具有復雜結構和折疊的溝和回的事實(Fischl 等，2008; Zhang 等，2020)。盡管如此，由于大腦皮層復雜折疊的歐幾里得度量非歐幾里得幾何(Seong et al。將腦圖像作為普通圖像處理，并將基于歐幾里得距離的深度模型直接應用于缺乏神經生物學基礎的腦圖像，可能導致不同腦區的信號混合和拓撲結構的破壞(Glasser 等，2013)。相反，沿著腦表面的距離與神經生物學和大腦皮層幾何形狀更一致(Fischl 2012; Glasser 等，2013; Honnorat 等，2015)。基于上述原因，采用大腦皮層表面的距離代替 CNN 模型中的歐幾里得度量來獲得具有神經生物學意義的結果更為合理。
To address these issues, we propose a novel framework to bridge the gap between 3D MRI data and 2D CNN models by mapping the 3D cerebral cortex into 2D images and utilizing transfer learning to improve network performance. The proposed framework can be roughly divided into 4 steps. The first step is to process the cortical data with FreeSurfer and transform the cerebral cortex into 3D surface meshes and vertex-wise cortical shape metrics (Glasser etal. 2013). The 3D surface meshes are then topologically and equally mapped into 2D planar meshes through an area-preserving geometry mapping approach (Zhao etal. 2013) and further converted into 2D images for the subsequent analysis. The converted images reflect the distance along the brain surface of different brain regions, and the convolution on the converted images is similar to the convolution along the cortical surface, which is more consistent with neurological significance. The third step is to train the models using transfer learning. We choose the pretrained ResNet-50 (He etal. 2016) and DenseNet-121 (Huang etal. 2017) as the backbone networks. The pretrained models are fine-tuned with the acquired 2D images. Finally, the results from different metrics are ensembled using the stacking ensemble method (Wolpert 1992) to generate final individual-level classification results. The effectiveness of the proposed method is demonstrated with sex classification.
為了解決這些問題，我們提出了一種新的框架，通過將3D 大腦皮層映射成2D 圖像，并利用傳遞學習來提高網絡性能，從而在3D MRI 數據和2D CNN 模型之間架起一座橋梁。提出的框架大致可分為4個步驟。第一步是用 FreeSurfer 處理皮層數據，并將大腦皮層轉換成3D 表面網格和頂點形狀指標(Glasser et al。2013)。然后通過面積保持幾何映射方法(Zhao et al。2013)將3D 表面網格拓撲和等量地映射到2D 平面網格中，并進一步轉換成2D 圖像以用于隨后的分析。轉換后的圖像反映了不同腦區沿腦表面的距離，轉換后圖像的卷積與沿皮質表面的卷積相似，更符合神經學意義。第三步是利用遷移學習對模型進行訓練。我們選擇預先訓練的 ResNet-50(He et al. 2016)和 DenseNet-121(Huang et al. 2017)作為骨干網絡。預訓練模型與獲得的二維圖像進行了微調。最后，使用疊加集成方法(Wolpert 1992)對不同度量的結果進行集成，生成最終的個體層次分類結果。性別分類驗證了該方法的有效性。
Moreover, previous studies have reported sex differences of structures and functions in autism spectrum disorder (ASD) (Bejerot etal. 2012; Loomes etal. 2017; Bedford etal. 2020; Liu etal. 2020). A reasonable assumption is that the sex-related features may be helpful in ASD classification. Thus, we further develop a 2-stage transfer learning framework for the classification of ASD by using the sex classification of healthy people as an intermediate task to reduce the distribution differences of the source domain and the target domain for better performance.
此外，先前的研究報道了自閉癥光譜結構和功能的性別差異(Bejerot 等，2012; Loomes 等，2017; Bedford 等，2020; Liu 等，2020)。一個合理的假設是，性別相關特征可能有助于 ASD 的分類。因此，我們進一步發展了一個兩階段的遷移學習框架，以健康人的性別分類為中間任務，以減少源域和目標域的分布差異，以獲得更好的表現。
The contributions of this paper can be summarized as follows:
本文件的貢獻可概述如下:
A novel framework is proposed to bridge the gap between 3D MRI data and 2D CNN models.
提出了一種新的框架來彌合三維 MRI 數據和二維 CNN 模型之間的差距。
We demonstrate the effectiveness of transfer learning in MRI studies under our framework.
在我們的框架下，我們證明了磁共振成像研究中遷移學習的有效性。
We introduce a 2-stage transfer learning method for brain imaging analysis and demonstrate that the sex classification of healthy people could be used as an intermediate task to improve the ASD classification performance.
介紹了一種用于腦成像分析的兩階段遷移學習方法，并論證了健康人的性別分類可以作為提高 ASD 分類性能的中間任務。
Materials and Methods材料及方法 Data and Preprocessing數據和預處理The data used in sex classification come from the Human Connectome Project (HCP) S1200 release (Van Essen etal. 2012), including 1113 subjects (505 females vs. 606 males). Subjects are scanned in 3T Siemens scanners in Washington University with the following parameters: spatial resolution?=?2?×?2?×?2mm3, time repetition (TR)?=?720ms, time echo (TE)?=?33.1ms, field of view (FoV) =?208?×?180mm2, slices?=?72, flip angle?=?52 degrees. Male and female subjects are matched in age and education.
性別分類中使用的數據來自人類連接組項目(HCP) S1200發布(Van Essen 等，2012) ，包括1113名受試者(505名女性對606名男性)。在華盛頓大學的3T 西門子掃描儀中掃描受試者，參數如下: 空間分辨率 = 2 × 2 × 2mm3，時間重復(TR) = 720ms，時間回波(TE) = 33.1 ms，視場(FoV) = 208 × 180mm2，切片 = 72，翻轉角 = 52度。男性和女性受試者在年齡和教育程度上是匹配的。
We use the large-scale publicly available dataset, the Autism Brain Imaging Data Exchange (ABIDE) dataset (Di Martino etal. 2014; Di Martino etal. 2017), for ASD classification. The ABIDE dataset consists of 2 subsets: ABIDE I and ABIDE II. ABIDE I contains 1112 subjects (539 ASD patients vs. 573 normal controls) collected from 16 sites and ABIDE II comprises 1114 subjects (521 ASD patients vs. 593 normal controls) collected from 19 sites. We first discard 219 samples from 2226 samples of ABIDE dataset whose scans fail to complete all steps of the Freesurfer preprocessing pipeline due to low-image quality (Backhausen etal. 2016). In addition, Freesurfer sometimes generates incorrect segmentation owing to the low-image quality and the challenging of whole-brain reconstruction, even though the sample passes the pipeline. So the segmentation quality is further checked by visual inspection, and 13 subjects whose segmentations are incorrect are excluded. Finally, a total of 1994 subjects are involved in the following analysis.
我們使用大規模公開可用的數據集，自閉癥腦成像數據交換(ABIDE)數據集(Di Martino 等，2014; Di Martino 等，2017)進行 ASD 分類。ABIDE 數據集由兩個子集組成: ABIDE I 和 ABIDE II。ABIDE I 包含從16個站點收集的1112個受試者(539個 ASD 患者對573個正常對照) ，ABIDE II 包含從19個站點收集的1114個受試者(521個 ASD 患者對593個正常對照)。我們首先丟棄2226個 ABIDE 數據集樣本中的219個樣本，由于圖像質量低，其掃描未能完成 Freesurfer 預處理流水線的所有步驟(Backhausen 等，2016)。此外，Freesurfer 有時會產生不正確的分割，由于低圖像質量和全腦重建的挑戰，即使樣本通過管道。進一步通過視覺檢測對分割質量進行了檢驗，排除了13個分割不正確的被試。最后，對1994名研究對象進行了以下分析。
As with many other MRI studies, we focus on the cerebral cortex. The cerebral cortex can be regarded as a thin folding surface with heterogeneous thickness, so it is impossible to transform it into a 2D image directly. In this study, the structural MRI preprocessing pipeline of FreeSurfer is adopted to preprocess data from both HCP and ABIDE (Glasser etal. 2013). The pipeline includes the segmentation of T1w volume, tessellation and topology correction of the initial white matter surface, spherical inflation of the white matter surface, registration to the fsaverage surface template, segmentation of sulci and gyri, pial surface generation, surface and volume anatomical parcellations, and morphometric measurements (Fischl 2012; Glasser etal. 2013). The 32k cortical meshes and vertex-wise cortical shape metrics, including thickness, sulcal depth, curvature, and myelin map, are generated from the cerebral cortex. Due to the lack of T2-weighted images, the myelin map for ABIDE is unavailable.
和其他許多核磁共振成像研究一樣，我們關注的是大腦皮層。大腦皮層可以看作是一個具有不均勻厚度的薄折疊表面，因此不可能將其直接轉換成二維圖像。在這項研究中，FreeSurfer 的結構 MRI 預處理流水線被用來預處理來自 HCP 和 ABIDE 的數據(Glasser et al。2013)。管道包括 T1w 體積的分割，初始白質表面的鑲嵌和拓撲校正，白質表面的球形膨脹，對平均表面模板的配準，腦溝和腦回的分割，軟腦膜表面生成，表面和體積解剖分區以及形態測量(Fischl 2012; Glasser 等，2013)。32k 的皮質網格和頂點狀的皮質形狀度量，包括厚度，溝深，曲率和髓鞘圖，是由大腦皮質產生的。由于缺乏 T2加權像，ABIDE 的髓鞘圖不可用。
Geometry Mapping幾何映射As mentioned above, each metric is composed of surfaces from 2 hemispheres. To adapt the 3D imaging data to 2D models, we need to map 3D cortical meshes into 2D images. A 3D mesh generated by FreeSurfer is a folded closed surface that could not be directly mapped into a planar mesh. However, vertexes corresponding to the medial wall that is close to the subcortical regions would have null values. We remove these vertexes and thus obtain unclosed meshes, which can be theoretically mapped into a regular planar mesh using geometry mapping approaches. Conformal mapping (Wang etal. 2011) and area-preserving mapping (Su etal. 2013; Zhao etal. 2013) are 2 commonly used geometry mapping approaches that map irregular 3D meshes as regular planar meshes. The former keeps the mapping of angles but leads to area distortion, and the latter will cause the opposite effects (Nadeem etal. 2016). The area distortion may seriously influence the training of deep models, so we adopt the area-preserving mapping approach. Considering the compatibility with deep learning, we map the brain image into a unit rectangle.
如上所述，每個度量由來自兩個半球的表面組成。為了使三維成像數據適應二維模型，我們需要將三維皮層網格映射成二維圖像。由 FreeSurfer 生成的3D 網格是一個不能直接映射到平面網格的折疊封閉表面。然而，對應于靠近皮質下區域的內側壁的頂點將具有空值。我們刪除這些頂點，從而獲得非封閉網格，這可以在理論上映射到一個規則的平面網格使用幾何映射方法。保形映射(Wang et al。2011)和面積保持映射(Su et al。2013; Zhao et al。2013)是兩種常用的幾何映射方法，將不規則的3D 網格映射為正則的平面網格。前者保持角度映射，但導致面積失真，后者將導致相反的效果(Nadeem et al。2016)。面積畸變會嚴重影響深度模型的訓練，因此采用了面積保持映射方法。考慮到與深度學習的兼容性，我們將大腦圖像映射成一個單位矩形。
Supposing 假設is the input surface mesh in 是輸入曲面的網格with Riemannian metric 用黎曼度量?, there is a unique conformal mapping 有一個唯一的保角映射according to the Riemann mapping theorem, where 根據黎曼映射定理Dis a unit square with 4 corners predefined as 4 vertexes equally distributed along the surface edge. Then there is a unique Brenier mapping 是一個單位正方形與4個角預定義為4個頂點均勻分布沿表面邊緣。然后是一個獨特的布雷尼爾映射?, which makes sure the area of each cell is preserved. The area-preserving mapping ( ，以確保每個單元格的面積得到保留Zhao etal. 2013 Zhao 等人2013年) is the combination of the Riemann mapping and Brenier mapping: )是黎曼映射和布雷尼爾映射的結合:?. In practice, the conformal mapping procedure can be implemented with the discrete Ricci Flow method. Supposing the vertexes are 在實際應用中，可以用離散 Ricci 流方法實現保角映射過程?, the curvature is 曲率是?, and the target curvature is 目標曲率是?, the conformal factor is defined as 保形因子定義為?, and the discrete Ricci flow can be represented by ( 離散的 Ricci 流可以用(Wang etal. 2011 Wang 等人2011年; Zhao etal. 2013 Zhao 等人2013年)close to original areas 接近原來的區域?, where ，在哪里represent the initial area of 表示... 的初始面積?. Assigning the height vector of the cells as 。將單元格的高度向量指定為?. For any given measurement 。任何給定的測量?, there must exist a unique ，必須存在一個獨特的that satisfies the area-preservation constraints for all cells. The Brenier mapping can be calculated with the energy function 滿足所有單元的面積保持約束。布雷尼爾映射可以用能量函數來計算Su etal. 2013). We set the target curvature to for 4 corners and 0 for other vertexes to generate a rectangular mesh. The 3D cortical mesh is then topologically and equably mapped onto a rectangular planar mesh without any tearing or overlaps after the geometry mapping. The acquired planar mesh cannot be directly utilized by convolutional networks, and the brain images need to have the same data form as the natural images used in the pretrained models. So it is necessary to transform it into 2D images. We use a weighted triangular interpolation approach based on barycentric coordinates to avoid the null values in the 2D images. The average of the points that fall within the same pixel is taken as the pixel’s value.這項研究的重點是大腦形狀指標的圖像，但在表面參量化過程中產生的特征，如貝爾特拉米系數，也包含有區分信息(Su et al。2013)。對于4個角，我們將目標曲率設置為 ，對于其他頂點，我們將目標曲率設置為0，以生成一個矩形網格。然后，在幾何映射之后，將三維皮層網格拓撲和均勻地映射到一個矩形平面網格上，沒有任何撕裂或重疊。所獲得的平面網格不能直接用于卷積網絡，而且腦圖像需要與預訓練模型中使用的自然圖像具有相同的數據形式。因此，有必要將其轉換成二維圖像。我們使用一個基于質心座標的加權三角插值方法來避免二維圖像中的零值。落在同一像素內的點的平均值作為該像素的值。
Training with CNN和 CNN 一起訓練 Deep Model Architecture深度模型體系結構After the area-preserving geometry mapping, the vertex-wise cortical shape metrics are mapped as 224?×?224 images. To demonstrate the reliability of our method, we use 2 popular deep convolutional networks (i.e., ResNet-50 and DenseNet-121) for experiments. ResNet adds skip connections between the adjacent layers and calculates residuals from inputs to outputs. It alleviates the gradient disappearance in deep learning and achieves better performance. DenseNet introduces skip connections between every 2 layers and uses concatenation operation instead of summation operation used in ResNet. Both models have been demonstrated to be robust and efficient in image classification, and the corresponding pretrained models are widely used and available online.
經過面積保持的幾何映射，頂點方向的皮層形狀度量映射為224 × 224圖像。為了證明該方法的可靠性，我們使用了2個流行的深卷積網絡(即 ResNet-50和 DenseNet-121)進行實驗。ResNet 添加了相鄰層之間的跳躍連接，并計算從輸入到輸出的殘差。它緩解了深度學習中的梯度消失現象，取得了較好的效果。DenseNet 在每2層之間引入跳躍連接，并使用串聯操作代替 ResNet 中使用的求和操作。兩種模型在圖像分類中均具有較強的魯棒性和較高的分類效率，相應的預訓練模型得到了廣泛的應用和在線應用。
Transfer Learning from ImageNet基于 ImageNet 的遷移學習Transfer learning is utilized to improve network performance on small datasets. Deep neural networks are first pretrained on the large-scale natural image dataset ImageNet. Then the fully connected layer is replaced to meet the class number of the target task, and the pretrained models are fine-tuned using the acquired 2D images of different metrics, respectively. The 10-fold cross-validation strategy is used to test the reliability of the classification performance. The samples are randomly shuffled and divided into 10-folds, without consideration of scanning site, sex, and patient/control ratio. In each experiment, we use 9-folds for training and the left fold for testing. Before training, the input images are normalized, and the resulted images can be formulated as ?, where and are the mean value and standard deviation of the input images, respectively.
利用傳遞學習提高小數據集上的網絡性能。首先在大規模自然圖像數據集 ImageNet 上對深度神經網絡進行預訓練。然后替換完全連通的層以滿足目標任務的類數，并分別利用獲得的不同度量的二維圖像對預訓練后的模型進行微調。10倍交叉驗證策略用于測試分類性能的可靠性。將樣本隨機分為10組，不考慮掃描部位、性別和患者/對照組比例。在每個實驗中，我們使用9倍的訓練和測試左折疊。在訓練之前，輸入圖像 被標準化，得到的圖像 可以表示為 ?，其中 和 分別是輸入圖像的平均值和標準差。
Mix-Up混亂A mix-up strategy is also introduced in the training procedure ( 在訓練過程中還引入了混合策略(Zhang etal. 2018 Zhang 等人2018年). Mix-up is a widely used data augmentation method in computer vision. It uses the linear interpolation of 2 random samples and their labels as virtual samples to improve the generalization capability of the network. The mix-up can be formulated asare randomly selected samples, 是隨機抽取的樣本,are corresponding labels, 是相應的標簽,and 還有are generated sample and label, respectively. The hyperparameter 分別生成樣本和標簽is used to adjust the mix ratio. We set 用來調整混合比例。我們設置as the uniform distribution between 0 and 1 in the study. The mix-up strategy is conducted in the first half of the model training procedure to improve the representation capability of the model. Then we refine-tune the networks with original data to improve the performance on real data. Ensemble Using Stacking使用堆疊的合奏After the training and testing procedure, we obtain the results of different metrics of both hemispheres. It is necessary to ensemble the metric-level results to generate the final individual-level classification results. Instead of simple voting or weighted voting methods, we adopt the hierarchical model ensemble method, that is, stacking, for individual-level ensemble (Wolpert 1992). Specifically, the results of different metrics are concatenated as the new input features of the individual-level classification model. The extreme gradient boosting (XGboost) (Chen and Guestrin 2016) is adopted as the stacking model. Compared with voting-based ensemble methods, stacking can automatically learn the weights of the input features and usually gets better results. The hyperparameters of XGboost are optimized with the grid search method.
經過訓練和測試過程，我們得到了兩個半球不同度量的結果。為了生成最終的個體層次分類結果，有必要對度量層次結果進行集成。我們不采用簡單的投票或加權投票的方法，而是采用層次模型集成的方法，即疊加，為個人水平集成(Wolpert 1992)。具體來說，不同度量的結果被連接起來作為個人層次分類模型的新輸入特征。采用極端梯度提升(xgost)(Chen and Guestrin 2016)作為堆疊模型。與基于投票的集成方法相比，疊加方法能夠自動學習輸入特征的權重，通常能夠得到更好的結果。利用網格搜索方法對 XGBoost 的超參數進行了優化。
Two-Stage Transfer Learning兩階段遷移學習For the classification of ASD, we introduce the 2-stage transfer learning approach for better performance. Although it is popular to use the models pretrained with large-scale natural image datasets on other fields, the domain differences between the source and target datasets will still affect the effectiveness of transfer learning (Jean etal. 2016). An effective approach is to use an intermediate domain to bridge the source domain and target domain. The model is first transferred from the source domain to the intermediate domain and then transferred from the intermediate domain to the target domain. For neuropsychiatric disorders such as ASD, sex classification of healthy people is an excellent intermediate domain task. Firstly, brain images from healthy people and ASD patients have similar features. Compared with neuropsychiatric disorders, healthy people’s data usually have better homogeneity. Moreover, sex labels are credible, which is vital in brain imaging analysis. In the 2-stage transfer learning, we first convert both the intermediate domain (HCP) and the target domain (ABIDE) from 3D MRI data to 2D images with the Freesurfer pipeline. The models are transferred from ImageNet to the sex classification of healthy people with the HCP dataset. Then the acquired models are further transferred to the classification of ASD with the ABIDE dataset. In the 2-stage transfer learning framework, the models are fine-tuned twice using the intermediate domain and the target domain, respectively.
對于 ASD 的分類，我們引入了兩階段遷移學習方法以獲得更好的性能。盡管在其他領域使用大規模自然圖像數據集預訓練的模型是流行的，但是源和目標數據集之間的領域差異仍然會影響轉移學習的有效性(Jean et al。2016)。一種有效的方法是使用一個中間域來連接源域和目標域。模型首先從源域轉移到中間域，然后從中間域轉移到目標域。對于像 ASD 這樣的神經精神障礙，健康人的性別分類是一個很好的中間領域的任務。首先，健康人和自閉癥患者的大腦圖像具有相似的特征。與神經精神障礙相比，健康人的數據通常具有更好的同質性。此外，性別標簽是可信的，這是至關重要的腦成像分析。在兩階段傳輸學習中，我們首先使用 Freesurfer 流水線將三維 MRI 數據中的中間域(HCP)和目標域(ABIDE)轉換為二維圖像。模型通過 HCP 數據集從 ImageNet 轉移到健康人的性別分類。然后利用 ABIDE 數據集將獲得的模型進一步轉化為 ASD 的分類。在兩階段遷移學習框架中，分別使用中間域和目標域對模型進行兩次微調。
Visualization想象To interpret the classification results of the models and locate the cortical shape morphometric differences, the occlusion test is adopted to measure the importance of different regions in the classification (Zeiler and Fergus 2014). Specifically, we cover the image with a 30?×?30 black square and calculate the accuracy drop, which is regarded as the importance of the covered region in the classification. Then we move the square to the next region with a stride of 4 until the whole image is covered. We finally get occlusion test maps and resize them to the same size as the original image. We use the average of all images as the final results. The occlusion test results are then reconstructed as 3D meshes and visualized with Connectome Workbench visualization software.
為了解釋模型的分類結果和定位皮層形態學差異，采用遮擋測試來測量不同區域在分類中的重要性(Zeiler and Fergus 2014)。具體地說，我們用一個30 × 30的黑色正方形覆蓋圖像，并計算精度下降，這被認為是覆蓋區域在分類中的重要性。然后我們將正方形移動到下一個區域，步長為4，直到整個圖像被覆蓋。我們最終得到遮擋測試圖，并將它們調整到與原始圖像相同的大小。我們使用所有圖像的平均值作為最終結果。然后將遮擋測試結果重建為三維網格，并用 Connectome Workbench 可視化軟件進行可視化。
Results結果 Training Details培訓詳情The models are trained on an Ubuntu 18.04.1 server with 2 8-core Intel E5 2609 1.7GHz processors and 4 NVIDIA GTX-V100 graphical processing units. The code is written in Python and Pytorch framework (Paszke etal. 2019). The models pretrained on ImageNet are acquired from torchvision (https://download.pytorch.org/models/). Each model is trained for 125 epochs with a batch size of 64, of which the first 75 epochs are trained with mix-up, and the latter 50 epochs are trained with original data, and the model of the last epoch is retained for testing. Stochastic gradient descent and cross-entropy loss are adopted for model optimization. The learning rate is set to 0.01 initially, divided by 10 every 25 epochs for the first 75 epochs, and is then fixed at 0.0001 for the following 50 epochs. The momentum is set to 0.9. The hyperparameters are optimized using the grid search strategy.
這些機型在 Ubuntu 18.04.1服務器上進行培訓，該服務器配有2個8核 Intel E526091.7 GHz 處理器和4個 NVIDIA GTX-V100圖形處理單元。代碼是用 Python 和 Pytorch 框架編寫的(Paszke 等，2019)。在 ImageNet 上預先訓練的模型是從 torchvision ( https://download.pytorch.org/models/)獲得的。每個模型訓練125個歷元，批量大小為64個，其中前75個歷元用混合訓練，后50個歷元用原始數據訓練，最后一個歷元的模型保留用于測試。模型優化采用隨機梯度下降和交叉熵損失。最初將學習率設置為0.01，除以前75個紀元中每25個紀元的10個，然后將后50個紀元的學習率固定為0.0001。動量設定為0.9。利用網格搜索策略對超參數進行了優化。
Area-Preserving Geometry Mapping Results面積保持幾何映射結果The sketch diagram of area-preserving geometry mapping for cortical meshes is shown in Figure 1. To visually compare the 3D cortical meshes and the corresponding 2D images, we map the Desikan–Killiany (D–K) atlas (Desikan etal. 2006) as a 2D planar atlas (Fig. 2). Brain regions are illustrated in different colors for visualization. For the 3D atlas, a series of different views are needed to show the complete information of the whole brain, and some regions are still hard to observe due to the complex folding of the cerebral cortex. However, our 2D atlas can avoid these disadvantages and show the whole brain without occlusion in one view, which demonstrates the potential and superiority of our method in the visualization of brain images.
皮層網格的面積保持幾何映射示意圖如圖1所示。為了在視覺上比較3D 皮層網格和相應的2D 圖像，我們將 Desikan-Killiany (D-K)圖譜(Desikan 等，2006)映射為2D 平面圖譜(圖2)。大腦區域以不同的顏色顯示。對于三維地圖集，需要一系列不同的視圖來顯示整個大腦的完整信息，而且由于大腦皮層的復雜折疊，一些區域仍然難以觀察到。然而，我們的二維地圖集可以避免這些缺點，并顯示整個大腦沒有遮擋在一個視圖，這表明我們的方法在可視化的腦圖像的潛力和優越性。
Figure 1 圖1100">Open in new tab 打開新標簽Download slide 下載幻燈片Overview of the proposed framework. FreeSurfer is used to generate 3D cortical meshes and vertex-wise cortical shape metrics. The 3D mesh is then converted into a planar mesh using area-preserving geometry mapping. Different metrics are calculated using transfer learning with ImageNet, and then the results are further ensembled with a stacking approach to get individual-level results. The sketch diagram of the geometry mapping is also displayed. The red points distributed evenly on the edge are the selected corners.
擬議架構概覽。FreeSurfer 用于生成3D 皮層網格和頂點形狀度量。然后使用面積保持幾何映射將三維網格轉換成平面網格。使用 ImageNet 的轉移學習計算不同的度量，然后將結果進一步集成到一個堆疊方法中，以獲得個體級別的結果。還顯示了幾何映射的示意圖。在邊緣上均勻分布的紅點是被選中的角。
Figure 2 圖2100">Open in new tab 打開新標簽Download slide 下載幻燈片The 3D D-K atlas (A) with different views and the 2D D–K atlas (B) generated by our method. The corresponding brain regions of 2 atlases are shown in the same colors.
該方法生成了具有不同視圖的三維 D-K 圖譜(A)和具有不同視圖的二維 D-K 圖譜(B)。2個地圖集的相應腦區顯示為相同的顏色。
Sex Classification Results性別分類結果Sex classification is a fundamental problem in brain imaging analysis. There is a long debate about whether male and female brains are distinguishable, and many studies attempt to solve the problem with machine learning methods (Weis etal. 2020). We perform the sex classification task on the HCP dataset, and the results are shown in Table 1. Two different deep models are adopted to measure the effectiveness of the proposed method. For comparison, we test the models trained from scratch first and achieve 89.67% accuracy for ResNet and 92.99% accuracy for DenseNet. Furthermore, we test transfer learning by transferring the models pretrained on the source dataset (ImageNet) to the target dataset (HCP). Transfer learning achieves an accuracy of 94.34% for ResNet and 95.06% for DenseNet, resulting in improvements of 4.67% and 2.07% in accuracy. The results demonstrate that transfer learning could significantly boost classification performance. The receiver operating characteristic (ROC) curves and confusion matrices are shown in and , respectively. The proposed method achieves the best area under ROC curves (AUC) score of 0.9854. The high accuracy of our experiment on sex classification demonstrates the effectiveness of our framework and suggests that males and females are distinguishable with the cortical shape metrics revealed by structural MRI.
性別分類是腦成像分析中的一個基本問題。關于男性和女性的大腦是否可以區分，有一個長期的爭論，許多研究試圖用機器學習方法來解決這個問題(Weis et al. 2020)。我們在 HCP 數據集上執行性別分類任務，結果如表1所示。采用兩種不同的深度模型來衡量該方法的有效性。為了進行比較，我們首先測試了從頭開始訓練的模型，ResNet 的準確率為89.67% ，DenseNet 的準確率為92.99% 。此外，我們通過將源數據集(ImageNet)上預先訓練的模型轉移到目標數據集(HCP)來測試遷移學習。轉移學習在 ResNet 和 DenseNet 中分別達到了94.34% 和95.06% 的準確率，分別提高了4.67% 和2.07% 的準確率。結果表明，遷移學習可以顯著提高分類性能。ROC曲線(ROC)曲線和混淆矩陣分別顯示在補充圖 S1和 S2中。該方法在 ROC 曲線下的最佳面積為0.9854。我們的性別分類實驗的高準確性證明了我們的框架的有效性，并表明男性和女性是可以區分的皮質形狀指標由結構 MRI 顯示。
Sex classification results of cerebral cortex based on HCP
| Methods 方法 | Acc (%) 進度(%) | Sen (%) 森(%) | Spc (%) 規格(%) | AUC |
|---|---|---|---|---|
| ResNet | 89.67 | 91.42 | 87.57 | 0.9615 |
| DenseNet | 92.99 | 93.73 | 92.11 | 0.9818 |
| ResNet (transfer) ResNet (傳輸) | 94.34 | 95.21 | 93.29 | 0.9832 |
| DenseNet (transfer) 致密網(轉讓) | 95.06 | 95.87 | 94.08 | 0.9854 |
Acc, Sen, and Spc refer to accuracy, sensitivity, and specificity, respectively. The transfer refers to transfer learning from ImageNet to HCP. The best accuracy, sensitivity, specificity, and AUC are shown in bold.
Open in new tab 打開新標簽Fig. 3). Results from 2 hemispheres are concatenated as the inputs of the individual-level classifiers. The metrics-level accuracies are improved with a range of 4.49–11.01% and 1.53–4.13% for ResNet and DenseNet with transfer learning, respectively. The significant improvement with transfer learning on each metric demonstrates the stability and effectiveness of our framework. The myelin map achieves the best accuracy. The curvature performs worse than other metrics but gains the most significant improvement with transfer learning.不同指標的計算結果也采用了疊加法(圖3)。來自兩個半球的結果被串聯起來作為個體層次分類器的輸入。通過轉移學習，ResNet 和 DenseNet 的度量級精度分別提高了4.49-11.01% 和1.53-4.13% 。在每個指標上的遷移學習的顯著改進證明了我們框架的穩定性和有效性。髓鞘圖達到了最佳的準確性。曲率比其他度量表現更差，但是通過遷移學習獲得了最顯著的改善。
Figure 3 圖3100">Open in new tab 打開新標簽Download slide 下載幻燈片Classification results of single metric on sex classification (left) and ASD classification (right). In sex classification, the results on thickness, sulcal depth, curvature, and myelin map are shown to investigate the effectiveness of transfer learning under our framework. In ASD classification, the results of thickness, sulcal depth, and curvature are shown to observe performance improvement with the proposed method. The “transfer” represents the transfer learning from ImageNet to the target dataset, and the “2 stage” refers to the transfer learning from ImageNet to ABIDE with HCP as the intermediate domain.
性別分類單指標分類結果(左)和 ASD 分類單指標分類結果(右)。在性別分類中，結果顯示厚度，溝深度，曲率和髓鞘地圖，以調查有效的遷移學習在我們的框架下。在 ASD 分類中，通過對厚度、溝深和曲率的分析，可以觀察到該方法對 ASD 分類性能的改善。“轉移”是指從 ImageNet 到目標數據集的轉移學習，“2階段”是指以 HCP 為中間域的從 ImageNet 到 ABIDE 的轉移學習。
Moreover, we explore the effects of total intracranial volume (TIV) on sex classification (Sanchis-Segura etal. 2020) under our framework. The results show that our framework still works well with matched TIV ().
此外，在我們的框架下，我們探討了顱內總體積(TIV)對性別分類的影響(Sanchis-Segura 等，2020)。結果表明，我們的框架仍然與匹配的 TIV 工作得很好(補充表 S1)。
ASD Classification ResultsASD 分類結果We further apply our method to a multisite ASD dataset (ABIDE) to distinguish patients from healthy controls. Due to the lack of T2-weighted images, the myelin maps are not available for the ABIDE dataset, so we only use thickness, sulcal depth, and curvature for the classification of ASD. The results are shown in Table 2.
我們進一步將我們的方法應用于多位點 ASD 數據集(ABIDE) ，以區分患者和健康對照。由于缺乏 T2加權像，ABIDE 數據集無法獲得髓鞘圖，因此我們只能使用厚度、溝深和曲率對 ASD 進行分類。結果如表2所示。
ASD classification results of cerebral cortex on ABIDE
| Methods 方法 | Acc (%) 進度(%) | Sen (%) 森(%) | Spc (%) 規格(%) | AUC |
|---|---|---|---|---|
| PCA?+?SVM PCA + SVM | 58.12 | 48.99 | 66.82 | 0.6102 |
| DenseNet (slice) 致密網(片) | 61.83 | 52.83 | 69.86 | 0.6693 |
| DenseNet (3D volume) DenseNet (3D 卷) | 62.39 | 54.53 | 69.38 | 0.6355 |
| DenseNet (3D mesh) 致密網絡(3D 網格) | 61.13 | 53.24 | 68.15 | 0.6438 |
| ResNet | 63.04 | 52.40 | 72.51 | 0.6756 |
| DenseNet | 63.64 | 55.80 | 70.62 | 0.6725 |
| ResNet (transfer) ResNet (傳輸) | 65.89 | 60.28 | 70.90 | 0.6996 |
| DenseNet (transfer) 致密網(轉讓) | 65.59 | 57.29 | 72.99 | 0.7018 |
| ResNet (2-stage) ResNet (2階段) | 67.70 | 62.73 | 72.13 | 0.7199 |
| DenseNet (2-stage) 致密網絡(2階段) | 67.85 | 61.66 | 73.36 | 0.7237 |
The transfer refers to direct transfer learning from ImageNet to ABIDE, whereas the 2 stage represents the ImageNet-HCP-ABIDE transfer learning strategy. The PCA?+?SVM, DenseNet (slice), and DenseNet (3D volume) are based on the 3D cerebral cortex for a fair comparison. The DenseNet (3D mesh) is based on the 3D cortical shape metrics. The best accuracy, sensitivity, specificity, and AUC are shown in bold.
Open in new tab 打開新標簽ASD classification results of cerebral cortex on ABIDE
大腦皮層 ASD 在 ABIDE 上的分類結果
| Methods 方法 | Acc (%) 進度(%) | Sen (%) 森(%) | Spc (%) 規格(%) | AUC |
|---|---|---|---|---|
| PCA?+?SVM PCA + SVM | 58.12 | 48.99 | 66.82 | 0.6102 |
| DenseNet (slice) 致密網(片) | 61.83 | 52.83 | 69.86 | 0.6693 |
| DenseNet (3D volume) | 62.39 | 54.53 | 69.38 | 0.6355 |
| DenseNet (3D mesh) | 61.13 | 53.24 | 68.15 | 0.6438 |
| ResNet | 63.04 | 52.40 | 72.51 | 0.6756 |
| DenseNet | 63.64 | 55.80 | 70.62 | 0.6725 |
| ResNet (transfer) ResNet (傳輸) | 65.89 | 60.28 | 70.90 | 0.6996 |
| DenseNet (transfer) 致密網(轉讓) | 65.59 | 57.29 | 72.99 | 0.7018 |
| ResNet (2-stage) ResNet (2階段) | 67.70 | 62.73 | 72.13 | 0.7199 |
| DenseNet (2-stage) 致密網絡(2階段) | 67.85 | 61.66 | 73.36 | 0.7237 |
The transfer refers to direct transfer learning from ImageNet to ABIDE, whereas the 2 stage represents the ImageNet-HCP-ABIDE transfer learning strategy. The PCA?+?SVM, DenseNet (slice), and DenseNet (3D volume) are based on the 3D cerebral cortex for a fair comparison. The DenseNet (3D mesh) is based on the 3D cortical shape metrics. The best accuracy, sensitivity, specificity, and AUC are shown in bold.
遷移是指從 ImageNet 到 ABIDE 的直接遷移學習，而兩個階段代表 ImageNet-HCP-ABIDE 遷移學習策略。PCA + SVM，DenseNet (切片)和 DenseNet (3D 體積)是基于3D 大腦皮層進行公平比較。致密網(3D 網格)是基于3D 皮質形狀度量。最佳的準確性、靈敏度、特異性和 AUC 以粗體顯示。
Open in new tab 打開新標簽We first train and test ResNet and DenseNet from scratch and obtain the accuracies of 63.04% and 63.64%, respectively. Furthermore, we test direct transfer learning from ImageNet to the ABIDE dataset and achieve the accuracies of 65.89% for ResNet and 65.59% for DenseNet. Then the 2-stage transfer learning is tested based on the hypothesis that sex classification on healthy people can provide valuable features for the diagnostic classification of neuropsychiatric disorders. The models are transferred from ImageNet to HCP first and further transferred to ABIDE. The 2-stage transferred ResNet and DenseNet achieve the accuracies of 67.70% and 67.85%, respectively. The direct transfer learning from ImageNet brings increases of 2.85% for ResNet and 1.95% for DenseNet. Although the 2-stage transfer learning achieves improvements of 4.66% for ResNet and 4.21% for DenseNet in accuracy, the best AUC score is 0.7237.
首先從零開始對 ResNet 和 DenseNet 進行訓練和測試，得到的準確率分別為63.04% 和63.64% 。此外，我們測試了從 ImageNet 到 ABIDE 數據集的直接轉移學習，結果表明，對于 ResNet 和 DenseNet，準確率分別為65.89% 和65.59% 。然后基于健康人的性別分類可以為神經精神障礙的診斷分類提供有價值的特征的假設，對兩階段遷移學習進行了檢驗。模型首先從 ImageNet 轉移到 HCP，然后再轉移到 ABIDE。兩級傳輸的 ResNet 和 DenseNet 的準確率分別為67.70% 和67.85% 。ImageNet 的直接轉移學習使 ResNet 增加了2.85% ，DenseNet 增加了1.95% 。雖然兩階段遷移學習在準確性方面對 ResNet 和 DenseNet 分別提高了4.66% 和4.21% ，但最佳 AUC 評分為0.7237。
Moreover, we validate transfer learning from HCP to ABIDE to investigate the role of sex classification in 2-stage transfer learning. ResNet and DenseNet achieve accuracies of 65.44% and 65.25%, respectively, indicating that the pretraining on sex classification is helpful for the ASD classification.
此外，我們驗證了從 HCP 到 ABIDE 的遷移學習，以探討性別分類在兩階段遷移學習中的作用。ResNet 和 DenseNet 分別達到了65.44% 和65.25% 的準確率，說明性別分類的預訓練有助于 ASD 分類。
The results of thickness, sulcal depth, and curvature are shown in Figure 3. The results of single metrics are consistent with the individual-level results. Direct transfer learning achieves better results than training from scratch, whereas the 2-stage transfer learning reaches the highest accuracies in all metrics. Thickness seems to perform better in the ASD classification, whereas sulcal depth and curvature achieve considerable performance. Compared with direct transfer learning, 2-stage transfer learning brings more performance improvements in thickness and sulcal depth.
厚度、溝深和曲率的結果如圖3所示。單個指標的結果與個人層面的結果是一致的。直接遷移學習比從頭開始的學習效果更好，而兩階段遷移學習在所有指標中都達到了最高的精度。厚度似乎表現更好的 ASD 分類，而溝深度和曲率取得了相當大的性能。與直接遷移學習相比，兩階段遷移學習在厚度和溝深方面有更大的提高。
Moreover, we use the leave-one-site-out cross-validation to investigate the performance of different models on unseen sites, which can further demonstrate the generalization ability. In each experiment, one site is used as the testing set, and the rest are used as the training set. The results are shown in . The model trained from scratch, transfer learning, and 2-stage transfer learning achieve lower accuracies (61.34%, 63.45%, 65.41%) than those in 10-fold cross-validation. It is reasonable because testing on the unseen site is usually more difficult. However, the classification performance benefits from transfer learning and 2-stage transfer learning in leave-one-site-out cross-validation as well, indicating the robustness and effectiveness of the proposed framework.
此外，我們使用剩余一個站點的交叉驗證來研究不同模型在未見站點上的表現，這可以進一步證明推廣能力。在每個實驗中，一個站點作為測試集，其余站點作為訓練集。結果見補充表 S2。從頭開始訓練的模型、遷移學習和2階段遷移學習的準確率(61.34% 、63.45% 、65.41%)低于10倍交叉驗證的模型。這是合理的，因為在看不見的站點上進行測試通常更加困難。然而，在「留一個地點」的交叉驗證中，遷移學習和兩階段遷移學習均有助分類表現，顯示建議架構的穩健性和成效。
Comparison With Other Methods on ASD ClassificationASD 分類方法與其他方法的比較Many methods have been used for ASD classification based on the ABIDE dataset (Sabuncu etal. 2015; Aghdam etal. 2018; Monté-Rubio etal. 2018; Arya etal. 2020; Shahamat and Abadeh 2020). However, the sample size and brain features used in these studies vary a lot, making it difficult for horizontal comparison. To better measure the property of the proposed method and make a fair comparison, we compare the proposed framework with 4 other methods, including the support vector machine (SVM) (Chang and Lin 2011), slice-based 2D CNN, volume-based 3D CNN, and mesh-based 3D CNN using identical samples and brain features. Since ResNet and DenseNet have comparable performance, we only test the corresponding methods using the DenseNet architecture. We only consider the cerebral cortex in these experiments for a fair comparison.
根據 ABIDE 數據集(Sabuncu 等，2015; Aghdam 等，2018; Monté-ubio 等，2018; Arya 等，2020; Shahamat 和 Abadeh 2020) ，已經使用了許多方法進行 ASD 分類。然而，在這些研究中使用的樣本量和大腦特征差異很大，使得橫向比較很困難。為了更好地測量提出的方法的特性并進行公平的比較，我們將提出的框架與其他4種方法進行了比較，包括使用相同樣本和大腦特征的支持向量機(SVM)(Chang and Lin 2011)、基于切片的2D CNN、基于體積的3D CNN 和基于網格的3D CNN。由于 ResNet 和 DenseNet 具有相當的性能，我們只使用 DenseNet 體系結構測試相應的方法。在這些實驗中，我們只考慮大腦皮層，以便進行公平的比較。
The results of these methods are shown in Table 2. SVM is one of the classic methods for brain MRI. The data are preprocessed and reshaped into a vector, and the principal component analysis (PCA) (Wold etal. 1987) is adopted for feature extraction. An accuracy of 58.12% is achieved with SVM, which is significantly lower than that of the proposed 2-stage transfer learning, indicating the superiority of our framework. Some studies use 2D CNNs to analyze brain images by cutting them into slices. Similarly, in the slice-based DenseNet, the 3D brain images are cut into slices in 3 directions, and the pretrained models are fine-tuned using the acquired slices. The results of the slices are finally ensembled with stacking. The mix-up, stacking, and transfer learning strategies are adopted to ensure a fair comparison. Although the brain MRI is 3D, the slice-based 2D models analyze slices of one subject independently, leading to the loss of structural information (Khodatars etal. 2020; Wen etal. 2020). Moreover, the conversion leads to the loss of interslice information, resulting in a suboptimal accuracy of 61.83%. In volume-based 3D DenseNet, we use a 3D model with the same depth as the 2D model and get an accuracy of 62.39%. Mix-up is also adopted in the 3D model. Compared with these 2 methods, our method achieves significant performance improvement of 6.02% and 5.46%, powerfully demonstrating the superiority of our framework. Moreover, we train a 3D DenseNet using 3D cortical shape metrics to further examine the influence of different distance measurement methods. We resample the 3D surface meshes of each shape metric (thickness, sulcal depth, and curvature) into the 3D matrix, and then we train 3D DenseNets on the obtained 3D matrices. We use the same training strategies as the 2D models, and the mix-up and stacking are used for a fair comparison. The 3D model uses the Euclidean distance directly and achieves an accuracy of 61.13%, which is lower than our framework.
這些方法的結果如表2所示。支持向量機是腦磁共振成像的經典方法之一。這些數據被預處理并重新形成一個矢量，并采用主成分分析(PCA)(Wold et al. 1987)進行特征提取。支持向量機算法的準確率為58.12% ，明顯低于所提出的兩階段遷移學習算法，說明了該算法的優越性。一些研究使用2D CNN 通過將大腦圖像切成片來分析它們。類似地，在基于切片的 DenseNet 中，將3D 腦圖像分成3個方向的切片，并使用獲得的切片對預先訓練好的模型進行微調。切片的結果最終與堆疊結合在一起。采用混合學習、疊加學習和遷移學習策略，以確保公平比較。盡管大腦 MRI 是3D 的，但基于切片的2D 模型獨立分析一個受試者的切片，導致結構信息的丟失(Khodatars 等，2020; Wen 等，2020)。而且，這種轉換會導致切片間信息的丟失，從而導致61.83% 的次優準確率。在基于體積的3D DenseNet 中，我們使用了與2D 模型具有相同深度的3D 模型，得到了62.39% 的精度。在三維模型中也采用了混合處理。與上述兩種方法相比，本文提出的方法在性能上分別取得了6.02% 和5.46% 的顯著提高，有力地證明了本文框架的優越性。此外，我們利用三維皮層形狀指標訓練一個三維致密網，以進一步檢驗不同距離測量方法的影響。我們將每個形狀度量(厚度、溝深和曲率)的三維表面網格重采樣到三維矩陣中，然后在得到的三維矩陣上訓練三維 DenseNet。我們使用相同的訓練策略作為二維模型，混合和堆疊是用于一個公平的比較。這個3 d 模型直接使用了歐幾里得度量，達到了61.13% 的準確率，比我們的框架要低。
Ablation Study消融研究We investigate the effectiveness of each module in our model with an ablation study (Table 3). DenseNet is adopted as the base model. The stacking and mix-up strategies are investigated. We combine stacking and mix-up with the base model to get new models. Then the acquired models are tested with the ABIDE dataset. The stacking and mix-up bring improvements of 1.50% and 0.70% accuracy, respectively, demonstrating the effectiveness of the stacking and mix-up training strategies.
我們通過消融研究調查了模型中每個模塊的有效性(表3)。基本模型采用了致密網絡(DenseNet)。研究了疊加和混合策略。我們結合疊加和混合的基礎模型得到新的模型。然后使用 ABIDE 數據集測試獲取的模型。疊加訓練和混合訓練的準確率分別提高了1.50% 和0.70% ，證明了疊加訓練和混合訓練策略的有效性。
Results of ablation study
| Stacking 堆疊 | Mix-up 搞錯了 | Acc (%) 進度(%) | Sen (%) 森(%) | Spc (%) 規格(%) | AUC |
|---|---|---|---|---|---|
| 63.94 | 56.01 | 70.99 | 0.6822 | ||
| √ | 65.44 | 57.83 | 72.23 | 0.6998 | |
| √ | 64.64 | 56.02 | 72.04 | 0.7017 | |
| √ | √ | 67.85 | 61.66 | 73.36 | 0.7237 |
All experiments are based on the proposed 2-stage transfer learning framework. The average of all metrics is used as the result when the stacking is not adopted. The best results are shown in bold.
Open in new tab 打開新標簽Results of ablation study
消融研究結果
| Stacking 堆疊 | Mix-up 搞錯了 | Acc (%) 進度(%) | Sen (%) 森(%) | Spc (%) 規格(%) | AUC |
|---|---|---|---|---|---|
| 63.94 | 56.01 | 70.99 | 0.6822 | ||
| √ | 65.44 | 57.83 | 72.23 | 0.6998 | |
| √ | 64.64 | 56.02 | 72.04 | 0.7017 | |
| √ | √ | 67.85 | 61.66 | 73.36 | 0.7237 |
All experiments are based on the proposed 2-stage transfer learning framework. The average of all metrics is used as the result when the stacking is not adopted. The best results are shown in bold.
所有的實驗都基于所提出的兩階段遷移學習框架。如果沒有采用堆疊，則使用所有度量的平均值作為結果。最佳結果以粗體顯示。
Open in new tab 打開新標簽 Visualization Results視像化結果As mentioned above, we utilize the occlusion test to visualize the critical regions for sex classification and ASD classification. We average the results of all subjects to obtain the group-level differences. The results of thickness, sulcal depth, curvature, and myelin map are calculated, respectively. The most critical regions are shown in Figure 4. The red color indicates highly discriminative brain regions, whereas the blue color denotes less discriminative regions.
如上所述，我們利用遮擋測試來可視化性別分類和 ASD 分類的關鍵區域。我們對所有受試者的結果進行平均，以獲得群體水平的差異。分別計算了厚度、溝深、曲率和髓鞘圖。最關鍵的區域如圖4所示。紅色表示高度區分的大腦區域，而藍色表示較少區分的區域。
Figure 4 圖4100">Open in new tab 打開新標簽Download slide 下載幻燈片Visualization of discriminative regions for sex classification and ASD classification using occlusion test. The most discriminative brain regions are marked. The red regions contribute more to classification.
基于遮擋試驗的性別分類和 ASD 分類區域可視化研究。大腦中最具辨別力的區域被標記出來。紅色區域對分類的貢獻更大。
Discussion討論 Methodology研究方法In this study, we use deep learning for MRI imaging analysis for several reasons. As the size of the dataset increases, the representation ability of traditional machine learning methods has reached limits. On the contrary, CNN models have many parameters to be trained and show better representational capability of fitting high-dimensional data such as brain images. Traditional machine learning algorithms like SVM usually depend on hand-crafted features. CNN models can automatically extract local features and distributed representations without complicated feature engineering, which is of great significance for finding brain differences and searching neurological biomarkers. Moreover, deep learning can be adapted to different tasks due to its strong adaptability. The transfer learning techniques for deep learning are matured and thus the pretrained models can be used small sample tasks for better performance.
在本研究中，我們使用深度學習進行 MRI 成像分析有幾個原因。隨著數據集規模的增大，傳統機器學習方法的表示能力已經達到了極限。相反，細胞神經網絡模型有許多參數需要訓練，并顯示出更好的表征能力來擬合高維數據，如腦圖像。像 SVM 這樣的傳統機器學習算法通常依賴于手工制作的特性。細胞神經網絡模型可以自動提取局部特征和分布式表征，而不需要復雜的特征工程，這對于發現大腦差異和尋找神經生物標志物具有重要意義。此外，深度學習具有很強的適應性，可以適應不同的任務。用于深度學習的遷移學習技術已經成熟，因此預先訓練的模型可以用于小樣本任務以獲得更好的性能。
Mapping the 3D cerebral cortex as 2D images also brings several benefits. The complex geometry and folding patterns of the cerebral cortex hinder its analysis. For example, 2 adjacent voxels in Euclidean space may be anatomically or functionally segregated due to the non-Euclidean geometry of the folding cerebral cortex. Current CNN models are based on Euclidean distance and ignore the structural features of the cerebral cortex, which is straightforward but coarse. The signatures from different brain regions are mixed after the convolutional operations, which is harmful in searching diagnostic biomarkers. Different from current deep learning models, the proposed framework uses the distance along the cortical surface to measure the relative positions of different brain regions, which is more neurobiologically relevant. Compared with other models based on Euclidean distance, our framework could better preserve the structural layout of the brain and obtain results with neurobiological significance. Moreover, many 2D pretrained models are available for transfer learning under our framework, which can significantly improve network performance. It is also evident that the converted 2D images are substantially compressed from 3D raw data while keeping the most valid information, leading to high efficiency during model training. Freesurfer also provides a 2D mapping, but the topological structure is not preserved because the mapped cerebral cortex is torn. The mapped 2D image of Freesurfer is irregular, which is harmful in the training of deep models. Moreover, even though we use the Freesurfer for preprocessing in this paper, our framework is also compatible with other preprocessing tools such as CIVET (MacDonald etal. 2000) and Fastsurfer (Henschel etal. 2020) only if they can generate similar cortical surface meshes and vertex-wise shape metrics.
將3D 大腦皮層映射為2D 圖像也會帶來一些好處。大腦皮層的復幾何和折疊模式妨礙了它的分析。例如，由于折疊的大腦皮層的非歐幾里得幾何，歐幾里得空間中相鄰的兩個體素可能在解剖學上或功能上被隔離。目前的 CNN 模型是基于歐幾里得度量的，忽略了大腦皮層的結構特征，大腦皮層雖然簡單但是粗糙。不同腦區的特征信號在卷積術后混雜在一起，不利于尋找診斷性的生物標志物。與目前的深度學習模型不同，提出的框架使用沿皮層表面的距離來測量不同大腦區域的相對位置，這在神經生物學上更相關。與其他基于歐幾里得度量的模型相比，我們的框架能夠更好地保存大腦的結構布局，并獲得具有神經生物學意義的結果。此外，在我們的框架下，許多二維預訓練模型可用于遷移學習，這可以顯著提高網絡性能。顯然，轉換后的二維圖像基本上是從三維原始數據壓縮而來的，同時保留了最有效的信息，從而提高了模型訓練的效率。Freesurfer 還提供了一個2D 映射，但是由于映射的大腦皮層被撕裂，拓撲結構沒有得到保留。自由沖浪運動員的二維映射圖像不規則，不利于深度模型的訓練。此外，即使我們在本文中使用 Freesurfer 進行預處理，我們的框架也與 CIVET (MacDonald et al。2000)和 Fastsurfer (Henschel et al。2020)等其他預處理工具兼容，只要它們能夠生成類似的皮層表面網格和頂點形狀指標。
Transfer Learning From ImageNet從 ImageNet 遷移學習The training of deep models requires a large-scale dataset, which is troublesome in brain imaging analysis since the collection of data is expensive and time-consuming. Many studies use models pretrained on natural image to medical image classification and achieve success. However, natural images are 2D planar ones, whereas brain images are always 3D, hindering the application of transfer learning in brain imaging studies. Our solution is to transform the cerebral cortex into 2D images and make transfer learning from natural images to brain images applicable. In this study, the models are first pretrained on a large natural image dataset, that is, ImageNet, and then fine-tuned with the converted 2D brain images. We demonstrate the effectiveness of transfer learning from natural images to MRI data by a robust and significant performance improvement of both sex classification and ASD classification. The success of the proposed deep transfer learning framework is expected and reasonable. Even though brain images are different from natural images, the pretrained model can provide universal features, especially low-level features, which can be effectively reused in brain imaging analysis. In the converted 2D brain images, morphometric features such as sulcal depth can be regarded as a kind of textural feature in the image processing field. The models pretrained on natural images have learned a mass of knowledge on texture features and are excellent in extracting texture features, which is helpful in the training of the brain morphometric features. Transfer learning also provides models with suitable initial parameters and makes the network easier to converge for small-scale datasets.
深度模型的訓練需要大規模的數據集，由于數據的收集成本高、耗時長，給腦成像分析帶來了很大的困難。許多研究利用基于自然圖像的預訓練模型對醫學圖像進行分類，并取得了成功。然而，自然圖像是二維平面圖像，而大腦圖像往往是三維的，阻礙了轉移學習在腦成像研究中的應用。我們的解決方案是將大腦皮層轉換成二維圖像，并使從自然圖像到大腦圖像的轉移學習適用。在這項研究中，模型首先在一個大型自然圖像數據集，即 ImageNet 上進行預訓練，然后用轉換后的2D 腦圖像進行微調。我們證明了從自然圖像到 MRI 數據的轉移學習的有效性，通過性別分類和 ASD 分類的強大和顯著的性能改進。所提出的深度遷移學習框架的成功預期是合理的。盡管腦圖像不同于自然圖像，但預訓練模型可以提供通用特征，特別是低層特征，可以有效地重用于腦圖像分析。在轉換后的二維腦圖像中，腦溝深度等形態測量特征可以看作是圖像處理領域的一種紋理特征。在自然圖像上預訓練的模型學習了大量關于紋理特征的知識，在提取紋理特征方面表現出色，有助于訓練大腦形態特征。傳遞學習還為模型提供了合適的初始參數，使得小規模數據集的網絡收斂更加容易。
Two-Stage Transfer Learning兩階段遷移學習The ABIDE dataset used in this study is collected from over 30 sites. The differences in scanning machines and experimental parameters usually introduce intersite data heterogeneity (Chen etal. 2015). Compared with single-site ASD classification studies, such multisite studies are more difficult (Katuwal etal. 2016). In contrast with many other studies that use part of the ABIDE dataset, our study involves all available subjects from ABIDE-I and ABIDE-II, which is more challenging but fairer. We have demonstrated that transfer learning from natural image classification to ASD classification works. To further improve the performance on ABIDE, we propose a 2-stage transfer learning framework using the HCP as an intermediate domain and achieve higher accuracy than direct transfer learning from ImageNet. HCP is an excellent intermediate dataset due to its good data quality and better homogeneity. As aforementioned, direct transfer learning from ImageNet could provide useful low-level features. Compared with ImageNet, HCP is more similar to the data of ABIDE. A reasonable guess is that the data distribution inconsistency between natural images and brain images of ASD can be alleviated with brain images of healthy people. The fine-tuning on HCP makes the features more suitable for the ASD classification. To validate our hypothesis, we calculate the filter variation of different convolutional layers (). Compared with direct transfer learning, the second-stage transfer learning from HCP to ABIDE shows a smaller variation in both low-level and high-level filters, suggesting that the first-stage transfer learning brings benefits in both low-level and high-level features. Moreover, sex classification is an excellent intermediate task. It is well known that sex is closely related to some neuropsychiatric disorders. For example, ASD appears 4 times greater in males than females. Previous studies have investigated the sex differences in ASD and emphasized the critical role of sex in ASD studies (Lawrence etal. 2020). The models fine-tuned on the HCP dataset have learned sex-related features, which may be helpful in the classification of ASD, considering the sex-biased phenomenon of the disorder. The doubtless sex label of HCP also ensures the reliability of the first-stage transfer learning. Moreover, we evaluate the effect of age by grouping the ABIDE into 2 groups based on whether the age of the sample is covered by HCP. There is no significant difference in performance improvement for 2-stage transfer learning between 2 groups. Our 2-stage transfer learning provides a new approach for solving the small sample problem in the classification of neuropsychiatric disorders by using both natural images and other brain imaging data.
Important Regions for Sex and ASD Classification性別和 ASD 分類的重要區域The brain differences between males and females have been explored in many previous studies. In this study, we find some critical regions in the sex classification based on shape metrics, that is, thickness, sulcal depth, curvature, and myelin map. The occlusion test maps directly visualize the importance of different brain regions in the sex classification. As shown in Figure 4, we get 4 different occlusion test maps for the brain shape metrics, whereas some metric-shared discriminative regions exist. We find that the superior frontal cortex, superior parietal cortex, supramarginal cortex, paracentral cortex, precuneus, temporal pole, and right lingual cortex exhibit higher discriminative power in the sex classification, which have also been reported in the previous studies with different imaging modalities. One of our previous studies has reported the sex-related structural and functional differences in the superior frontal cortex, right supramarginal cortex, right lingual cortex, and left superior parietal cortex (Wang etal. 2012). The gray matter volume differences of the precuneus and temporal pole are demonstrated in previous studies (Ruigrok etal. 2014). The sex differences of the superior frontal cortex, superior parietal cortex, and paracentral cortex are also observed during the risk-taking tasks (Lee etal. 2009). The identified brain regions are closely related to behavioral or cognitive differences between males and females. The superior frontal cortex is a vital brain region involved with various cognitive and motor tasks, including working memory, self-awareness, and attention (Li etal. 2013). The superior parietal cortex mainly focuses on visuospatial and attention processing, long-term and working memory (Koenigs etal. 2009). The lingual cortex is closely related to vision tasks and word processing (Mechelli etal. 2000). Precuneus participates in visuospatial imagery, episodic memory retrieval, and self-processing operations (Cavanna and Trimble 2006). The temporal pole is linked to social and emotional processing (Snowden etal. 2004). The paracentral cortex controls the motor and sensory innervations of the lower extremity, including muscles and the urinary bladder (Spasojevi? etal. 2013).
ASD is a developmental disorder characterized by difficulty in social interactions, verbal and nonverbal communication deficits, and stereotyped activities and limited interests (Lord etal. 2018). In this study, we find that the superior frontal cortex, precentral cortex, postcentral cortex, inferior temporal cortex, middle temporal cortex, left superior temporal cortex, and right fusiform are critical regions in the classification of ASD in more than 2 metrics. These regions are also investigated in other studies. For example, the curvature and folding index features from frontal and temporal cortices are dominant in the early detection of ASD (Katuwal etal. 2016). Differences in the right inferior temporal cortex and right fusiform are reported between ASD patients and normal controls (Shahamat and Abadeh 2020). A functional magnetic resonance imaging (fMRI) study reveals group differences in the development of the superior temporal cortex (Prigge etal. 2013). The change of postcentral cortex, precentral cortex, and superior frontal cortex is also reported (Chen etal. 2015). The frontal cortex is thought to be related to high-order cognition, social and emotional functions, language, which are deficient in ASD (Carper and Courchesne 2005). Some studies report motor function abnormalities in ASD, which is regarded to be related to the precentral and postcentral (Müller etal. 2001). Temporal regions are related to social perception, language, and the “theory of mind,” which are impaired in ASD (Gendry Meresse etal. 2005). Fusiform plays an essential role in face perception, which is the key feature of normal social functioning in humans. However, the fusiform cortex is found hypoactive in patients with ASD, which cause the abnormalities in face perception and social interactions (van Kooten etal. 2008).
自閉癥是一種發展障礙擁有屬性的社交障礙，缺乏語言和非言語交際，活動定型和興趣有限(Lord et al。2018)。在本研究中，我們發現上額葉皮層，中央前皮層，中央后皮層，下顳葉皮層，中顳葉皮層，左上顳葉皮層和右梭形是 ASD 分類的關鍵區域，超過2個指標。其他研究也對這些區域進行了調查。例如，額葉和顳葉皮層的曲率和折疊指數特征在 ASD 的早期發現中占主導地位(Katuwal 等，2016)。報道了 ASD 患者與正常對照者的右顳下皮質和右梭形區域的差異(Shahamat 和 Abadeh 2020)。一項功能性磁共振成像成像(fMRI)研究揭示了顳上皮質發育的群體差異(Prigge et al. 2013)。中央后皮層，中央前皮層和額上皮層的變化也有報道(Chen et al。2015)。額葉皮層被認為與高級認知、社會和情感功能、語言有關，這些都是 ASD 所缺乏的(Carper and Courchesne 2005)。一些研究報道 ASD 的運動功能異常，被認為與中樞前后相關(Müller et al. 2001)。時間區域與社會知覺、語言和“心理理論”有關，這些在 ASD 中受到損害(Gendry Meresse et al. 2005)。梭形在人的面孔感知中起著重要作用，是人類正常社會功能的關鍵特征。然而，梭形皮質在 ASD 患者中被發現是低活性的，這導致了面部感知和社會交往的異常(van Kooten et al. 2008)。
Our results are consistent with the conclusions of previous sex and ASD studies, validating the reliability of our results. It should also be noted that the brain regions with group differences are observed in different metric maps, indicating that the alterations of these brain regions are stable.
我們的結果與之前的性別和自閉癥研究的結論一致，驗證了我們結果的可靠性。還應該注意的是，在不同的度量圖中可以觀察到具有群體差異的大腦區域，這表明這些大腦區域的改變是穩定的。
Limitations and Future Work限制和未來的工作Although the proposed framework has made some progress, there are still several limitations. Firstly, there is a lack of fair comparison with other studies due to different sample sizes and brain features. Secondly, the cerebellum and subcortical regions are not involved in the analysis, with which the classification performance may be further improved. Recent studies have shown that the cerebellum and subcortical regions can also be converted into surface meshes (Chye etal. 2019; Sereno etal. 2020), and we will follow-up on the relevant studies and further complete our framework. Thirdly, we only use structural MRI in this study, but the proposed framework can be further extended to functional MRI. The combined structural and functional MRI may achieve better performance. We will promote our framework into fMRI data in the future.
雖然提出的框架已經取得了一些進展，但仍然存在一些局限性。首先，由于樣本量和大腦特征的不同，缺乏與其他研究的公平比較。其次，不涉及小腦和皮質下區域，這樣可以進一步提高分類性能。最近的研究表明，小腦和皮質下區域也可以轉化為表面網格(Chye 等，2019; Sereno 等，2020) ，我們將跟進相關研究并進一步完善我們的框架。第三，在本研究中我們只使用結構性磁共振成像，但是所提出的框架可以進一步擴展到功能性磁共振成像。結構和功能相結合的 MRI 可以獲得更好的性能。我們將在未來將我們的框架推廣到功能磁共振成像數據。
Conclusion結論In this paper, we propose a framework to map the 3D cerebral cortex into 2D images with geometry mapping and facilitate transfer learning from natural images to brain images. In this way, the mature algorithm and techniques for 2D images in computer vision can be easily applied in brain image analysis. The topological information of brain structure is preserved, which is plausible for cortical visualization and neurobiological analysis. We validate the effectiveness of our framework on sex and ASD classification with both traditional transfer learning and a novel 2-stage transfer learning and achieved significant performance improvement. The proposed framework creatively applies 2D pretrained models to cortical shape-based classification, shedding new light for brain image analysis.
在本文中，我們提出了一個框架來映射三維大腦皮層到二維圖像的幾何映射，并促進從自然圖像到大腦圖像的轉移學習。這樣，成熟的計算機視覺二維圖像處理算法和技術就可以很容易地應用于腦圖像分析。保留了大腦結構的拓撲信息，可用于皮層可視化和神經生物學分析。我們通過傳統的遷移學習和一種新的兩階段遷移學習驗證了我們的框架在性別和 ASD 分類上的有效性，并取得了顯著的性能改善。該框架創造性地將二維預訓練模型應用于基于皮層形狀的分類，為腦圖像分析提供了新的思路。
Notes筆記Conflict of Interest: None declared.
利益沖突: 未申報。
Funding資金National Key Research and Development Program (grant 2018YFB1305101); the National Natural Science Foundation of China (grant 62036013, 61722313, 61773391); the Science & Technology Innovation Program of Hunan Province (grant 2018RS3080).
國家重點研究開發項目(贈款2018YFB1305101) ; 國家自然科學基金(贈款62036013,61722313,61773391) ; 湖南省科技創新項目(贈款2018RS3080)。
關鍵詞：
今日最新！斗羅大陸：獨孤博首次出場腳氣太重遭嫌棄，寧榮榮：快用達克寧 獨孤博首次出場是在天斗皇家學院被雪星親王邀請前來助陣費，而當時獨孤
天風證券：給予百亞股份買入評級 天風證券股份有限公司孫海洋,尉鵬潔近期對百亞股份進行研究并發布了研
上海市建工醫院有編制嗎 上海建工醫院成立于1953年，二級甲等綜合醫院，核定床位400張，為事業 
世體談巴薩外租球員：阿布德有望留隊 朗格萊&烏姆蒂蒂將被清洗 直播吧6月26日訊6月30日是非常關鍵的日子，尤其對于巴薩來說，若俱樂部 
行業頂流配置，無愧旗艦輕客，江鈴福特新款輕客曝光 傳承近60年造車文化的江鈴福特，始終在輕客市場引領行業發展，更是 
巔峰極速漂移技巧 漂移教程分享 很多玩家之前沒有接觸過賽車游戲，突然接觸巔峰極速，那么很多的彎道你 環球快消息！??基于區域保留幾何映射的大腦皮層深度遷移學習| brain cortex | Oxford Academic Deep transfer learning for cerebral cortex using area-pr
