Luận văn thạc sĩ Vật lý kỹ thuật: An end-to-end pipeline for intracranial hemorrhage brain image registration

VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY

THIS THESIS IS COMPLETED AT

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY – VNU-HCM

Supervisor 1: Assoc Prof Huynh Quang Linh Supervisor 2: Prof Sozo Inoue Examiner 1: Prof Phan Bach Thang

Examiner 2: PhD Hoang Manh Ha

This master’s thesis is defended at HCM City University of Technology, VNU- HCM City on July 23rd, 2023

Master’s Thesis Committee: 1 Chairman : PhD Ly Anh Tu

2 Secretary : PhD Pham Thi Hai Mien 3 Examiner 1 : Prof Phan Bach Thang 4 Examiner 2 : PhD Hoang Manh Ha 5 Member : PhD Pham Tan Thi

Approval of the Chairman of Master’s Thesis Committee and Dean of Faculty of Applied Science after the thesis being corrected (If any)

CHAIRMAN OF DEAN OF FACULTY OF THESIS COMMITTEE APPLIED SCIENCE

TASK SHEET OF THE MASTER’S THESIS

Name : Lê Nhật Tân ID : 1970501

Date of Birth : February 25, 1997 Place of birth : Binh Thuan Major : Engineering Physics Major code : 8520401

I THESIS’S TITLE : An end-to-end pipeline for Intracranial Hemorrhage Brain Image Registration.

(Quy trình đầu cuối kết hợp hình ảnh xuất huyết nội sọ não) II TASKS :

Constructing a comprehensive procedure that integrates intracerebral hemorrhage images and aligns pathological images with a standardized image atlas

Addressing the obstacles inherent in training images containing irregular structures within a practical deep learning model for image alignment

III DATE OF ASSIGNMENT : January 2023.

IV DATE OF COMPLETION : June 2023.

V SUPERVISOR : Assoc Prof Huynh Quang Linh, Prof Sozo Inoue

Ho Chi Minh city, ………, 2023

DEAN OF FACULTY OF APPLIED SCIENCE

(Name & Sign)

ACKNOWLEDGEMENT

I would like to thank for Doctor Pham Tan Thi and Assoc Prof Huynh Quang Linh from Ho Chi Minh University of Technology, who inspired me to follow the research career Your meticulous attention to detail and thorough understanding of the subject matter have constantly challenged me to think critically and strive for excellence

I would like to convey gratitude to my supervisor Professor Sozo Inoue for his strong encouragements and directions Although this Master course was a challenge for me due to the Covid 19 spreading and my first abroad journey, my knowledge and future research direction has been continuously developing under his supervising

I would also like to thank Syoji Kobashi Sensei from University of Hyogo, Koichi Arimura Sensei from Kyushu University Hospital, Koji Iihara Sensei from National Cerebral and Cardiovascular Center Hospital for creating an interesting research on brain medical image and providing me valuable comments in my medical imaging research works

I want to give a special thanks to Nishimura san, Uchida san, and Yoshinaga san, who helped me a lot when I first arrived in Japan; for my peers Kaneko san, Adachi san, Kazuma san, Ryu san, Nazmun, my seniors Defry, Fikry, Alia, Farina, and Tina and juniors Min chan, Quynh, Miyake san, who made my study time in Japan greater and more enjoyable

With deep appreciation,

Le Nhat Tan

ABSTRACT

Intracranial Hemorrhage is a dangerous intra-brain bleeding event, that threatens human life if there is no timely recognition and treatment To assist the diagnosis and treatment planning for this serious event, attributes of the bleeding behavior should be analyzed over an extensive population of patients Non-rigid image registration could be a effective tool to simplify this analysis process due to its ability to transfer characteristics of the abnormality from the collected data to the certain reference template However, current advances in image registration still face a huge challenge due to the dissimilarity in image intensity among the images of disease subjects This thesis proposed a comprehensive pipeline including a traditional transformation and healthy tissue generation to handle the high-variance real-world collected dataset and the dissimilarity challenge in the Computed Tomography image of the Brain with Intracranial Hemorrhage, which is the new image scenario in this field of research The performance of the proposed pipeline was compared with the state-of-the-art deep learning model including with the traditional transformation through visual inspection, hematoma change rate and sum square different metrics The test result presented that the similar-based model performance was substantially fluctuated by the abnormal volume, while our pipeline significantly reduces its affection to improve the hematoma displacement and preservation, and keeping the considerable performance in the non-hematoma structure similarity The approach proposed in this work could remarkably contribute to further study on disease behavior analysis.

THE COMMITMENT OF THE THESIS’ AUTHOR

I, Le Nhat Tan, hereby declare that the master thesis titled "An end-to-end pipeline for Intracranial Hemorrhage Brain Image Registration" submitted to Ho Chi Minh University of Technology is a genuine and original work conducted by myself under the guidance of Assoc Prof Huynh Quang Linh and Prof Sozo Inoue

Thesis author,

Le Nhat Tan

TABLE LIST viii

ACRONYMS AND ABBREVIATIONS ix

CHAPTER 1: INTRODUCTION 1

1.1 Research Background and Practical Significance 1

1.2 Research Contribution 4

1.3 Research Goals 5

CHAPTER 2: LITERATURE REVIEW 6

2.1 From Traditional to Deep Learning Approaches 6

2.2 Similarity-based Deep Learning Approaches 10

2.3 Image Registration for Data with Abnormality 11

CHAPTER 3: DATASET 14

CHAPTER 4: METHODOLOGY 17

4.1 End-to-end pipeline 17

4.2 Hematoma Inpainting 19

4.3 Image Pre-alignment by Affine Transformation 21

4.4 Deep Similarity-Based Model 23

4.5 Evaluation Method 24

CHAPTER 5: RESULTS AND ANALYSIS 28

5.1 Visual Inspection 28

FIGURE LIST

Figure 1 1: The Intracranial Hemorrhage with a bleeding region occurs in the Computed Tomography (CT) and Magnetic Resonance Image (MRI) [2] 2Figure 3 1: The bleeding region annotation by physicians There is slices representative for 3 groups of hematoma volume with the hematoma to brain ratio is less than 0.05 (A), from 0.05 to 0.1 (B), and larger than 0.1 (C) The marked area is represented by the green line surrounding the bleeding area………14 Figure 3 2: The hematoma-brain shape volume ratio of all the test Brain ID 15Figure 4 1: The proposed image registration pipeline In which the real-world origin source image experiences the inpainting process and the affine trans-formation before feeding into the deep learning model The output of deep learning model, the deformable vector field, is used to warp the pre-align source data……… 19 Figure 4 2: The mask (down left) defines the hematoma region in the origin image dataset (top left) After inpainting process, the normal tissue within masked region are generated (right) 20Figure 4 3: The affine transformation process The Original image dataset (left) is pre-alignment by affine transformation with the target space as the Reference image 22Figure 4 4: In the encoder (gray color), the stridden convolution was used One more layer was added to the decoder (blue color) to transform the results into a flow field The number in each layer represents the number of used kernels, the below number is the resolution respectively with the size of each slice after the convolution and deconvolution process 23Figure 4 5: Model Optimization In training stage, the output DVF is used to warp the Inpainted Source image The loss optimization process includes the MSE loss is calculated by the difference between the Warped image and the Reference image, and the regularization loss is calculated within the transformation field (the DVF) 24

Figure 4 6: SSD Calculation Process The Reference and the Transferred slice are excluded the hematoma region before calculating 27

Figure 5 1: Visual Inspection of small-volume hematoma Columns from left to right are:Original slices, Reference slices, Results of DL pipeline and Results of DL-IP pipeline 29Figure 5 2: Visual Inspection of high-volume hematoma Columns from left to right are: Original slices, Reference slices, Results of DL pipeline and Results of DL-IP pipeline 29Figure 5 3: Visual Inspection of hematoma locates near by the skull structure Columns from left to right are: Original slices, Reference slices, Results of DL pipeline and Results of DL-IP pipeline 30Figure 5 4: Line plot of the hematoma change rate (excluding skull-close cases) after processing by the Deep Learning alone (orange) and with Image Inpainting (blue) follow by the hematoma volume 32Figure 5 5: The box plot of SSD value of DL (right) and DL-IP (left) pipeline 34Figure 5 6: The Sum Square Difference value of DL (orange) and DL-IP (blue) pipeline follow by the hematoma volume plot 35

Figure 6 1: The affection of hematoma volume on the hematoma displacement performance 38Figure 6 2: The affection of hematoma location on the hematoma displacement performance 38Figure 6 3: Comparison with and without using Affine Transformation as a Pre-aligment process Columns from left to right are: Original, Pre-aligment, Reference, the results on Original, and the results on Pre-alignemnt images 41

TABLE LIST

Table 2 1: Comparison of traditional standard and deep learning method 8

Table 5 1: Result summary of the visual inspection………30

Table 5 2: Summary of the hematoma change rate results 33

Table 5 3: Summary of the SSD results 33

ACRONYMS AND ABBREVIATIONS

Combination (The proposed method)

15 SPM Statistical Parametric Mapping

CHAPTER 1: INTRODUCTION

1.1 Research Background and Practical Significance

Intracranial Hemorrhage (ICH) is a life-threatening event that happens when a bleeding region occurs inside the human brain, leading to a high mortality rate without timely recognition In an ICH event, the presence of a bleeding area [1, 2], or hematoma region, could be inside the brain tissue (such as Intracerebral Hemorrhage [3], Cerebral microhemorrhage [2]), or between the brain tissue and the skull (such as Epidural [4], Subdural Hemorrhage [5]) The appearance of this unusual fluid leads to the continuous increment of the intracranial pressure [6], one of the main causes of the deadly event, strokes The common causes of ICH are brain damage by external factors [6], called traumatic ICH, and the consequence of vessel-related diseases such as hypertension and amyloid angiopathy, called non-traumatic ICH [6] According to the enabled population studies from 1980 to the end of 2008 summary [7], within one month after recognition, the incidence of ICH significantly increased and the mortality of 40.4 % Therefore, fast and accurate diagnosis assistant tools are needed to provide timely treatment planning Moreover, summarizing hematoma characteristics in a particular population of ICH patients could remarkably contribute to disease behavior analysis In this work, an image processing pipeline is deployed to support the abnormality's attribute summarizing on the ICH brain images dataset

Computed Tomography (CT) is considered as the standard imaging technique for ICH diagnosis due to its rapidity and clarity to show the basic characteristics of the hematoma such as location, bleeding extension [7] CT is an anatomical imaging method that can detail provide human organ structures based on their differences in X-ray energy absorption The image acquisition takes a few minutes to complete a brain CT scan This is significant shorter compared to another anatomical imaging, the Magnetic Resonance Imaging (MRI), which takes around 30 to 60 minutes for a brain scan In the acquired image, the hematoma region can be easily recognized as

a high gray level area, mostly in white color, standing out against the background structures of low pixel values Therefore, CT is the most effective method for fast ICH screening and monitoring In this study, the collected CT scan of the brain with ICH (in Chapter 3) is used for image processing deployment

Figure 1 1: The Intracranial Hemorrhage with a bleeding region occurs in the Computed Tomography (CT) and Magnetic Resonance Image (MRI) [2]

Image Registration is one of the most concerning research topics in the medical image analysis field due to its wide applications In clinical diagnosis, a medical image acquisition process could be implemented in different modalities, patients, or practice time To productively mine the diagnosis-support information, unlike the rigid transformation which just handles the object relocation, the non-rigid image registration is commonly conducted to map all the characteristics of anatomical structures in the Source image to the matching structures in the Reference image Depending on the application, the Source and Reference image could be taken by the same imaging modality (unimodal) or a different modality (multimodal), in the same patients at different times (intra-patient) or in various patients (inter-patient)

Techniques applied to medical image registration are diverse, ranging from traditional methods to deep learning (Section 2.1) In which, deep-learning-based approaches, especially the deep similarity-based model (Section 2.2), recently showed the great impact in this research field Consequently, an UNET-like architecture is expected to achieve a considerable performance on this dataset Image registration has considerable applications in medical image analysis, such as atlas generation [8, 9, 10], tumors monitoring [11], disease analysis or treatment assistant [12] However, there is no study accessing to the image registration of the ICH brain CT application In this work, we aim to propose the inter-patient image registration on this kind of medical data to warp the real-world collected disease image into the normal reference space, which could remarkably support further research on the characteristics of the hematoma region, provide additional anatomical information that the real-world Source data poorly contained In conclusion, the disease characteristic from collected data is summarized in the brain template, which eases the behavior analysis and other feature or information extraction steps

There are several challenges when deploying the registration process for the medical image with abnormalities If the pair of registration images contains the structural differences due to the appearance of tumors or lesions, it will cause many difficulties for the optimization process because mostly unimodal registration model relies on the structural similarity between the Source and the Reference image [13, 14, 10, 11] Several techniques have been proposed to minimize its affection, however, unreasonable retention [13, 11] and region losses [10, 14] are presented so that the abnormalities displacement is not optimized (Section 2-3) In this work, the new image registration pipeline (Section 4) included a normal tissue generation technique (Section 4 -3) has been proposed to overcome the challenge and fasten the optimization process This approach is expected to significantly reduce the confusion for the deep similarity-based model within the abnormal region in the registration process Moreover, evaluation metrics dedicated to this new image scenario has been proposed in Section 4.5

To successfully apply the medical processing tools from research experiments into the real world, several challenges need to be considered, especially the dataset shift [15] With 3D images collected from real-world experiment, it has several issues due to the high variance among the datasets In this work, a traditional transformation approach is considered to use as a pre-alignment process (Section 4-2) to reduce the variance in our collected dataset Therefore, this thesis aims to build a completed pipeline for the practical application of the ICH Brain CT image

The results presented the affection of the abnormal region volume to the hematoma displacement and normal structure mapping(Section 5.2), moreover, it proved that the proposed pipeline provide a remarkable performance on the ICH Brain CT dataset in both abnormal area and normal tissue structure registration, and outperformed the common pipeline especially in the case of high-volume and skull-close hematoma in abnormal region displacement (Section 5)

1.2 Research Contribution

The new image scenario is handled in this work, in which the ICH Brain CT image is transformed to the normal brain template Commonly, the modality used for brain image registration is Magnetic Resonance Image (MRI) due to the ability to present clearer anatomical structure [8, 9, 10, 11, 33, 38] Therefore, the CT scan registration is more challenge However, with the better performance in ICH screening, the study on the unimodal brain CT registration is needed for the disease analysis If a good ICH brain CT image registration tool is provided, it could remarkably support the further analysis on the hematoma behavior This study provides the investigation and the particular process to handle the image registration in this new image scenario

As other registration studies in the medical image with abnormality, the corresponding structure challenge is needed In this work, a new pipeline in Section 3 has been proposed to deal with this challenge and compared to the state-of-the-art pipeline for medical image registration The high-variance dataset is deployed in this

dis-work, which could make confusion for the similarity-based deep learning model in the optimization stage This work integrated a traditional transformation including with deep learning model and the normal tissue generation technique to carry out the real-world collected dataset

1.3 Research Goals

- Proposing and evaluating the end-to-end non-rigid image registration process for the new image scenario, the ICH Brain CT image, which considerably contributes to the intracranial bleeding region behavior analysis

- Being able to deal with the high-variance real-world data in practical application

- Overcoming the image differences for the deep similarity-based model, current considerable challenge, in the abnormality image registration application

CHAPTER 2: LITERATURE REVIEW

Due to its broad and effective application in medical image analysis, image registration methods are continuously developing from the iterative optimization algorithm of traditional techniques to learning-based approaches of deep learning technology in the Artificial Intelligence era (Section 2.1) Similarity-based methods have shown their considerable performance, especially in unimodal registration, because the resemblance of anatomical structure among the population of patients is useful for image texture matching The general concept and studies in this group of methods are present in Section 2.2 The registration process for the medical image with tumors or lesions is able to provide huge support for disease management and treatment However, there is a huge challenge in the image registration process for image with abnormalities Section 2.3 provides the summary of studies on abnormal image registration to highlight its application and challenges

2.1 From Traditional to Deep Learning Approaches

The Image Registration was initially developed based on a semi-manual iteration process to search for the optimal transformation field, which warps the input Source image into the target Reference space This optimization process relies on the transformation model, the similarity metrics, and the search strategy [16] Firstly, there are three types of registration transformation models, which are rigid, affine, and especially the non-rigid model Based on the type of model, the set of elementary parameters is initialized Secondly, this model carries on the similarity metric minimization process, based on the type of metric, the better the registration result could be in case the lower or the higher value Several similarity metrics have been widely used such as the Mean Squared Error (MSE) [17], Cross-Correlation (CC) [18, 19] or Mutual Information (MI) The final important factor in this process is the search algorithm In traditional techniques, the model iteratively finds and choose the best local minimum, which has the lowest total similarity loss There are several traditional non-rigid registration approaches and toolbox has been proposed and gave considerable performance such as Statistical Parametric Mapping (SPM) [20] or

Elastix [21] In the study of Klein et al [22] 14 different registration techniques were used to make the generalized comparison into the new subject populations, Symmetric Normalization (SyN) [18] and DARTEL [17] presented the highest performance Moreover, the Advanced Normalization Tools (ANTs) toolbox [19] is considered the standard toolbox for image registration in the traditional way due to its success in medical image registration application [23]

However, there are several problems when handling image registration using traditional methods In its optimization process, the transformation field is re-optimized from scratch when working on each unseen data [24] As a result, the testing duration significantly increases due to huge computation being needed and the performance of the model being unstable, it depends on the complexity of the testing case Nevertheless, the fast process is always a priority in the emergency department And in the behavior analysis application, a stable result is important to objectively make a comparison Moreover, in the high variance test data, it could be a challenge due to the complexity increasing among the dataset, then the reproductivity of this method might significantly decrease Therefore, a method to make the most of the available information among the population of patients from the dataset can be the solution for this high-variance data case

Nowadays in the era of AI, learning-based approaches are being continuously developed to overcome traditional method's limitations and improve the registration performance in the medical image field [25] Unlike traditional methods, Deep Learning approaches try to optimize the transformation field through the training process over the certain dataset Taking advantage of effective feature extraction layers, the similar characteristics of image anatomical structures are learned to productively generate the high reproductivity in the unseen dataset, even it is high variance data due to acquired information after learning from a population of patients In addition, there is no optimization in the testing stage of the deep learning model, so it remarkably reduces the testing time compared to traditional approaches [8, 26] Moreover, convolutional neural network architecture with the spatial transformer

networks [27] can rise the learning performance because of the ability to flexibly capture features from object orientation or variation from the input images by enabling local spatial manipulation [28] In summary, the deep learning approach can improve the reproductivity and performance of the registration process, especially in high variance datasets, and significantly reduce the testing time The brief comparison is summarized in Table 2.1

Table 2 1: Comparison of traditional standard and deep learning method

Standard Traditional Model (TSM)

Deep Learning Model (DL)

Common Points Obtaining the deformable vector field (in TSM) or the deformable model (in DL) by optimizing the similarity loss function, e.g Mean Squared Error

(MSE) or Cross-Correlation (CC)

Optimization Stage - The optimization process is directly performed after inputting a pair of Source and Reference images

- Iteratively conducting the search algorithm by the defined model to find the optimized DVF, which has the optimized similarity metrics value

- The optimization process is conducted in the training stage only, within the whole population of training data

- Fine-tuning the defined model parameters to optimize the loss function (with similarity metrics)

pre-Testing Stage For every testing pair, the optimization process takes a lot of time due to starting from scratch

Difficult to optimize on the unseen high variation between pair of images, especially in images with abnormalities

For every testing pair, the DVF is obtained in a short time due to the fast-predicting process of the optimized model

The model learned characteristics in the entire population of this kind of specific data, therefore possibly handling the unseen data with high variation

In recent years, the application of deep learning approaches for image registration has significantly increased [25] In which, there are few studies in Supervised [29] and the weakly supervised [30] method even it achieved the considerable performance However, the annotation for image registration is complicated, time-consuming, and difficult to obtain the accomplished label Consequently, the Unsupervised Approach is paid the most attention from the research community [25] Unsupervised Approach contains similarity-based and features-based methods In which, feature-based methods try to pre-define meaning features which possibly can handle structure matching between the Source and Reference data before implementing the training process On the other hand, similarity-based methods use similarity metrics like the traditional method, try to optimize the intensity-similar between the Source and Reference image This group of method is particularly explained in Section 2.2

In this research, to achieve the pre-alignment within the real-world dataset, a traditional approach is considered to apply, then a deep learning model optimizes the

final transformation field after learning on the population of patients from a certain dataset through the training process This work takes advantage of the lightweight traditional technique to partially reduce the variance among the dataset, then speed up the training process by deep learning model and improve its performance

2.2 Similarity-based Deep Learning Approaches

Similar to traditional methods, Similarity-based Deep Learning Approaches obtain the deformable vector field (DVF) by minimizing the intensity differences among the images by optimizing the similarity metric However, while the traditional method relies on the iterative search engine with the initialized model to find the local minimum, DL tries to fine-tune the pre-defined model parameters through the optimization process of the similarity loss function in the training stage Several similarity-based metrics are commonly used as the main loss function in the DL model, such as Mean Squared Error (MSE) [9, 31, 32], Cross-Correlation (CC) [3], or Normalized Cross-Correlation (NCC) [8, 34] Moreover, regularization terms are regularly used as a smoothing constraint in addition to the similarity loss [9, 11, 31] This constraint aims to penalize the loss within the transformation field and avoid high-value vectors that lead to large pixel displacement, and overfitting in the optimization process

Fully convolutional network (FCN) architecture is a powerful tool to evolve intensity-based learning for image registration applications This design is able to learn the pixel correspondence between the input pair of Source and Reference data through the feature extraction convolution function of encoder layers, then generates the DVF including the pixel displacement by the upsampling transpose convolution function of decoder layers For example, the 9-layer FCN model in Li et al work [34] was proposed and successfully applied to the Brain image registration by maximizing an image-wise similarity among the input datasets The FCN structure is also integrated into the Inverse-Consistent Deep Networks of Zhang et al work [9], which allows the input pair of images can symmetrically be transformed into another Besides the FCN, the Stacked Autoencoders structure with several layers of sparse

autoencoders is applied and demonstrated the remarkable performance [32] However, the most common structure has been applied in unimodal medical image registration is the UNET-like structure, which is the FCN including the skip connection to combine the features between the encoder and decoder path In the UNET-like structure of Kuang et al work [33], the FAIM model has been proposed with 2 additional convolution layers is designed to the transpose convolution of the decoder to improve the overall performance in brain image registration The Voxel Morph model [31] is inspired by the UNET structure, and additional layers have been added to combine image features and extract the final transformation field This is considered a standard deep learning model for image registration, especially for unimodal atlas generation applications, due to the high performance provided [8, 31] In this work, the UNET-like structure is considered to apply as the core model for unimodal and inter-patient brain CT image registration

2.3 Image Registration for Data with Abnormality

Depending on the application, the image scenario used for the registration process could be different Most research on medical image registration is deployed in the healthy subject to generate the organ atlas or further anatomical research [8, 9, 31] On the other hand, the image registration on the subject with tumors, lesions also remarkably contribute to the following treatment follow-up or the disease analysis In which, the intra-patient registration is commonly applied for the treatment follow-up [35] or evaluate the treatment adaptation [36] Therefore, the image used in this application is acquired from the same patient, before and after treatment, or between several treatment processes In contrast, the inter-patient registration tries to normalize the anatomical structure with abnormalities from a population of patients to generate the atlas for the disease [11, 13, 14, 37] So that in this case, the Source and Reference images are randomly picked from a certain dataset among disease subjects This group of works contributes to the research on the behavior of the tumor, such as its preferential locations In Parisot et al study [13], the conclusion about the privileged location of the brain low-grade gliomas tumor has been shown based on

the registration process In addition, the mapping process from tumors or lesion images into reference template anatomical space also contributes to the inter-patient abnormal region behavior analysis [10, 38] by the presence of the tumors or lesions in the standard space Consequently, the difficulty in these works is the difference between the Source image with abnormality and the Reference normal image In our application, the inter-patient registration process had been conducted from the ICH Brain CT image to the reference normal brain template

Several methods in both traditional and deep-learning approaches have been applied to abnormal image registration In traditional techniques, common registration toolbox, such as DARTEL and SPM, is productively applied to warp the image with abnormalities to the template [10, 14, 38] In which the tumor and non-tumor structures were separately transformed into the target space On the other hand, the hybrid approach defines the specific purpose for the included process In which, the machine learning process is responsible to search for the abnormal region while the traditional technique takes on the registration process [13, 37] This approach takes advantage of the machine learning classification model to classify the tumor region from the background tissue [13] or stimulate the abnormal region in the healthy subject template [37] to correspond with the Source image Moreover, a similarity-based Deep Learning model was applied to handle the brain gliomas MRI image registration [11] In Estienne et al work, the designed model contains 1 encoder to extract the similarity feature from pair of input images, while 2 decoders for tumor segmentation and image registration

Unlike registration in the healthy subject image, the huge challenge of the image with an abnormality is the dis-similarity within the tumor or lesion region and the healthy tissue To solve this challenge, several pipelines including tumor region processing have been proposed In Gooya et al study [37], the expectation-maximization algorithm was used to generate the tumoral region in the reference template, optimize this abnormal region to be as symmetrical as possible to the patient image However, the loss would arise in the stimulated region due to the difference

between normal tissue and its replacement with tumors The common pipeline for this challenge is tumor region relaxing and tumor seeding In tumor seeding techniques, 2 traditional methods are separately applied to transform the tumor region and the non-tumor region For example, several studies tried SPM for tumor registration and DARTEL or SyN for non-tumor registration [10, 14] Although achieving a considerable performance, there are many losses that occur in the abnormal region after processing On the other hand, in tumor relaxing techniques, the tumor region is defined by classification method [13] or segmentation method [11, 13] and lightly affected after the registration process Therefore, it leads to the abnormal region might remain the same while the object shape between the Source and Reference data is different Moreover, the deep learning model contain segmentation and registration [11] is resource-consuming

To deal with the huge challenge in the image with abnormality registration and overcome the limitation in current approaches, this work proposed a lightweight pipeline to process the lesion region in the ICH brain CT image, which can reduce the differences among the abnormal dataset, without any loss or disproportionate retention in hematoma region

CHAPTER 3: DATASET

In this study, the image acquired from the Computed Tomography (CT) was used for analysis With the ability to quickly and clearly present the bleeding region, CT is considered as the standard method for the ICH diagnosis in the Emergency Department (ED) of the hospital [6] Additionally, the brain CT image can productively provide robust information such as bleeding characteristics, hematoma volume, and the extension of the blood region Although the non-radiation imaging method, Magnetic Resonance Imaging (MRI), can safely provide a high-resolution anatomical image, the time required for a scan is significantly longer Moreover, in

Figure 3 1: The bleeding region annotation by physicians There is slices representative for 3 groups of hematoma volume with the hematoma to brain ratio is less than 0.05 (A), from 0.05 to 0.1 (B), and larger than 0.1 (C) The marked area

is represented by the green line surrounding the bleeding area

some cases, the performance of the MRI is just equivalent to the CT (Intracerebral Hemorrhage [3]), and worst at several ICH sub-type (Subarachnoid Hemorrhage [5])

due to difficulty to show blood matters by T1-weighted and T2-weighted MRI sequences [6] Therefore, CT remains the optimal choice for rapid ICH monitoring

The volume of the hematoma in this dataset is varied In ED, the bleeding region volume could be estimated based on the slice containing the largest area of hematoma [6] The hematoma to brain volume ratio of the test data is from 1% to 30% (Figure 3.2) In this experiment, the affection of the hematoma volume to the registration performance is investigated by the deep similarity-based model

Figure 3 2: The hematoma-brain shape volume ratio of all the test Brain ID

The dataset used in this study is the 3D Computed Tomography image of the Brain with an Intracranial Hemorrhage In this dataset, the extravascular bleeding regions present within different intracranial spaces [1], which means there are several different ICH types that would be handled in this research In which, accounts for the most cases are subtypes of Intracerebral Hemorrhage, causing 15-30% of strokes [3], such as Basal ganglia hemorrhage or Cerebellar hemorrhage, which are the most common type of Intracerebral Hemorrhage [3]

All ICH Brain CT images was collected in Kyushu University Hospital There are total of 157 brain CT scan of 157 ICH different patients has been analyzed in this work The annotation of bleeding region is provided by physicians This annotation is used for defining the region to conduct the image inpainting, which is explained in Section 4.3

CHAPTER 4: METHODOLOGY

To handle the image registration in this new image scenario, the Intracranial Hemorrhage brain CT image registration to the reference space and overcome the dis-similarity within the abnormality region, this work proposed the new pipeline including a pre-alignment and normal tissue generation technique in Section 4.1 In the pre-alignment stage, the affine transformation, which was applied to partly reduce the brain shape differences, is explained in Section 4.2 Following is the image inpainting process, which was used to reduce the dis-similarity within the hematoma region by generating the normal brain, is explained in Section 4.3 Then the deep similarity-based model, which was used to learn the robust information from the dataset, is shown in Section 4.4 Finally, the dedicated evaluation method to assess the results is introduced in Section 4.5

4.1 End-to-end pipeline

To overcome the dis-similarity and real-world dataset challenge, this work proposed an image registration process including an image inpainting technique and the affine transformation The detailed idea is explained in Figure 4.1

In the beginning, we have the Origin Source data of the collected ICH brain CT scan and Reference data of the normal brain template are correspondingly denoted as

So and R After processing the final result, the Transferred data, is achieved and denoted as T

Firstly, an image inpainting function fi is used to generate the normal tissue

within the hematoma region in Origin Source data So The abnormal region is defined

by ICH mask, which is annotated by doctors, denoted as Mo As equation (3.1) show,

the inpainting function fi take So and Mo as input, then the inpainted image SI is obtained In this work, an diffused-based inpainting function has been applied and explain in Section 4.2

Secondly, an pre-alignment transformation fucnction, denoted as fa , is applied

to both origin So (equation 3.2) and inpainted image SI (equantion 3.3) In this process, the Reference image is used as a target spatial orientation and object shape Afterwards, two pre-aligment volume, 𝑺𝒐𝒂 (pre-align origin image) and 𝑺𝑰𝒂(pre-align origin image), are achieved In this work, the affine transformation has been applied and explain in Section 4.3

Finally, the Transferred volume T is achieved by spatial warping the pre-align

Origin Source volume 𝑺𝒐𝒂 by the corresponding DVF

Figure 4 1: The proposed image registration pipeline In which the world origin source image experiences the inpainting process and the affine trans-formation before feeding into the deep learning model The output of deep learning

real-model, the deformable vector field, is used to warp the pre-align source data

4.2 Hematoma Inpainting

Image Inpainting is a common research topic in the medical image analysis field that aims to generate and stimulate the missing region by knowledge from the remaining region or learning process The image inpainting process is frequently applied for the restoration of damaged photos or object removal in photo editing [42] Deep learning approaches provide remarkable performance in medical image inpainting by Generative Adversarial Network and Convolutional Neural Network architecture [43] However, most studies handle the missing region generation by supervised method, in which the missing area is cut from the normal image In the traditional method, the path-based and diffused-based methods rely on the background information to stimulate the needed region In the path-based method, a path-by-path searching process is conducted to find the best matching structure in the remaining image In the diffused-based method, the missing region is filled by smoothly propagating image content from the boundary to the interior of its region [42] The stimulation process is defined by a function, such as Bi-Harmonic function [44], Fast Marching Method (FMM) [45], Navier-Stokes function [46]