INTRODUCTION
Research Background and Practical Significance
Intracranial Hemorrhage (ICH) is a life-threatening event that happens when a bleeding region occurs inside the human brain, leading to a high mortality rate without timely recognition In an ICH event, the presence of a bleeding area [1, 2], or hematoma region, could be inside the brain tissue (such as Intracerebral Hemorrhage [3], Cerebral microhemorrhage [2]), or between the brain tissue and the skull (such as Epidural [4], Subdural Hemorrhage [5]) The appearance of this unusual fluid leads to the continuous increment of the intracranial pressure [6], one of the main causes of the deadly event, strokes The common causes of ICH are brain damage by external factors [6], called traumatic ICH, and the consequence of vessel- related diseases such as hypertension and amyloid angiopathy, called non-traumatic ICH [6] According to the enabled population studies from 1980 to the end of 2008 summary [7], within one month after recognition, the incidence of ICH significantly increased and the mortality of 40.4 % Therefore, fast and accurate diagnosis assistant tools are needed to provide timely treatment planning Moreover, summarizing hematoma characteristics in a particular population of ICH patients could remarkably contribute to disease behavior analysis In this work, an image processing pipeline is deployed to support the abnormality's attribute summarizing on the ICH brain images dataset
Computed Tomography (CT) is considered as the standard imaging technique for ICH diagnosis due to its rapidity and clarity to show the basic characteristics of the hematoma such as location, bleeding extension [7] CT is an anatomical imaging method that can detail provide human organ structures based on their differences in X-ray energy absorption The image acquisition takes a few minutes to complete a brain CT scan This is significant shorter compared to another anatomical imaging, the Magnetic Resonance Imaging (MRI), which takes around 30 to 60 minutes for a brain scan In the acquired image, the hematoma region can be easily recognized as
2 a high gray level area, mostly in white color, standing out against the background structures of low pixel values Therefore, CT is the most effective method for fast ICH screening and monitoring In this study, the collected CT scan of the brain with ICH (in Chapter 3) is used for image processing deployment
Figure 1 1: The Intracranial Hemorrhage with a bleeding region occurs in the Computed Tomography (CT) and Magnetic Resonance Image (MRI) [2]
Image Registration is one of the most concerning research topics in the medical image analysis field due to its wide applications In clinical diagnosis, a medical image acquisition process could be implemented in different modalities, patients, or practice time To productively mine the diagnosis-support information, unlike the rigid transformation which just handles the object relocation, the non-rigid image registration is commonly conducted to map all the characteristics of anatomical structures in the Source image to the matching structures in the Reference image Depending on the application, the Source and Reference image could be taken by the same imaging modality (unimodal) or a different modality (multimodal), in the same patients at different times (intra-patient) or in various patients (inter-patient)
3 Techniques applied to medical image registration are diverse, ranging from traditional methods to deep learning (Section 2.1) In which, deep-learning-based approaches, especially the deep similarity-based model (Section 2.2), recently showed the great impact in this research field Consequently, an UNET-like architecture is expected to achieve a considerable performance on this dataset Image registration has considerable applications in medical image analysis, such as atlas generation [8, 9, 10], tumors monitoring [11], disease analysis or treatment assistant [12] However, there is no study accessing to the image registration of the ICH brain
CT application In this work, we aim to propose the inter-patient image registration on this kind of medical data to warp the real-world collected disease image into the normal reference space, which could remarkably support further research on the characteristics of the hematoma region, provide additional anatomical information that the real-world Source data poorly contained In conclusion, the disease characteristic from collected data is summarized in the brain template, which eases the behavior analysis and other feature or information extraction steps
There are several challenges when deploying the registration process for the medical image with abnormalities If the pair of registration images contains the structural differences due to the appearance of tumors or lesions, it will cause many difficulties for the optimization process because mostly unimodal registration model relies on the structural similarity between the Source and the Reference image [13,
14, 10, 11] Several techniques have been proposed to minimize its affection, however, unreasonable retention [13, 11] and region losses [10, 14] are presented so that the abnormalities displacement is not optimized (Section 2-3) In this work, the new image registration pipeline (Section 4) included a normal tissue generation technique (Section 4 -3) has been proposed to overcome the challenge and fasten the optimization process This approach is expected to significantly reduce the confusion for the deep similarity-based model within the abnormal region in the registration process Moreover, evaluation metrics dedicated to this new image scenario has been proposed in Section 4.5
To successfully apply the medical processing tools from research experiments into the real world, several challenges need to be considered, especially the dataset shift [15] With 3D images collected from real-world experiment, it has several issues due to the high variance among the datasets In this work, a traditional transformation approach is considered to use as a pre-alignment process (Section 4-2) to reduce the variance in our collected dataset Therefore, this thesis aims to build a completed pipeline for the practical application of the ICH Brain CT image
The results presented the affection of the abnormal region volume to the hematoma displacement and normal structure mapping(Section 5.2), moreover, it proved that the proposed pipeline provide a remarkable performance on the ICH Brain CT dataset in both abnormal area and normal tissue structure registration, and outperformed the common pipeline especially in the case of high-volume and skull-close hematoma in abnormal region displacement (Section 5).
Research Contribution
The new image scenario is handled in this work, in which the ICH Brain CT image is transformed to the normal brain template Commonly, the modality used for brain image registration is Magnetic Resonance Image (MRI) due to the ability to present clearer anatomical structure [8, 9, 10, 11, 33, 38] Therefore, the CT scan registration is more challenge However, with the better performance in ICH screening, the study on the unimodal brain CT registration is needed for the disease analysis If a good ICH brain CT image registration tool is provided, it could remarkably support the further analysis on the hematoma behavior This study provides the investigation and the particular process to handle the image registration in this new image scenario
As other registration studies in the medical image with abnormality, the dis- corresponding structure challenge is needed In this work, a new pipeline in Section
3 has been proposed to deal with this challenge and compared to the state-of-the-art pipeline for medical image registration The high-variance dataset is deployed in this
5 work, which could make confusion for the similarity-based deep learning model in the optimization stage This work integrated a traditional transformation including with deep learning model and the normal tissue generation technique to carry out the real-world collected dataset.
Research Goals
- Proposing and evaluating the end-to-end non-rigid image registration process for the new image scenario, the ICH Brain CT image, which considerably contributes to the intracranial bleeding region behavior analysis
- Being able to deal with the high-variance real-world data in practical application
- Overcoming the image differences for the deep similarity-based model, current considerable challenge, in the abnormality image registration application
LITERATURE REVIEW
From Traditional to Deep Learning Approaches
The Image Registration was initially developed based on a semi-manual iteration process to search for the optimal transformation field, which warps the input Source image into the target Reference space This optimization process relies on the transformation model, the similarity metrics, and the search strategy [16] Firstly, there are three types of registration transformation models, which are rigid, affine, and especially the non-rigid model Based on the type of model, the set of elementary parameters is initialized Secondly, this model carries on the similarity metric minimization process, based on the type of metric, the better the registration result could be in case the lower or the higher value Several similarity metrics have been widely used such as the Mean Squared Error (MSE) [17], Cross-Correlation (CC) [18, 19] or Mutual Information (MI) The final important factor in this process is the search algorithm In traditional techniques, the model iteratively finds and choose the best local minimum, which has the lowest total similarity loss There are several traditional non-rigid registration approaches and toolbox has been proposed and gave considerable performance such as Statistical Parametric Mapping (SPM) [20] or
7 Elastix [21] In the study of Klein et al [22] 14 different registration techniques were used to make the generalized comparison into the new subject populations, Symmetric Normalization (SyN) [18] and DARTEL [17] presented the highest performance Moreover, the Advanced Normalization Tools (ANTs) toolbox [19] is considered the standard toolbox for image registration in the traditional way due to its success in medical image registration application [23]
However, there are several problems when handling image registration using traditional methods In its optimization process, the transformation field is re- optimized from scratch when working on each unseen data [24] As a result, the testing duration significantly increases due to huge computation being needed and the performance of the model being unstable, it depends on the complexity of the testing case Nevertheless, the fast process is always a priority in the emergency department And in the behavior analysis application, a stable result is important to objectively make a comparison Moreover, in the high variance test data, it could be a challenge due to the complexity increasing among the dataset, then the reproductivity of this method might significantly decrease Therefore, a method to make the most of the available information among the population of patients from the dataset can be the solution for this high-variance data case
Nowadays in the era of AI, learning-based approaches are being continuously developed to overcome traditional method's limitations and improve the registration performance in the medical image field [25] Unlike traditional methods, Deep Learning approaches try to optimize the transformation field through the training process over the certain dataset Taking advantage of effective feature extraction layers, the similar characteristics of image anatomical structures are learned to productively generate the high reproductivity in the unseen dataset, even it is high variance data due to acquired information after learning from a population of patients
In addition, there is no optimization in the testing stage of the deep learning model, so it remarkably reduces the testing time compared to traditional approaches [8, 26] Moreover, convolutional neural network architecture with the spatial transformer
8 networks [27] can rise the learning performance because of the ability to flexibly capture features from object orientation or variation from the input images by enabling local spatial manipulation [28] In summary, the deep learning approach can improve the reproductivity and performance of the registration process, especially in high variance datasets, and significantly reduce the testing time The brief comparison is summarized in Table 2.1
Table 2 1: Comparison of traditional standard and deep learning method
Common Points Obtaining the deformable vector field (in TSM) or the deformable model (in DL) by optimizing the similarity loss function, e.g Mean Squared Error
(MSE) or Cross-Correlation (CC)
Optimization Stage - The optimization process is directly performed after inputting a pair of Source and Reference images
- Iteratively conducting the search algorithm by the defined model to find the optimized DVF, which has the optimized similarity metrics value
- The optimization process is conducted in the training stage only, within the whole population of training data
- Fine-tuning the pre- defined model parameters to optimize the loss function (with similarity metrics)
9 Testing Stage For every testing pair, the optimization process takes a lot of time due to starting from scratch
Difficult to optimize on the unseen high variation between pair of images, especially in images with abnormalities
For every testing pair, the DVF is obtained in a short time due to the fast- predicting process of the optimized model
The model learned characteristics in the entire population of this kind of specific data, therefore possibly handling the unseen data with high variation
In recent years, the application of deep learning approaches for image registration has significantly increased [25] In which, there are few studies in Supervised [29] and the weakly supervised [30] method even it achieved the considerable performance However, the annotation for image registration is complicated, time-consuming, and difficult to obtain the accomplished label Consequently, the Unsupervised Approach is paid the most attention from the research community [25] Unsupervised Approach contains similarity-based and features-based methods In which, feature-based methods try to pre-define meaning features which possibly can handle structure matching between the Source and Reference data before implementing the training process On the other hand, similarity-based methods use similarity metrics like the traditional method, try to optimize the intensity-similar between the Source and Reference image This group of method is particularly explained in Section 2.2
In this research, to achieve the pre-alignment within the real-world dataset, a traditional approach is considered to apply, then a deep learning model optimizes the
10 final transformation field after learning on the population of patients from a certain dataset through the training process This work takes advantage of the lightweight traditional technique to partially reduce the variance among the dataset, then speed up the training process by deep learning model and improve its performance.
Similarity-based Deep Learning Approaches
Similar to traditional methods, Similarity-based Deep Learning Approaches obtain the deformable vector field (DVF) by minimizing the intensity differences among the images by optimizing the similarity metric However, while the traditional method relies on the iterative search engine with the initialized model to find the local minimum, DL tries to fine-tune the pre-defined model parameters through the optimization process of the similarity loss function in the training stage Several similarity-based metrics are commonly used as the main loss function in the DL model, such as Mean Squared Error (MSE) [9, 31, 32], Cross-Correlation (CC) [3], or Normalized Cross-Correlation (NCC) [8, 34] Moreover, regularization terms are regularly used as a smoothing constraint in addition to the similarity loss [9, 11, 31] This constraint aims to penalize the loss within the transformation field and avoid high-value vectors that lead to large pixel displacement, and overfitting in the optimization process
Fully convolutional network (FCN) architecture is a powerful tool to evolve intensity-based learning for image registration applications This design is able to learn the pixel correspondence between the input pair of Source and Reference data through the feature extraction convolution function of encoder layers, then generates the DVF including the pixel displacement by the upsampling transpose convolution function of decoder layers For example, the 9-layer FCN model in Li et al work [34] was proposed and successfully applied to the Brain image registration by maximizing an image-wise similarity among the input datasets The FCN structure is also integrated into the Inverse-Consistent Deep Networks of Zhang et al work [9], which allows the input pair of images can symmetrically be transformed into another Besides the FCN, the Stacked Autoencoders structure with several layers of sparse
11 autoencoders is applied and demonstrated the remarkable performance [32] However, the most common structure has been applied in unimodal medical image registration is the UNET-like structure, which is the FCN including the skip connection to combine the features between the encoder and decoder path In the UNET-like structure of Kuang et al work [33], the FAIM model has been proposed with 2 additional convolution layers is designed to the transpose convolution of the decoder to improve the overall performance in brain image registration The Voxel Morph model [31] is inspired by the UNET structure, and additional layers have been added to combine image features and extract the final transformation field This is considered a standard deep learning model for image registration, especially for unimodal atlas generation applications, due to the high performance provided [8, 31]
In this work, the UNET-like structure is considered to apply as the core model for unimodal and inter-patient brain CT image registration.
Image Registration for Data with Abnormality
Depending on the application, the image scenario used for the registration process could be different Most research on medical image registration is deployed in the healthy subject to generate the organ atlas or further anatomical research [8, 9, 31] On the other hand, the image registration on the subject with tumors, lesions also remarkably contribute to the following treatment follow-up or the disease analysis
In which, the intra-patient registration is commonly applied for the treatment follow- up [35] or evaluate the treatment adaptation [36] Therefore, the image used in this application is acquired from the same patient, before and after treatment, or between several treatment processes In contrast, the inter-patient registration tries to normalize the anatomical structure with abnormalities from a population of patients to generate the atlas for the disease [11, 13, 14, 37] So that in this case, the Source and Reference images are randomly picked from a certain dataset among disease subjects This group of works contributes to the research on the behavior of the tumor, such as its preferential locations In Parisot et al study [13], the conclusion about the privileged location of the brain low-grade gliomas tumor has been shown based on
12 the registration process In addition, the mapping process from tumors or lesion images into reference template anatomical space also contributes to the inter-patient abnormal region behavior analysis [10, 38] by the presence of the tumors or lesions in the standard space Consequently, the difficulty in these works is the difference between the Source image with abnormality and the Reference normal image In our application, the inter-patient registration process had been conducted from the ICH Brain CT image to the reference normal brain template
Several methods in both traditional and deep-learning approaches have been applied to abnormal image registration In traditional techniques, common registration toolbox, such as DARTEL and SPM, is productively applied to warp the image with abnormalities to the template [10, 14, 38] In which the tumor and non- tumor structures were separately transformed into the target space On the other hand, the hybrid approach defines the specific purpose for the included process In which, the machine learning process is responsible to search for the abnormal region while the traditional technique takes on the registration process [13, 37] This approach takes advantage of the machine learning classification model to classify the tumor region from the background tissue [13] or stimulate the abnormal region in the healthy subject template [37] to correspond with the Source image Moreover, a similarity-based Deep Learning model was applied to handle the brain gliomas MRI image registration [11] In Estienne et al work, the designed model contains 1 encoder to extract the similarity feature from pair of input images, while 2 decoders for tumor segmentation and image registration
Unlike registration in the healthy subject image, the huge challenge of the image with an abnormality is the dis-similarity within the tumor or lesion region and the healthy tissue To solve this challenge, several pipelines including tumor region processing have been proposed In Gooya et al study [37], the expectation- maximization algorithm was used to generate the tumoral region in the reference template, optimize this abnormal region to be as symmetrical as possible to the patient image However, the loss would arise in the stimulated region due to the difference
13 between normal tissue and its replacement with tumors The common pipeline for this challenge is tumor region relaxing and tumor seeding In tumor seeding techniques, 2 traditional methods are separately applied to transform the tumor region and the non-tumor region For example, several studies tried SPM for tumor registration and DARTEL or SyN for non-tumor registration [10, 14] Although achieving a considerable performance, there are many losses that occur in the abnormal region after processing On the other hand, in tumor relaxing techniques, the tumor region is defined by classification method [13] or segmentation method [11, 13] and lightly affected after the registration process Therefore, it leads to the abnormal region might remain the same while the object shape between the Source and Reference data is different Moreover, the deep learning model contain segmentation and registration [11] is resource-consuming
To deal with the huge challenge in the image with abnormality registration and overcome the limitation in current approaches, this work proposed a lightweight pipeline to process the lesion region in the ICH brain CT image, which can reduce the differences among the abnormal dataset, without any loss or disproportionate retention in hematoma region
DATASET
In this study, the image acquired from the Computed Tomography (CT) was used for analysis With the ability to quickly and clearly present the bleeding region,
CT is considered as the standard method for the ICH diagnosis in the Emergency Department (ED) of the hospital [6] Additionally, the brain CT image can productively provide robust information such as bleeding characteristics, hematoma volume, and the extension of the blood region Although the non-radiation imaging method, Magnetic Resonance Imaging (MRI), can safely provide a high-resolution anatomical image, the time required for a scan is significantly longer Moreover, in
Figure 3 1: The bleeding region annotation by physicians There is slices representative for 3 groups of hematoma volume with the hematoma to brain ratio is less than 0.05 (A), from 0.05 to 0.1 (B), and larger than 0.1 (C) The marked area is represented by the green line surrounding the bleeding area some cases, the performance of the MRI is just equivalent to the CT (Intracerebral Hemorrhage [3]), and worst at several ICH sub-type (Subarachnoid Hemorrhage [5])
15 due to difficulty to show blood matters by T1-weighted and T2-weighted MRI sequences [6] Therefore, CT remains the optimal choice for rapid ICH monitoring
The volume of the hematoma in this dataset is varied In ED, the bleeding region volume could be estimated based on the slice containing the largest area of hematoma [6] The hematoma to brain volume ratio of the test data is from 1% to 30% (Figure 3.2) In this experiment, the affection of the hematoma volume to the registration performance is investigated by the deep similarity-based model
Figure 3 2: The hematoma-brain shape volume ratio of all the test Brain ID
The dataset used in this study is the 3D Computed Tomography image of the Brain with an Intracranial Hemorrhage In this dataset, the extravascular bleeding regions present within different intracranial spaces [1], which means there are several different ICH types that would be handled in this research In which, accounts for the most cases are subtypes of Intracerebral Hemorrhage, causing 15-30% of strokes [3], such as Basal ganglia hemorrhage or Cerebellar hemorrhage, which are the most common type of Intracerebral Hemorrhage [3]
16 All ICH Brain CT images was collected in Kyushu University Hospital There are total of 157 brain CT scan of 157 ICH different patients has been analyzed in this work The annotation of bleeding region is provided by physicians This annotation is used for defining the region to conduct the image inpainting, which is explained in Section 4.3
METHODOLOGY
End-to-end pipeline
To overcome the dis-similarity and real-world dataset challenge, this work proposed an image registration process including an image inpainting technique and the affine transformation The detailed idea is explained in Figure 4.1
In the beginning, we have the Origin Source data of the collected ICH brain CT scan and Reference data of the normal brain template are correspondingly denoted as
S o and R After processing the final result, the Transferred data, is achieved and denoted as T
Firstly, an image inpainting function f i is used to generate the normal tissue within the hematoma region in Origin Source data S o The abnormal region is defined by ICH mask, which is annotated by doctors, denoted as M o As equation (3.1) show, the inpainting function f i take S o and M o as input, then the inpainted image S I is obtained In this work, an diffused-based inpainting function has been applied and explain in Section 4.2
18 Secondly, an pre-alignment transformation fucnction, denoted as f a , is applied to both origin S o (equation 3.2) and inpainted image S I (equantion 3.3) In this process, the Reference image is used as a target spatial orientation and object shape Afterwards, two pre-aligment volume, 𝑺 𝒐 𝒂 (pre-align origin image) and 𝑺 𝑰 𝒂 (pre-align origin image), are achieved In this work, the affine transformation has been applied and explain in Section 4.3
Thirdly, an fully convolutional network model was used to learn from the similarity between the pre-align inpainted dataset 𝑺 𝑰 𝒂 and the reference brain space through R the training process (Equation 3.4) Then the deformable vector field, denoted as DVF , needed to transform the Source data is optimized from the trained model An similarity-based deep learning model is ultilized in this work, detail of this deep model is explained in Section 4.4
Finally, the Transferred volume T is achieved by spatial warping the pre-align Origin Source volume 𝑺 𝒐 𝒂 by the corresponding DVF
Figure 4 1: The proposed image registration pipeline In which the real- world origin source image experiences the inpainting process and the affine trans- formation before feeding into the deep learning model The output of deep learning model, the deformable vector field, is used to warp the pre-align source data.
Hematoma Inpainting
Image Inpainting is a common research topic in the medical image analysis field that aims to generate and stimulate the missing region by knowledge from the remaining region or learning process The image inpainting process is frequently applied for the restoration of damaged photos or object removal in photo editing [42] Deep learning approaches provide remarkable performance in medical image inpainting by Generative Adversarial Network and Convolutional Neural Network architecture [43] However, most studies handle the missing region generation by supervised method, in which the missing area is cut from the normal image In the traditional method, the path-based and diffused-based methods rely on the background information to stimulate the needed region In the path-based method, a path-by-path searching process is conducted to find the best matching structure in the remaining image In the diffused-based method, the missing region is filled by smoothly propagating image content from the boundary to the interior of its region [42] The stimulation process is defined by a function, such as Bi-Harmonic function [44], Fast Marching Method (FMM) [45], Navier-Stokes function [46]
Figure 4 2: The mask (down left) defines the hematoma region in the origin image dataset (top left) After inpainting process, the normal tissue within masked region are generated (right)
In this work, the image inpainting process is handled and aims to generate normal tissue within the hematoma region The dis-similarity between the abnormal Source data and the normal brain in the Reference data is a big challenge for the optimization process of the similarity-based deep learning model Therefore, if the hematoma region is considered as a normal structure in the optimization process, the registration performance could be well improved due to matching content with the corresponding structure in the normal subject brain template
Lightweight diffused-based methods has been investigated for the hematoma inpainting Because hematoma commonly locates in a certain brain region, such as Epidural Hemorrhage locates in epidural structure between the skull and brain tissue
21 [2], Intracerebral Hemorrhage locates inside the certain brain tissue [3] Therefore it possible to generate normal tissue within hematoma region by the surrounding tissue information There are 3 different diffused function had been applied in this work: the Bi-Harmonic function [44], the Fast Marching Method (FMM) [45], the Navier- Stokes function [46] Finally, the Bi-Harmonic-function were chosen to process the ICH brain CT dataset due to its best performance
The inpainting process is shown as Figure 4.3 Firstly, the hematoma annotation by physicians was used to define the stimulating region Then, normal brain tissues are generated within the annotated region by the Bi-Harmonic-function [44] after processing The inpainted dataset lately is used to train the non-rigid registration deep learning model in Section 4.4.
Image Pre-alignment by Affine Transformation
In real-world image registration, the pre-alignment process significantly contributes to the final registration performance, especially in the intensity-based method Due to the lack of pre-processing step, images among the real-world dataset highly vary in the object shape and spatial orientation Consequently, a pre-alignment approach is needed to reduce its variance, in which a normal brain template can be used as a reference space The unified object orientation and offset can be obtained by the initial rigid transformation, which quickly and approximately achieves the global location among the dataset [39, 40] However, in addition to the unified location, the global shape is also needed to improve the performance of the following non-rigid registration process As a result, the shearing and scaling processes are important to obtain the global shape without many changes in the original anatomical characteristics
Affine transformation is a geometric transformation technique that includes the step including a full set of the linear transformation: offset, rotation, translation, shearing, and scaling [41] This process maps the central object shape from the Source data into the reference shape, remains the line parallel relationship while distance
22 ratios along a line could not be preserved In this work, the affine transformation is considered to be used to improve the correspondence of the central object among the images in the real-world dataset, especially in spatial orientation and the brain shape (Figure 4.3) In addition, this pre-alignment process can remarkably reduce the training time and significantly improve the performance of the similarity-based deep learning model [26] thanks to the calculation complexity reduction
Figure 4 3: The affine transformation process The Original image dataset (left) is pre-alignment by affine transformation with the target space as the
The 3D Affine Transformation was set up and specialized in Elastix toolbox [21] to fit with the ICH Brain CT dataset The target brain shape and location are defined from the reference volume Then its function was used to pre-align the original source and the inpainted source dataset, before feeding to the deep learning model and the warping process The impact of this process was clearly shown in Section 6
Deep Similarity-Based Model
A deep similarity-based model is considered to handle the ICH brain image registration One of the state-of-the-art deep learning models named VoxelMorph [8] could provide a significant performance on the pre-align images, which is an UNET- like structure with encoder, decoder, and skip connection [47] Taking advance from the feature extraction function of convolutional layers, the intensity correspondence between the Source and the Reference images is acquired In other words, it can predict the pixel displacement from the Source to the Reference image The skip connection aims to reduce the gap between the up-sampling matrix and the compatible origin matrix In the end, a final layer is added to extract the DVF from the feature matrix This similarity-based model is expected to achieve considerable performance on the pre-align and unpainted ICH brain images
Figure 4 4: In the encoder (gray color), the stridden convolution was used One more layer was added to the decoder (blue color) to transform the results into a flow field The number in each layer represents the number of used kernels, the below number is the resolution respectively with the size of each slice after the convolution and deconvolution process
In this work, the training process aims to optimize the loss function including similarity metrics and a regularization term (Figure 4.5) In which, the Mean Square Error (MSE) was used to penalize the dis-similarity between the Source and Reference data Besides MSE, regularization plays an important role in the optimization process Which the efficient usage, this term is able to control the smoothness of the DVF, in another word it avoids large pixel movement in the DVF
24 conducted by the similarity optimization process In this work, the 𝑙 2 regularization was used to penalize the local spatial variations in the transformation field The main purpose of using regularization is to avoid excessive displacement of a single element to find the relevant region
Figure 4 5: Model Optimization In training stage, the output DVF is used to warp the Inpainted Source image The loss optimization process includes the MSE loss is calculated by the difference between the Warped image and the Reference image, and the regularization loss is calculated within the transformation field (the
Evaluation Method
The results achieved from the proposed method with hematoma inpainting process (DLIP) were compared with the state-of-the-art deep learning model and the affine transformation (DL) [26, 8], which was investigated in our previous work, to objectively evaluate its performance In the DL case, the Source dataset was directly pre-aligned by the Affine transformation and fed to the deep learning model
Performance evaluation in image registration is a challenge process due to the difficulty to achieve the good annotation [48] Especially in the real-world application, the annotation could not be quickly achieved and a expert in this kind of medical image also required for labeling stage In this work, due to lack of annotation in the real-world dataset, there are 3 evaluation metrics are considered in 2
25 performance factors: hematoma structure displacement and non-hematoma structure similarity These 2 factors were qualitatively evaluated by visual inspection, the common method for real-world image registration evaluation, and quantitatively evaluated by the hematoma change rate for hematoma displacement, and the Sum Squared Different (SSD) between the Transferred Image and the Reference Image for non-hematoma structure similarity
In the visual inspection, the results from DL-IP and DL process were visually compared follow by the hematoma displacement and the background structure differences between the Transferred Slices and the Reference Slices in both cases The slice to slice was displayed and compared The hematoma characteristics displacement such as texture, shape after the warping process from the Original Source data were considered Other anatomical structures were also experienced the slice correspondence checking between the Transferred slice of both cases and the Reference slice However, because this is a qualitative method therefore just the huge different performances can be seen after checking The result of visual inspection is presented in Section 5.1
Hematoma change rate (HCR) is the metric to check the region preservation after the transformation process Ideally, the change of the brain shape should be equal to the inside structure fluctuation This work proposed the HCR metric, inspired by the change ratio in Estienne et al work in the brain MRI image with Gliomas tumor [11], to investigate on the hematoma displacement The formula of HCR is proposed in Equation 4.1, it is the subtraction between the ratio of the hematoma region on the Transferred volume to the hematoma region on the Origin volume and the ratio of the brain shape on the Reference volume to the brain shape on the Origin volume The brain shape ratio is considered as the standard change ratio that other structure transformation for characteristics preservation The lower value of HCR, the better result it gives Additionally, the ideal value is 0, which means that the abnormality change ratio is the same as the standard transformation ratio The hematoma region segmentation on the Transferred volume was obtained by thresholding process along
26 with post-skull removing, while the origin hematoma region already annotated by the physician The results of HCR values are evaluated in 3 different groups for the visual inspection: small-volume, large-volume, and skull-close hematoma
• H T and H O are respectively the hematoma volume on the Transferred and Origin images respectively;
• B R and B O are the brain volume on the Reference and Origin images, respectively
The SSD metric was used to check the pixel similarity between the Transferred volume and the Reference volume except for the hematoma region This pixel difference could partially show the anatomical structure correspondence between the target space and the resulting image In addition to the hematoma displacement, the non-hematoma structure correlation is also important in medical image registration, it could show the insight of the abnormality characteristics related to other structures This metric was calculated slice to slice, and the total value was obtained for each volume To avoid the obvious difference within the abnormal region, the hematoma region was excluded in this calculation (Equation 4.2) This region is obtained by the thresholding processing as in the HCR calculation stage The average SSD value among the testing data is recorded for analysis
• T i , M i , R i are respectively the Transferred, Hematoma Mask, Reference images of sclice i
• (T i × (1 － M i ) is the Transferred slice excluded the hematoma region
• (R i × (1 － M i ) is the Reference slice excluded the hematoma region
Figure 4 6: SSD Calculation Process The Reference and the Transferred slice are excluded the hematoma region before calculating
RESULTS AND ANALYSIS
Visual Inspection
In visual inspection, three cases of hematoma, high, small volume hematoma and the hematoma locates near skull structures, is seprately evaluted and compared with the stateof-the-art deep learning model The hematoma texture and non- hematoma structure correspondence after registration were checked Check the hematoma region correspondence Several respresentative cases were shown in Figure 5.2 (high-volume hematoma), Figure 5.1 (small-volume hematoma), Figure 5.3 (skull-close hematoma) The results is summarized in Table 5.1
In small-volume hematoma cases, the hematoma displacement and non- hematoma structures correspondence was good and almost the same in 2 the proposed process pipeline and the standard pipeline The transferred structures highly correspond with the reference space, while the hematoma shape and texture are preserved
The huge difference in registration performance between the process with and without image inpainting In the deep learning model only case, the hematoma preservation is bad, the texture of hematoma is discontinuous, dispersed to neighbor normal tissues On the other hand, the deep learning model with image inpainting still kept the high performance in this case of hematoma As a result, the boundary
29 structures became stable, not be scattered by the wrong hematoma region displacement
Figure 5 1: Visual Inspection of small-volume hematoma Columns from left to right are:Original slices, Reference slices, Results of DL pipeline and Results of
Figure 5 2: Visual Inspection of high-volume hematoma Columns from left to right are: Original slices, Reference slices, Results of DL pipeline and Results of
Figure 5 3: Visual Inspection of hematoma locates near by the skull structure Columns from left to right are: Original slices, Reference slices, Results of DL pipeline and Results of DL-IP pipeline
The inpainting process also showed its efficiency in the cases of hematoma located near the skull structure High performance is achieved, especially in hematoma displacement and texture preservation While in the deep learning model, the hematoma structures experienced a huge loss, almost disappearing in some target slices due to the confusion between hematoma and bone structure
Table 5 1: Result summary of the visual inspection
Without Image Inpainting With Image Inpainting
• The bleeding region diffuses to the background structure
• The hematoma texture is not preserved, many losses occur surrounding the border of the bleeding region
• The hematoma region is stable in the specific area
• The hematoma texture is preserved, few losses are presented in the final results
Many losses and disappearances of the bleeding structure are presented due to skull and hematoma structure gray level similarity
Hematoma Displacement
As the visual inspection evaluation, the hematoma change rate (HCR) also was separately evaluated in these 3 cases depending on the volume and location The results also were compared with the Vorxel Morph [8] model results and shown in Table 5.2 Calculating the HCR could show insight into hematoma displacement and its characteristic preservation In addition, the results were presented through a line graph (Figure 5.4) of the change of the HCR with the magnitude of hematoma to investigate the affection of bleeding region volume on the hematoma registration performance
There is an increasing trend of the HCR value when the bleeding region gets bigger (Figure 5.4) It proved that the more the abnormal region increases, the higher HCR value is recorded This value remarkably increases in the DL line However, the HCR value is not increased much in the case of DL-IP In summary, increased
32 bleeding area volume affected the performance of the DL, while the DL-IP process significantly reduced this affection
In DL results, the abnormalities displacement performance is unstable, significantly depending on the location and the volume of the abnormal region In which, the small-volume hematoma registration achieved good results with the HCR is low, around 0.234 However, in big-volume and skull-near hematoma cases, very high HCR values were recorded, (HCR 0.5) This worst results may cause by large abnormal region in big-volume hematoma and similar gray level between bleeding region and skull structure in skull-close location As a result, the overall registration performance was poor
Figure 5 4: Line plot of the hematoma change rate (excluding skull-close cases) after processing by the Deep Learning alone (orange) and with Image
Inpainting (blue) follow by the hematoma volume
As showed in Table 5.2, the deep learning along with image inpainting is outperformed the other in in all evaluation groups In small-volume hematoma, DL-
IP achieved outstanding performance with the lowest HCR value (0.122) Unlike DL results, this model is not affected by the bleeding location, still keeps the high performance in the skullclose hematoma The HCR in big-volume hematoma lightly increases but still remarkably higher than the DL pipeline
Table 5 2: Summary of the hematoma change rate results
Non-hematoma structure mapping
Table 5 3: Summary of the SSD results
The Sum Square Difference (SSD) is the metrics to quantity the similarity between the Transferred and the Reference image except for the abnormal region The results of DL and DL-IP pipeline are shown in Table 5.3 The analysis on SSD metrics could partially provide the useful information for non-hematoma structure registration performance
Figure 5 5: The box plot of SSD value of DL (right) and DL-IP (left) pipeline
The overall result has shown in Figure 5.6 and Figure 5.5, the performance difference between DL and DL-IP is not considerable However, the average value presented in Table 5.3 shows that the SSD value of DL is slightly better than DL-IP
In addition, there is no relation between the SSD value and the hematoma volume as shown in Figure 5.6 The performance of both methods is unstable, it may be caused by the different contrast in the images
Figure 5 6: The Sum Square Difference value of DL (orange) and DL-IP
(blue) pipeline follow by the hematoma volume plot
DISCUSSION
Currently, although the deep similarity-based model presents a substantial performance in the medical image registration application, especially in unimodal non-rigid registration, many difficulties also are shown in the case of a medical image with tumors or lesions In this approach, the deformable vector field (DVF) to transform the Source data into the Reference space is optimized by minimizing the loss function including a similarity metric The optimized DVF would result in the Transferred data with high pixel correspondence to the Reference data However, in the image with tumors or lesions, the optimization process could face difficulties when searching for pixel correspondence within the abnormal region Therefore, if the model tries to specialize the tumor/lesion region with common brain tissue, the result will appear several issues in abnormalities preservation or displacement As the results shown in Section 5.1, the hematoma region is scattered to brain tissue, it is the cause of the optimization process to minimize the difference within the disease region In addition, the larger the volume of this area, the more it will affect subscription performance (5.4) This work proposed a idea to replace the abnormal region by generating the normal tissue This generation process simplifies the optimization process because the model easier to find the pixel correspondence among the normal tissue
Unlike other studies, to overcome the dis-similarity within the hematoma region between the ICH Source data and the Reference data, this work proposed a fast and efficient framework including the Image Inpainting to generate the normal tissue within the bleeding region In Parisot et al [13] and Estienne et al study [11], a segmentation process was included aim to relax the constraints within the predicted abnormal region Nevertheless, when relaxing the region of the tumor, it causes the tumor unchanged after mapping to Reference space, while the target brain space is different with the Source brain space Our work generates normal tissue within the abnormal region, it could provide better hematoma displacement due to matching structure geometric, map the bleeding to the correct anatomical structure of the target
37 space Moreover, one more decoder was used [11] to provide the gliomas tumor segmentation information, it is resource-consuming due to the increase in the number of deep learning model parameters The Image Inpainting approach used in this work is a low resource-usage tool, we can quickly achieve the Inpainted image without deep optimization Several traditional approaches also were used to separately map the tumor by SPM5 [20] and non-tumor structure by DARTEL [17] in Mutsaerts et al study [14] and Visser et al study [10] But limitations in reproductivity and processing time of this kind of method are needed to be resolved Our work takes advantage of the promptitude in the testing stage of the deep learning model and efficiently applies traditional image processing methods to do the pre-alignment and pre-processing stages In summary, the usage of image inpainting can better reduce the structure dissimilarity affection and give a remarkable performance while using few resources and quickly giving results
The hematoma locates nearby skull structures is the common case of ICH, which is present in Subdural hemorrhage [2, 1], which approximately occurs in 15% of all head trauma cases [5], or Epidural hemorrhage [4] Or in some cases of intracerebral hemorrhage, the hematoma locates close to the skull structure in the CT image So that if the image registration approach does not give high performance in this type of ICH, its practical applicability will be greatly reduced In similarity-based approaches, the optimization process tries to minimize the intensity difference, even within the abnormal structure Therefore, it merges the hematoma region to the skull structures due to the similar gray level among them Consequently, a significant loss is presented in both visual inspection results and HCR metric results of the DL model Unlike the general approach, the image inpainting is used aims to replace the abnormal region by generating normal tissue This process successfully avoids incorrect optimization, keeps and improves the registration performance in this kind of hematoma position As a result, the registration results in the visual inspection and hematoma change rate by the DL-IP method outperformed the VoxelMorph model in the skull-close hematoma case
Figure 6 1: The affection of hematoma volume on the hematoma displacement performance
Figure 6 2: The affection of hematoma location on the hematoma displacement performance
The hematoma inpainting process significantly improves the bleeding region displacement performance, which is significantly affected by the volume of the abnormal structure (Figure 6.1 and 6.2) In both DL and DL-IP approaches, the increasing HCR value can be seen when the volume of the abnormal region expands due to the huge difference inside Nevertheless, in DL-IP, just a minor increase has been presented in the HCR metric, and considerable performance is recorded in the visual inspection evaluation It proved that the dis-similarity could be solved by the image inpainting, resulting in the hematoma displacement being significantly improved However, the higher HCR value in big-volume hematoma may cause by
39 the current image inpainting technique efficiency Because in this processing, a lightweight inpainting method is a preferred use There are several cases of big- volume hematoma the diffused-based function could not well generate the normal tissue due to surrounding lesion Despite insignificantly higher HCR value in big- volume hematoma, the inpainting method will be improved to achieve the better abnormalities displacement and its characteristic displacement
Evaluation in the non-rigid medical image registration is a challenging process, especially in the image with abnormalities, due to difficulties in annotation [48] In this work, the visual inspection, hematoma change rate (HCR), and sum square difference (SSD) have been proposed and successfully applied to this imaging scenario As a common evaluation approach in real-world application [10], the visual inspection has been applied in this work to qualitatively assess the overall registration performance Through the anatomical structure correspondence and the abnormal region characteristics, the model performance can be visually appraised As result in Section 4, the qualitative visual inspection result summary also matches with the quantitative metrics output However, the quantitative metrics were additionally applied to provide a specific comparison The hematoma change rate metric showed that it is a reasonable approach to evaluate the bleeding region registration performance The bleeding region characteristics preservation is an important factor for the ICH brain registration and the hematoma change rate had proved that the DL-
IP pipeline is the better method to overcome the abnormal image registration In Visser et al study [10], the abnormal region is assessed by the overlap metric between the registered tumor and the annotated tumor Compared to our work, this annotation process is complex, time-consuming, and needs an expert to handle the labeling process Moreover, it could be a huge challenge to label the correct target tumor in the Reference space Besides the abnormal region, the healthy structure displacement is also important to assess the overall performance The dice score is frequently used to check the tissue overlapping between the Transferred and Reference data [11, 8] However in these works, they used the open-source and annotated dataset, so the dice
40 score can be easily accessed Moreover, the automatic segmentation tool could cause inaccuracies In this work the whole structure annotation is not pre-defined, therefore the specialized SSD metric was proposed for non-hematoma region registration performance Nevertheless, the SSD has a limitation when applied to this kind of dataset The image contains the difference in pixel-level even it is the same anatomical structure due to different contrast among the dataset Therefore, the SSD value is unstable between the image to image Although it was unstable, the non- hematoma structure registration performance could be seen through the boxplot in Figure 5.5, which showed comparable performance between both approaches
The affine transformation significantly contribute to the non-rigid model registration performance As shown in Figure 6.1 and the result of our previous work [26], it could reduce the variance in real-world dataset, fasten the optimization process and partly improve the final results Because the encoder-decoder structure learns the pixel correspondence by exploring the hierarchical features from the pair of input images, then it could not achieve high performance in the case of the high- variance dataset Consequently, in a real-world application, the affine transformation that is set up in this work can be used to optimize the registration process
In future work, the fully automatic pipeline could be implemented to apply for realworld applications and improve the current performance Firstly, the bleeding region mask used in the image inpainting step could be automatically obtained instead of the annotation by physicians Currently, many deep learning approaches have been proposed and achieved remarkable performance in the ICH brain CT image [49] therefore it could be applied to handle the hematoma segmentation Secondly, the image inpainting process could be handled by an unsupervised approach [50] to provide a better performance, due to the current in-optimized results of the inpainting model in big-volume hematoma cases Finally, in addition to SSD, an evaluation metric should be added to the experiment The landmark correspondence [48, 9] could be used, however, the annotation process is complex and needs huge support from experienced physicians In summary, more learning-based techniques could be
41 applied in the hematoma segmentation and image inpainting process, the landmark correspondence is considered to be a complementary metric for SSD value
Figure 6 3: Comparison with and without using Affine Transformation as a Pre-aligment process Columns from left to right are: Original, Pre-aligment, Reference, the results on Original, and the results on Pre-alignemnt images
CONCLUSION
This work has presented the comprehensive pipeline to deploy the non-rigid image registration for the Intracranial Hemorrhage Brain Computed Tomography image, the new image scenario in image regisration field, from pre-alignment the real-world collected data to global anatomical geometric obtainment In which, the affine transformation significantly reduced the object shape, spatial orientation difference among the dataset and contributed to the overall performance of the deep learning model The result of the hematoma change rate metric showed that the abnormal volume considerably affects the bleeding region displacement performance and texture preservation of the deep similaritybased model This work proposed a pipeline including the image inpainting to overcome the biggest challenge in the image registration of disease subjects, the dis-similarity within the abnormal region among the dataset The presented pipeline is proved that outperformed the standard similarity-based model in hematoma characteristics displacement and preservation, while mostly remaining a similar performance in non-hematoma structure registration In conclusion, the proposed pipeline achieved remarkable performance in this kind of new image scenario, which could considerably support the further study on the Intracranial Hemorrhage behavior analysis
1 N T Le, S Kobashi, K Arimura, K Iihara and S Inoue, "Intracranial Hemorrhage Brain Image Non-rigid Registration from Real-world Dataset to Reference Space," 2021 Joint 10th International Conference on Informatics,
Electronics & Vision (ICIEV) and 2021 5th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Kitakyushu, Japan, 2021, pp 1-7, doi: 10.1109/ICIEVicIVPR52578.2021.9564140
[1] D Puyó and F Gaillard, “Intracranial haemorrhage,” Radiopaedia.org, May
[2] J J Heit, Michael, and M Wintermark, “Imaging of Intracranial Hemorrhage,” Journal of Stroke, vol 19, no 1, pp 11–27, Jan 2017, doi: 10.5853/jos.2016.00563
[3] W C Ziai and J R Carhuapoma, “Intracerebral Hemorrhage,” Continuum, vol 24, no 6, pp 1603–1622, Dec 2018, doi: 10.1212/con.0000000000000672
[4] B Rasuli and F Gaillard, “Extradural haemorrhage,” Radiopaedia.org, May
[5] D J Bell, “Subdural haemorrhage,” radiopaedia.org, Apr 2023, doi:
[6] J A Caceres and J N Goldstein, “Intracranial Hemorrhage,” Emergency Medicine Clinics of North America, vol 30, no 3, pp 771–794, Aug 2012, doi: 10.1016/j.emc.2012.06.003
[7] J S Smith, E F Chang, Guy Rosenthal, M Meeker, C V Koch, Geoffrey
T Manley, and M C Holland., “The Role of Early Follow-Up Computed Tomography Imaging in the Management of Traumatic Brain Injury Patients with Intracranial Hemorrhage,” Journal of Trauma-injury Infection and Critical Care, vol 63, no 1, pp 75–82, Jul 2007, doi:
[8] G Balakrishnan, A Y Zhao, M R Sabuncu, A V Dalca, and J V Guttag,
An Unsupervised Learning Model for Deformable Medical Image Registration 2018 doi: 10.1109/cvpr.2018.00964
[9] J Zhang, “Inverse-Consistent Deep Networks for Unsupervised Deformable Image Registration,” arXiv.org, Sep 10, 2018 https://arxiv.org/abs/1809.03443
45 [10] M R Visser et al., “Accurate MR image registration to anatomical reference space for diffuse glioma,” Frontiers in Neuroscience, vol 14, Jun 2020, doi: 10.3389/fnins.2020.00585
[11] T Estienne et al., “Deep Learning-Based Concurrent brain registration and tumor segmentation,” Frontiers in Computational Neuroscience, vol 14, Mar 2020, doi: 10.3389/fncom.2020.00017
[12] A V D Hoorn, J L Yan, T J Larkin, N R Boonzaier, T Matys, and S J Price, “Validation of a semi-automatic co-registration of MRI scans in patients with brain tumors during treatment follow-up,” NMR in Biomedicine, vol 29, no 7, pp 882–889, Apr 2016, doi: 10.1002/nbm.3538
[13] S Parisot, H Duffau, S Chemouny, and N Paragios, “Joint Tumor Segmentation and Dense Deformable Registration of Brain MR Images,” in
[14] H.-J Mutsaerts et al., “Comparison of arterial spin labeling registration strategies in the multi-center GENetic frontotemporal dementia initiative (GENFI),” Journal of Magnetic Resonance Imaging, vol 47, no 1, pp 131–
[15] C B Kelly, A Karthikesalingam, M Suleyman, G S Corrado, and D King,
“Key challenges for delivering clinical impact with artificial intelligence,”
BMC Medicine, vol 17, no 1, Oct 2019, doi: 10.1186/s12916-019-1426-2.
[16] C Xiang, A Diaz-Pinto, N Ravikumar, and A F Frangi, “Deep learning in medical image registration,” Progress in Biomedical Engineering, Dec
[17] J Ashburner, “A fast diffeomorphic image registration algorithm,”
NeuroImage, vol 38, no 1, pp 95–113, Oct 2007, doi: 10.1016/j.neuroimage.2007.07.007.
[18] B B Avants, C J Epstein, M Grossman, and J C Gee, “Symmetric diffeomorphic image registration with cross-correlation: Evaluating
46 automated labeling of elderly and neurodegenerative brain,” Medical Image
Analysis, vol 12, no 1, pp 26–41, Feb 2008, doi: 10.1016/j.media.2007.06.004.
[19] B B Avants, N J Tustison, and H J Johnson, “Advanced Normalization Tools (ANTs),” Insight Journal, vol 2, no 1, pp 1–35, Jul 2009.
[20] W D Penny, K Friston, J Ashburner, S J Kiebel, and N Thomas,
Statistical Parametric Mapping: The analysis of functional brain images
2007 [Online] Available: http://ci.nii.ac.jp/ncid/BA82055992
[21] S R Klein, M Staring, K C Murphy, M A Viergever, and J P W Pluim,
“elastix: A Toolbox for Intensity-Based Medical Image Registration,” IEEE
Transactions on Medical Imaging, vol 29, no 1, pp 196–205, Jan 2010, doi: 10.1109/tmi.2009.2035616
[22] A Klein et al., “Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration,” NeuroImage, vol 46, no 3, pp 786–802, Jul 2009, doi: 10.1016/j.neuroimage.2008.12.037
[23] B B Avants, N J Tustison, G Song, P J Cook, A Klein, and J C Gee,
“A reproducible evaluation of ANTs similarity metric performance in brain image registration,” NeuroImage, vol 54, no 3, pp 2033–2044, Feb 2011, doi: 10.1016/j.neuroimage.2010.09.025
[24] F P M Oliveira and J M R S Tavares, “Medical image registration: a review,” Computer Methods in Biomechanics and Biomedical Engineering, vol 17, no 2, pp 73–93, Jan 2014, doi: 10.1080/10255842.2012.670855 [25] G Haskins, U Kruger, and P Yan, “Deep learning in medical image registration: a survey,” Machine Vision and Applications, vol 31, no 1–2, Jan 2020, doi: 10.1007/s00138-020-01060-x
[26] N T Le, S Kobashi, K Arimura, K Iihara and S Inoue, "Intracranial Hemorrhage Brain Image Non-rigid Registration from Real-world Dataset to Reference Space," 2021 Joint 10th International Conference on Informatics,
Electronics & Vision (ICIEV) and 2021 5th International Conference on
Imaging, Vision & Pattern Recognition (icIVPR), Kitakyushu, Japan, 2021, pp 1-7, doi: 10.1109/ICIEVicIVPR52578.2021.9564140
[27] M Jaderberg, K Simonyan, A Zisserman, and k kavukcuoglu “Spatial transformer networks” In C Cortes, N Lawrence, D Lee, M Sugiyama, and R Garnett, editors, Advances in Neural Information Processing Systems, vol 28, Curran Associates, Inc., 2015
[28] Y Fu, Y Lei, T Wang, W J Curran, T Liu, and X Yang, “Deep learning in medical image registration: a review,” Physics in Medicine and Biology, vol 65, no 20, p 20TR01, Oct 2020, doi: 10.1088/1361-6560/ab843e [29] X Yang, R Kwitt, M Styner, and M Niethammer, “Quicksilver: Fast predictive image registration – A deep learning approach,” NeuroImage, vol
158, pp 378–396, Sep 2017, doi: 10.1016/j.neuroimage.2017.07.008 [30] J Fan, X Cao, P T Yap, and D Shen, “BIRNet: Brain image registration using dual-supervised fully convolutional networks,” Medical Image Analysis, vol 54, pp 193–206, May 2019, doi: 10.1016/j.media.2019.03.006
[31] G Balakrishnan, A Y Zhao, M R Sabuncu, J V Guttag, and A V Dalca,
“VoxelMorph: A Learning Framework for Deformable Medical Image Registration,” IEEE Transactions on Medical Imaging, vol 38, no 8, pp
[32] G Wu, M.-J Kim, Q Wang, B C Munsell, and D Shen, “Scalable High- Performance Image Registration Framework by Unsupervised Deep Feature Representations Learning,” IEEE Transactions on Biomedical Engineering, vol 63, no 7, pp 1505–1516, Jul 2016, doi: 10.1109/tbme.2015.2496253 [33] D Kuang and T Schmah, “FAIM – A ConvNet Method for Unsupervised 3D Medical Image Registration,” in Lecture Notes in Computer Science, Springer Science+Business Media, 2019, pp 646–654 doi: 10.1007/978-3-
48 [34] H Li and Y Fan, “Non-rigid image registration using self-supervised fully convolutional networks without training data,” In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp 1075–
[35] Y Li, Y Kubota, M Okamoto, S Shiba, S Okazaki, T Matsui, S Komatsu, and T Ohno, “Determination of Deformable Image Registration Algorithms for Accumulating Dose in Carbon-ion Radiotherapy for Pancreatic Cancer,”
Anticancer Research, vol 41, no 2, pp 835–843, Jan 2021, doi:
[36] P Palaniappan et al., “Deformable image registration of the treatment planning CT with proton radiographies in perspective of adaptive proton therapy,” Physics in Medicine and Biology, vol 66, no 4, p 045008, Feb
[37] A Gooya et al., “GLISTR: Glioma Image Segmentation and Registration,” IEEE Transactions on Medical Imaging, vol 31, no 10, pp 1941–1954, Oct 2012, doi: 10.1109/tmi.2012.2210558
[38] A Mohamed, E I Zacharaki, D Shen, and C Davatzikos, “Deformable registration of brain tumor images via a statistical model of tumor-induced deformation,” Medical Image Analysis, vol 10, no 5, pp 752–763, Oct
[39] W Zhou, L Zhang, Y Xie, and C.-H Liang, “A Novel Technique for Prealignment in Multimodality Medical Image Registration,” BioMed Research International, vol 2014, pp 1–16, Jan 2014, doi:
[40] Y.-M Zhu, “Automatic pre-alignment for registration of medical images.” U.S Patent US20110116701A1, Aug 04, 2008
[41] K Kửser, “Affine Registration,” in Springer eBooks, 2014, pp 22–25 doi: 10.1007/978-0-387-31439-6_122
49 [42] O Elharrouss, N Almaadeed, S Al-Maadeed, and Y Akbari, “Image Inpainting: A Review,” Neural Processing Letters, vol 51, no 2, pp 2007–
[43] M.-T Tran, S.-H Kim, H.-J Yang, and G.-S Lee, “Multi-Task Learning for Medical Image Inpainting Based on Organ Boundary Awareness,” Applied
Sciences, vol 11, no 9, p 4247, May 2021, doi: 10.3390/app11094247
[44] S B Damelin and N S Hoang, “On Surface Completion and Image Inpainting by Biharmonic Functions: Numerical Aspects,” International Journal of Mathematics and Mathematical Sciences, vol 2018, pp 1–8, Feb
[45] A Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” Journal of Graphics, Gpu, and Game Tools, vol 9, no 1, pp 23–
[46] M Bertalmio, A.L Bertozzi, and G Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” In Proceedings of the 2001 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition CVPR
[47] O Ronneberger, P Fischer, and T Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Lecture Notes in Computer Science,
Springer Science+Business Media, 2015, pp 234–241 doi: 10.1007/978-3-
[48] Keelin Murphy et al “Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge,” IEEE Transactions on Medical Imaging, vol
30, no 11, pp 1901–1920, May 2011, doi: 10.1109/tmi.2011.2158349 [49] M D Hssayeni, M S Croock, A D Salman, H F Al-Khafaji, Z A Yahya, and B Ghoraani, “Intracranial Hemorrhage Segmentation Using a Deep Convolutional Model,” Data, vol 5, no 1, p 14, Feb 2020, doi:
50 [50] B Nguyen, “Unsupervised Region-based Anomaly Detection in Brain MRI with Adversarial Image Inpainting,” arXiv.org, Oct 05, 2020 https://arxiv.org/abs/2010.01942