Representation learning for knowledge graph using deep learning methods = học biểu diễn cho đồ thị tri thức sử dụng các kỹ thuật học sâu

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY MASTER THESIS Representation learning for Knowledge Graph using Deep Learning methods TONG VAN VINH Vinh.TV202705M@sis.hust.edu.vn School of Information and Communication Technology Supervisor: Assoc Prof Huynh Quyet Thang Supervisor’s signature Institution: School of Information and Communication Technology January 12, 2022 Graduation Thesis Assignment Name: Tong Van Vinh Phone: +84354095052 Email: Vinh.TV202705M@sis.hust.edu.vn; vinhbachkhoait@gmail.com Class: 20BKHDL-E Affiliation: Hanoi University of Science and Technology Tong Van Vinh - hereby warrants that the work and presentation in this thesis performed by myself under the supervision of Assoc Prof Huynh Quyet Thang All the results presented in this thesis are truthful and are not copied from any other works All references in this thesis including images, tables, figures, and quotes are clearly and fully documented in the bibliography I will take full responsibility for even one copy that violates school regulations Student Signature and Name Acknowledgement I would like to acknowledge and give my warmest thanks to my supervisor, Assoc Prof Huynh Quyet Thang inspired me a lot in my research career path I also thank Mr Huynh Thanh Trung, Dr Nguyen Quoc Viet Hung, and Dr Nguyen Thanh Tam for supporting me in giving birth to my brainchild and challenging myself by submitting it to the top-tier conferences I would also like to thank my committee members for letting my defense be an enjoyable moment and for your thoughtful comments and suggestion I would also like to give a special thanks to my girlfriend Thu Hue and my family as a whole for their mental support during my thesis writing process There is nothing to touch my love to you Moreover, in the absence of my friends, Tien Thanh, Trong Tuan, Hong Ngoc, Hieu Tran, Minh Tam, Quang Huy, Quang Thang, Ngo The Huan, I could hardly melt away all the tension from my work Thanks for always accompanying me through ups and downs Finally, this work was funded by Vingroup and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2020.ThS.BK.07 I enormously appreciate all the financial support from Vingroup, allowing me to stay focused on my research without worrying about my financial burden Abstract Knowledge graphs (KGs) have received significant attention in recent years Gaining more profound insight into the structure of knowledge graphs allows us to tackle many challenging tasks, such as knowledge graph alignment, knowledge graph completion, and question answering Recently, deep learning methods using the representation of knowledge graph entities (nodes) and relations (edges) in vector space have gained traction from the research community because of their flexibility and prospective performance The best way to evaluate how good a representation learning method is to use that representation to solve real-world tasks In terms of knowledge graphs, we can rank methods by their performance on tasks such as knowledge graph completion (KGC) or knowledge graph alignment (KGA) However, many research challenges still exist, such as enhancing the accuracy or simultaneously solving multiple tasks With such motivation, in the scope of our Master work, we address the three groups of crucial challenges in knowledge graph representation, namely (i) challenges in enhancing KGC performance, (ii) challenges in enhancing KGA performance, and (iii) challenges in enhancing both KGC and KGA simultaneously For the first class of challenges, we develop a model named NoGE which takes take advantage of not only the power of Graph Neural Networks (GNNs) but also the expressive power of quaternion vector space and the co-occurrence statistics of elements in KGs to achieve SOTA performance on the KGC task Moving to the second challenge group, we propose EMGCN, a special GNN architecture designed to exploit different types of information to better the final alignment results Finally, we propose IKAMI, the first multitask-learning model, to solve the two tasks simultaneously Our proposed techniques improve upon the state-ofthe-art for different tasks and thus cover an extensive range of applications Student Signature and Name TABLE OF CONTENTS CHAPTER INRODUCTION 1.1 Knowledge Graphs (KGs) 1.2 Knowledge graph completion and knowledge graph alignment 1.2.1 Knowledge graph completion 1.2.2 Knowledge graph alignment 1.2.3 The relation between completion and alignment 1.3 Research challenges 1.3.1 Handle knowledge graph completion challenges 1.3.2 Handle knowledge graph alignment challenges 1.3.3 Handle the challenges of solving the two task simultaneously 1.4 Thesis methodology 1.5 Contributions and Thesis Outline 1.6 Selected Publications CHAPTER BACKGROUND 11 2.1 Graph Convolutional Networks (GCNs) 11 2.2 Knowledge Graph Completion background 12 2.2.1 Incomplete knowledge graphs 12 2.2.2 Knowledge graph completion models 12 2.3 Knowledge Graph Alignment background 15 2.3.1 Previous approaches 15 2.3.2 Alignment constraints 16 2.3.3 Incomplete knowledge graph alignment 18 CHAPTER ENHANCING KNOWLEDGE GRAPH COMPLETION PERFORMANCE 19 3.1 Introduction 19 3.2 Dual quaternion background 20 3.3 NoGE 21 3.4 Experimental Results 23 3.4.1 Experiment setup 23 3.4.2 Main results 25 CHAPTER ENHANCING KNOWLEDGE GRAPH ALIGNMENT PERFORMANCE 27 4.1 Introduction 27 4.2 Overview of the Proposed Approach 28 4.2.1 Motivation 28 4.2.2 The entity alignment framework 29 4.3 Relation-aware Multi-order Embedding 30 4.3.1 GCN-based embedding model 30 4.3.2 Loss function 31 4.4 Alignment Instantiation 32 4.4.1 Single-order alignment matrices 33 4.4.2 Multi-order alignment matrix 33 4.4.3 Attribute Alignment 33 4.4.4 Puting It All Together 35 4.5 Empirical evaluation 35 4.5.1 Experimental setup 36 4.5.2 End-to-end comparison 38 4.5.3 Efficiency Test 39 4.5.4 Ablation Test 40 4.5.5 Hyperparameter sensitivity 41 4.5.6 Robustness to constraint violations 43 CHAPTER MULTITASK LEARNING FOR KNOWLEDGE GRAPH COMPLETION AND KNOWLEDGE GRAPH ALIGNMENT 45 5.1 Introduction 45 5.2 Incomplete Knowledge Graph Alignment 46 5.2.1 Challenges 46 5.2.2 Outline of the Alignment Process 47 5.3 Feature channel models 50 5.3.1 Pre-processing 50 5.3.2 Transitivity-based channel 50 5.3.3 Proximity-based channel 52 5.4 The complete alignment process 55 5.4.1 Alignment instantiation 55 5.4.2 Missing triples recovery 56 5.4.3 Link-augmented training process 58 5.5 Evaluation 59 5.5.1 Experimental Setup 59 5.5.2 End-to-end comparison 62 5.5.3 Robustness to KGs incompleteness 64 5.5.4 Saving of labelling effort 64 5.5.5 Qualitative evidences 66 CHAPTER CONCLUSION 68 LIST OF FIGURES 1.1 An illustration of knowledge graph 1.2 An example of knowledge graph completion 1.3 An example of knowledge graph entity alignment 1.4 Aligning incomplete KGs across domains 1.5 Encoder Decoder architecture for GNN based models 2.1 CNN and GCN comparison [37] 11 3.1 An illustration of our proposed NoGE 21 4.1 Overview of EMGCN framework 28 4.2 Computation time 38 4.3 Different supervision percentage 38 4.4 #GCN-layers 41 4.5 Embedding dim 41 4.6 Robustness to violations of entity consistency 44 4.7 Robustness to violations of relation consistency 44 5.1 Framework Overview 49 5.2 Running time (in log scale) on different datasets 63 5.3 Saving of labelling effort for entity alignment on D-W-V1 test set 5.4 Robustness of graph alignment models against noise on EN-DEV2 test set 65 5.5 Attention visualisation (EN-FR-V1 dataset) The model pays less attention to noisy relations 66 5.6 KGC performance comparison between TransE and IKAMI during training 67 65 LIST OF TABLES 3.1 Statistics of the experimental datasets 23 3.2 Experimental results on the CoDEx test sets 25 3.3 Ablation results on the validation sets 26 4.1 Statistics of real-world datasets 36 4.2 End to end comparison 38 4.3 Ablation Test 39 4.4 Different weighting schemes of GCN layers 42 4.5 Effects of similarity matrix coefficients 42 5.1 Summary of notation used 47 5.2 Dataset statistics for KG alignment 59 5.3 End-to-end KG alignment performance (bold: winner, underline: first runner-up) 62 5.4 Ablation study 63 5.5 Knowledge Graph Completion performance 63 5.6 Correct aligned relations in EN↔FR KGs 66 Table 5.3: End-to-end KG alignment performance (bold: winner, underline: first runner-up) Dataset Ver Metric V1 EN-DE V2 V1 EN-FR V2 V1 D-W V2 V1 D-Y V2 5.5.2 MTransE GCN-A BootEA RDGCN Alinet JAPE KDcoE MultiKE IKAMI Hit@1 Hit@10 MRR MR Hit@1 Hit@10 MRR MR 307 610 407 223.9 193 431 274 193.5 481 753 571 352.3 534 780 618 108.0 675 865 740 125.7 833 936 869 16.2 830 915 859 67.1 833 936 860 74.8 609 829 681 216.7 816 931 857 71.1 288 607 394 140.6 167 415 250 139.9 529 679 580 124.8 649 835 715 16.0 756 828 782 91.5 755 835 784 45.2 949 991 952 8.4 964 992 975 3.0 Hit@1 Hit@10 MRR MR Hit@1 Hit@10 MRR MR 247 563 351 251.9 240 240 336 206.0 338 680 451 562.2 414 796 542 131.3 507 794 603 227.7 660 906 745 25.7 755 880 800 156.1 847 934 880 61.7 387 829 487 483.2 580 877 689 94.0 263 595 372 175.6 292 624 402 89.1 581 721 628 197.0 730 869 778 27.3 749 843 782 97.8 864 924 885 12.1 907 992 935 7.2 978 998 986 1.2 Hit@1 Hit@10 MRR MR Hit@1 Hit@10 MRR MR 259 541 354 331.1 271 584 376 146.0 364 648 461 765.3 506 818 612 146.0 572 793 649 286.3 821 950 867 18.4 515 717 584 508.5 623 805 684 229.3 470 703 552 575.7 741 925 807 72.1 250 541 348 243.7 262 581 368 99.0 247 473 325 730.2 405 720 515 91.7 411 583 468 275.4 495 724 569 38.6 724 911 793 25.3 857 984 900 3.0 Hit@1 Hit@10 MRR MR Hit@1 Hit@10 MRR MR 463 733 559 245.6 443 707 533 85.2 465 661 536 1113.7 875 963 907 47.1 739 871 788 365.1 958 990 969 4.8 931 974 949 17.8 936 973 950 13.8 569 726 630 532.6 951 989 965 5.6 469 747 567 211.2 945 626 440 82.5 661 797 710 133.3 895 984 932 2.1 903 950 920 19.5 856 927 881 10.0 967 990 976 3.1 987 998 992 1.1 End-to-end comparison We report an end-to-end comparison of our alignment model against baseline methods on the real-world datasets in Table 5.3 It can be seen that our model outperformed the others in all scenarios Though using a multi-channel mechanism as GCN-Align and RDGCN, the gain of up to 10-20% of Hit@1 and MRR demonstrated the efficiency of relation-aware integration and knowledge transfer mechanism proposed in our work IKAMI, especially for denser version (v2) of the datasets Also, we achieved much higher results than the transitivity-based model MTransE, which justified the superiority of our proximity GNN-based model Among the baselines, RDGCN and BootEA, the two deep embedding-based techniques, were the runner-ups Overall, they achieved up to 93.6% of Hit@1 and 0.969 of MRR over all settings, except the noisy D-W dataset AliNet and GCNAlign also gave promising results, which demonstrates the power of graph neural networks for entity alignment On the other hand, the shallow embedding-based 62 Table 5.4: Ablation study D-W-V1 D-W-V2 Hit@1 Hit@10 MRR Hit@1 Hit@10 MRR Var Var1 Var2 Var3 Var4 Var5 Var6 Var7 685 691 716 722 421 468 379 863 883 903 908 741 752 639 750 762 783 791 498 515 468 784 818 828 822 512 556 628 942 962 960 970 782 799 875 841 870 873 876 641 663 712 Full model 724 911 793 832 974 883 Table 5.5: Knowledge Graph Completion performance Dataset Ver Metric DistMult TransE RotatE CompGCN IKAMI V1 EN-FR V2 V1 EN-DE V2 Hit@1 MRR Hit@1 MRR 177 311 193 337 239 365 195 340 251 381 205 357 324 421 314 413 485 621 329 469 Hit@1 MRR Hit@1 MRR 042 089 122 199 041 102 125 202 048 120 124 207 234 318 187 258 148 248 261 369 method MTransE achieved the lowest values for accuracy When it comes to the scalability, Figure 5.2 depicts the running time of the techniques GCN-Align is the fastest because of full-batch setting of GCN Our model requires the most running time, due to the accuracy-running time trade-off Note that our framework allows the users reducing the number of iteration of completion and alignment improvement to reduce the time in sacrificing the alignment accuracy D-Y-V2 D-Y-V1 D-W-V2 D-W-V1 EN-FR-V2 EN-FR-V1 EN-DE-V2 EN-DE-V1 MTransE 102 GCN-A BootEA RDGCN 103 Running time (s) Alinet IKAMI 104 Figure 5.2: Running time (in log scale) on different datasets 63 5.5.3 Robustness to KGs incompleteness We evaluate the robustness of our method against the incompleteness by first investigating the capability of our embeddings to discover missing links (a.k.a KG completion [39]) To this end, from the original KGs pair, we randomly removed 20% triples from the source graph Then, we recovered the missing triples by selecting the tail entity t that had the closet embeddings to the querying head entity and the relation pair h, r and vice-versa We compared IKAMI against four baseline KG completion techniques, namely DistMult [80], RotatE [81], TransE [3] and CompGCN [82] The result is shown in Table 5.5 It can be seen that IKAMI were either the winner or the first runner-up, despite that our technique was not specialized for this task Feature exchange between the proximity and transitivity channel can help to reconcile the KGs, which helps to reveal unseen relations from one graph based on similar patterns on the other As we not focus heavily on KG completion, interested readers can refer to other baselines [83]–[85] To fully investigate the robustness of the techniques against the KGs incompleteness, we conduct the second experiment where we choose the D-W-V2 as source KG and generate the target KG by removing the triples randomly to generate different levels of noise The result of the experiment is shown in Figure 5.4, where we only show the performance of IKAMI and the two best baselines RDGCN and AliNet In general, all methods suffer performance drop when the noise level increases Our model outperforms the baseline methods, with the Hit@1 goes from nearly 96% to around 92% when the edges removal ratio goes from 10% to 60%, thanks to the efficient feature exchange mechanism Our model keeps a margin of about 5% in Hit@1 with the runner-up (RDGCN) The performance of Alinet drops more dramatically than the others, with less than 0.3 of Hit@1 and 0.5 of Hit@10 when the noise level goes up to 60% 5.5.4 Saving of labelling effort In this experiment, we evaluated the ability of saving pre-aligned entities of the techniques by examining their performance under different level of supervision data for the D-W dataset It can be seen from Figure 5.3 that our model IKAMI outperformed other baselines for every level of supervision, especially for the lower ones We achieved a gain of around 20% for the level of 1% comparing to the second best baseline RDGCN This result demonstrates the capability of the knowledge transform between the KGs using in IKAMI in terms of saving 64 1.0 0.8 0.6 0.4 0.2 0.0 Hit@10 Hit@1 labelling effort 1% 3% 5% 10% 20% 50% 80% 1.0 0.8 0.6 0.4 0.2 0.0 Supervision percent 1% 3% 5% 10% 20% 50% 80% Supervision percent MRR (a) Hit@1 (b) Hit@10 1.0 0.8 0.6 0.4 0.2 0.0 1% 3% 5% 10% 20% 50% 80% Supervision percent (c) MRR 1.0 1.0 0.8 0.8 Hit@10 Hit@1 Figure 5.3: Saving of labelling effort for entity alignment on D-W-V1 test set 0.6 0.4 0.6 0.4 0.2 0.2 0.0 10% 20% 30% 40% 50% 60% 0.0 10% 20% 30% 40% 50% 60% Noise percent Noise percent MRR (a) Hit@1 (b) Hit@10 1.0 0.8 0.6 0.4 0.2 0.0 10% 20% 30% 40% 50% 60% Noise percent (c) MRR Figure 5.4: Robustness of graph alignment models against noise on EN-DE-V2 test set 65 Lucky Partners The Prisoner of Zenda g rin Bir Ronald Colman re t ar St r pa ace Le Prisonnier de Zenda ap ng rri a St hPl t Dea Double Chance t r rt Mo lieu thP lac Ronald Colman e re pa ap me tie r Santa Barbara (Californie) Surrey Acteur Figure 5.5: Attention visualisation (EN-FR-V1 dataset) The model pays less attention to noisy relations Table 5.6: Correct aligned relations in EN↔FR KGs country ↔ pays (country), birthPlace ↔ lieuNaissance (birth place), deathPlace ↔ lieuMort (dead place) starring ↔ apparaˆıtre (starring) , field ↔ domaine (domain) , developer ↔ d´eveloppeurs (developer) hometown ↔ nationalit´e (nationality) 5.5.5 Qualitative evidences In this section, we qualitatively interpret our technique by two case studies First, we visualized the attention coefficient of the relational triples of the entity Ronald Colman in Figure 1.4 processed by IKAMI It is clear from Figure 5.5 that the coefficient for the triples appearing in both KGs outweighed that of the triples appearing in only one KG (e.g BirthPlace triple, Profession triple) This depicts the capability of our attention mechanism in emphasizing the shared relational triples while mitigating the impact of the noisy ones Second, list some representative relation alignment generated by the relation embedding from IKAMI between EN and FR KGs Our technique efficiently captured the underlying semantic of the relation type and aligned them quite accurately, without the need of machine translation This also highlights the advantage of our relation representation learning and relation-aware propagation Second, we compare the KGC performance of IKAMI with the single-channel transitivity-based technique TransE during the training process It can be seen from Figure 5.6 that the fusion with proximity-based channel helps IKAMI not only converged faster but also achieved superior final result against TransE 66 0.65 TransE IKAMI 0.60 MRR 0.55 0.50 0.45 0.40 0.35 0.30 10 40 70 100 Epochs 130 160 190 Figure 5.6: KGC performance comparison between TransE and IKAMI during training 67 CHAPTER CONCLUSION In the bulk of our Master work, we address the representation learning on knowledge graphs, an essential but challenging problem in network science appearing in various applications, ranging from knowledge graph completion, knowledge graph alignment to commonsense question answering using external knowledge Developing effective deep neural network architectures for knowledge graph representation learning can thus boost the performance of many critical applications in various domains Given the importance of knowledge graph representation, we define two major tasks, namely knowledge graph completion and knowledge graph alignment, to evaluate the expressive power of deep architectures and tackle the three crucial challenges in the literature: • Enhancing knowledge graph completion performance We propose a novel model named NoGE, which first transforms a heterogeneous (KG) graph to a homogeneous (normal) graph We then define a weighted adjacency matrix to capture co-occurrence statistics of neighbor relationships among entities and relations and use this as the input of our proposed dual quaternion graph neural network to learn node and edge representations This expressive representation allows NoGE to achieve SOTA performance on various datasets • Enhancing knowledge graph alignment on large-scale attribute knowledge graph We propose EMGCN, which aims to capture not only KGs structural information but also attribute information To better capture local as well as global neighborhood structure around entities, we take advantage of all GCN representations of entities in the graphs Our unsupervised setting allow our model beat all the previous SOTA baselines • Enhancing knowledge graph completion and knowledge graph alignment at the same time We propose IKAMI, a multitask learning model, with two main feature channels aiming to capture not only the positional information but also node attribute and neighbourhood structure around entities This architecture achieves SOTA performance on the both tasks 68 REFERENCES [1] Q Wang, Z Mao, B Wang, and L Guo, “Knowledge graph embedding: A survey of approaches and applications,” IEEE Transactions on Knowledge and Data Engineering, vol 29, no 12, pp 2724–2743, 2017 [2] Z Sun, Q Zhang, W Hu, C Wang, M Chen, F Akrami, and C Li, “A benchmarking study of embedding-based entity alignment for knowledge graphs,” Proc VLDB Endow., vol 13, no 12, 2326–2340, 2020 [3] A Bordes, N Usunier, A Garcia-Dur´an, J Weston, and O Yakhnenko, “Translating embeddings for modeling multi-relational data,” in Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013, pp 2787–2795 [4] T Dettmers, P Minervini, P Stenetorp, and S Riedel, “Convolutional 2D Knowledge Graph Embeddings,” in AAAI, 2018, pp 1811–1818 [5] Z Sun, Z.-H Deng, J.-Y Nie, and J Tang, “Rotate: Knowledge graph embedding by relational rotation in complex space,” in International Conference on Learning Representations, 2019 [6] D Q Nguyen, D Q Nguyen, T D Nguyen, and D Phung, “ Convolutional Neural Network-based Model for Knowledge Base Completion and Its Application to Search Personalization,” Semantic Web, vol 10, no 5, pp 947–960, 2019 [7] I Balaˇzevi´c, C Allen, and T M Hospedales, “Tucker: Tensor factorization for knowledge graph completion,” in Empirical Methods in Natural Language Processing, 2019, pp 5185–5194 [8] B Yang, W.-t Yih, X He, J Gao, and L Deng, “Embedding Entities and Relations for Learning and Inference in Knowledge Bases,” in Proceedings of the International Conference on Learning Representations, 2015 ´ Gaussier, and G Bouchard, “Complex [9] T Trouillon, J Welbl, S Riedel, E Embeddings for Simple Link Prediction,” in ICML, 2016, pp 2071–2080 [10] M Schlichtkrull, T Kipf, P Bloem, R v d Berg, I Titov, and M Welling, “Modeling relational data with graph convolutional networks,” in ESWC, 2018, pp 593–607 [11] C Shang, Y Tang, J Huang, J Bi, X He, and B Zhou, “End-to-end structure-aware convolutional networks for knowledge base completion,” in AAAI, vol 33, 2019, pp 3060–3067 69 [12] S Vashishth, S Sanyal, V Nitin, and P Talukdar, “Composition-based multi-relational graph convolutional networks,” in ICLR, 2020 [13] T Gracious, S Gupta, A Kanthali, R M Castro, and A Dukkipati, “Neural latent space model for dynamic networks and temporal knowledge graphs,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, 2021, pp 4054–4062 [14] H T Trung, T Van Vinh, N T Tam, H Yin, M Weidlich, and N Q V Hung, “Adaptive network alignment with unsupervised and multi-order convolutional networks,” in IEEE 36th International Conference on Data Engineering (ICDE), 2020, pp 85–96 [15] M C Phan, A Sun, Y Tay, J Han, and C Li, “Pair-linking for collective entity disambiguation: Two could be better than all,” IEEE Transactions on Knowledge and Data Engineering, vol 31, no 7, pp 1383–1396, 2018 [16] G Wan and B Du, “Gaussianpath: A bayesian multi-hop reasoning framework for knowledge graph reasoning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, 2021, pp 4393–4401 [17] Y Yan, L Liu, Y Ban, B Jing, and H Tong, “Dynamic knowledge graph alignment,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, 2021, pp 4564–4572 [18] Y Zhang, H Dai, Z Kozareva, A J Smola, and L Song, “Variational reasoning for question answering with knowledge graph,” in Thirty-Second AAAI Conference on Artificial Intelligence, 2018, pp 6069–6076 [19] H Chen, H Yin, T Chen, Q V H Nguyen, W.-C Peng, and X Li, “Exploiting centrality information with graph convolutions for network representation learning,” in IEEE 35th International Conference on Data Engineering (ICDE), 2019, pp 590–601 [20] M Chen, I W Tsang, M Tan, and T J Cham, “A unified feature selection framework for graph embedding on high dimensional data,” IEEE Transactions on Knowledge and Data Engineering, vol 27, no 6, pp 1465–1477, 2014 [21] M Chen, Y Tian, M Yang, and C Zaniolo, “Multilingual knowledge graph embeddings for cross-lingual knowledge alignment,” in Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017, pp 1511– 1517 70 [22] Z Sun, W Hu, and C Li, “Cross-lingual entity alignment via joint attributepreserving embedding,” in International Semantic Web Conference, 2017, pp 628–644 [23] H Zhu, R Xie, Z Liu, and M Sun, “Iterative entity alignment via joint knowledge embeddings.,” in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017, pp 4258–4264 [24] Z Sun, W Hu, Q Zhang, and Y Qu, “Bootstrapping entity alignment with knowledge graph embedding,” in Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018, pp 4396–4402 [25] Y Wu, X Liu, Y Feng, Z Wang, R Yan, and D Zhao, “Relation-aware entity alignment for heterogeneous knowledge graphs,” in Proceedings of the 28th International Joint Conference on Artificial Intelligence, 2019, pp 5278–5284 [26] Z Wang, Q Lv, X Lan, and Y Zhang, “Cross-lingual knowledge graph alignment via graph convolutional networks,” in EMNLP, 2018, pp 349– 357 [27] Y Cao, Z Liu, C Li, Z Liu, J Li, and T.-S Chua, “Multi-channel graph neural network for entity alignment,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp 1452– 1461 [28] K Xu, L Wang, M Yu, Y Feng, Y Song, Z Wang, and D Yu, “Crosslingual knowledge graph alignment via graph matching neural network,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp 3156–3161 [29] Q Zhu, X Zhou, J Wu, J Tan, and L Guo, “Neighborhood-aware attentional representation for multilingual knowledge graphs,” in IJCAI, 2019, pp 1943–1949 [30] T N Kipf and M Welling, “Semi-supervised classification with graph convolutional networks,” in 5th International Conference on Learning Representations, 2017, pp 1–14 [31] H T Trung, N T Toan, T Van Vinh, H T Dat, D C Thang, N Q V Hung, and A Sattar, “A comparative study on network alignment techniques,” Expert Systems with Applications, vol 140, p 112 883, 2020 71 [32] T T Huynh, C T Duong, T H Quyet, Q V H Nguyen, A Sattar, et al., “Network alignment by representation learning on structure and attribute,” in Pacific Rim International Conference on Artificial Intelligence, 2019, pp 698–711 [33] Z Sun, C Wang, W Hu, M Chen, J Dai, W Zhang, and Y Qu, “Knowledge graph alignment network with gated multi-hop neighborhood aggregation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, 2020, pp 222–229 [34] X Zhao, W Zeng, J Tang, W Wang, and F Suchanek, “An experimental study of state-of-the-art entity alignment approaches,” IEEE Transactions on Knowledge & Data Engineering, no 01, pp 1–1, 2020 [35] T T Nguyen, T T Huynh, H Yin, V Van Tong, D Sakong, B Zheng, and Q V H Nguyen, “Entity alignment for knowledge graphs with multiorder convolutional networks,” IEEE Transactions on Knowledge and Data Engineering, vol 32, no 13, pp 1–14, 2021 [36] K Xu, C Li, Y Tian, T Sonobe, K Kawarabayashi, and S Jegelka, “Representation learning on graphs with jumping knowledge networks,” in International Conference on Machine Learning, 2018, pp 5449–5458 [37] Z Wu, S Pan, F Chen, G Long, C Zhang, and S Y Philip, “A comprehensive survey on graph neural networks,” IEEE transactions on neural networks and learning systems, vol 32, no 1, pp 4–24, 2020 [38] T Mikolov, I Sutskever, K Chen, G S Corrado, and J Dean, “Distributed Representations of Words and Phrases and their Compositionality,” in Advances in Neural Information Processing Systems 26, 2013, pp 3111–3119 [39] Z Wang, J Zhang, J Feng, and Z Chen, “Knowledge graph embedding by translating on hyperplanes,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 28, 2014 [40] G Ji, S He, L Xu, K Liu, and J Zhao, “Knowledge graph embedding via dynamic mapping matrix,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015, pp 687–696 [41] W Huang, G Li, and Z Jin, “Improved knowledge base completion by the path-augmented transr model,” in International Conference on Knowledge Science, Engineering and Management, Springer, 2017, pp 149–159 72 [42] D Q Nguyen, K Sirts, L Qu, and M Johnson, “Neighborhood Mixture Model for Knowledge Base Completion,” in Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, 2016, pp 40–50 [43] G Ji, K Liu, S He, and J Zhao, “Knowledge graph completion with adaptive sparse transfer matrix,” in Thirtieth AAAI conference on artificial intelligence, 2016 [44] Q Xie, X Ma, Z Dai, and E Hovy, “An interpretable knowledge transfer model for knowledge base completion,” arXiv preprint arXiv:1704.05908, 2017 [45] R Socher, D Chen, C D Manning, and A Ng, “Reasoning with neural tensor networks for knowledge base completion,” in Advances in neural information processing systems, 2013, pp 926–934 [46] I Balaˇzevi´c, C Allen, and T Hospedales, “Multi-relational poincar´e graph embeddings,” in Advances in Neural Information Processing Systems, 2019, pp 4465–4475 [47] T Lacroix, N Usunier, and G Obozinski, “Canonical tensor decomposition for knowledge base completion,” in International Conference on Machine Learning, PMLR, 2018, pp 2863–2872 [48] X Zhu, Y Xu, H Xu, and C Chen, “Quaternion convolutional neural networks,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp 631–647 [49] Y Lin, Z Liu, M Sun, Y Liu, and X Zhu, “Learning entity and relation embeddings for knowledge graph completion,” in AAAI, 2015 [50] M Chen, Y Tian, K.-W Chang, S Skiena, and C Zaniolo, “Co-training embeddings of knowledge graphs and entity descriptions for cross-lingual entity alignment,” arXiv preprint arXiv:1806.06478, 2018 [51] K Wang, Y Liu, X Xu, and D Lin, “Knowledge graph embedding with entity neighbors and deep memory network,” arXiv preprint arXiv:1808.03752, 2018 [52] S Zhang and H Tong, “Final: Fast attributed network alignment,” in KDD, 2016, pp 1345–1354 [53] Z Sun, S Vashishth, S Sanyal, P Talukdar, and Y Yang, “A re-evaluation of knowledge graph completion methods,” ACL, 2020 73 [54] A Bordes, N Usunier, A Garc´ıa-Dur´an, J Weston, and O Yakhnenko, “Translating Embeddings for Modeling Multi-relational Data,” in Advances in Neural Information Processing Systems 26, 2013, pp 2787–2795 [55] D Q Nguyen, T D Nguyen, and D Phung, “Quaternion graph neural networks,” in Asian Conference on Machine Learning, 2021 [56] D Q Nguyen, “A survey of embedding models of entities and relationships for knowledge graph completion,” in TextGraphs, 2020, pp 1–14 [57] S Zhang, Y Tay, L Yao, and Q Liu, “Quaternion knowledge graph embeddings,” in Advances in Neural Information Processing Systems, 2019, pp 2731–2741 [58] T Safavi and D Koutra, “CoDEx: A Comprehensive Knowledge Graph Completion Benchmark,” in EMNLP, 2020, pp 8328–8350 [59] M Clifford, “Preliminary sketch of biquaternions,” Proceedings of the London Mathematical Society, vol 1, no 1, pp 381–395, 1871 [60] F W Levi, Finite Geometrical Systems: Six Public Lectues Delivered in February, 1940, at the University of Calcutta University of Calcutta, 1942 [61] T N Kipf and M Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, 2017 [62] D Q Nguyen, T D Nguyen, and D Phung, “Universal graph transformer self-attention networks,” arXiv preprint arXiv:1909.11855, 2019 [63] W R Hamilton, “Ii on quaternions; or on a new system of imaginaries in algebra,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol 25, no 163, pp 10–13, 1844 [64] A Torsello, E Rodola, and A Albarelli, “Multiview registration via graph diffusion of dual quaternions,” in CVPR 2011, 2011, pp 2441–2448 [65] D Kingma and J Ba, “Adam: A Method for Stochastic Optimization,” arXiv preprint arXiv:1412.6980, 2014 [66] D Koutra, H Tong, and D Lubensky, “Big-align: Fast bipartite graph alignment,” in ICDM, 2013, pp 389–398 [67] T Man, H Shen, S Liu, X Jin, and X Cheng, “Predict anchor links across social networks via an embedding approach,” in IJCAI, vol 16, 2016, pp 1823–1829 [68] Z Luo, L Liu, J Yin, Y Li, and Z Wu, “Deep learning of graphs with ngram convolutional neural networks,” TKDE, vol 29, no 10, pp 2125– 2139, 2017 74 [69] C Chen, W Xie, T Xu, Y Rong, W Huang, X Ding, Y Huang, and J Huang, “Unsupervised adversarial graph alignment with graph embedding,” arXiv:1907.00544, 2019 [70] W Hamilton, Z Ying, and J Leskovec, “Inductive representation learning on large graphs,” in NIPS, 2017, pp 1024–1034 [71] G Kollias, S Mohammadi, and A Grama, “Network similarity decomposition (nsd): A fast and scalable approach to network alignment,” IEEE Transactions on Knowledge and Data Engineering, vol 24, no 12, pp 2232– 2243, 2011 [72] K Shu, S Wang, J Tang, R Zafarani, and H Liu, “User identity linkage across online social networks: A review,” Acm Sigkdd Explorations Newsletter, vol 18, no 2, pp 5–17, 2017 [73] S Pei, L Yu, G Yu, and X Zhang, “Rea: Robust cross-lingual entity alignment between knowledge graphs,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp 2175–2184 [74] P Velickovic, G Cucurull, A Casanova, A Romero, P Li`o, and Y Bengio, “Graph attention networks,” in 6th International Conference on Learning Representations, 2018, pp 1–12 [75] S Brody, U Alon, and E Yahav, “How attentive are graph attention networks?” arXiv preprint arXiv:2105.14491, 2021 [76] J Lehmann, R Isele, M Jakob, A Jentzsch, D Kontokostas, P N Mendes, S Hellmann, M Morsey, P Van Kleef, S Auer, et al., “Dbpedia–a largescale, multilingual knowledge base extracted from wikipedia,” Semantic web, vol 6, no 2, pp 167–195, 2015 [77] F M Suchanek, G Kasneci, and G Weikum, “Yago: A large ontology from wikipedia and wordnet,” Journal of Web Semantics, vol 6, no 3, pp 203–217, 2008 [78] Z Sun, W Hu, and C Li, “Cross-lingual entity alignment via joint attributepreserving embedding,” in International Semantic Web Conference, Springer, 2017, pp 628–644 [79] Q Zhang, Z Sun, W Hu, M Chen, L Guo, and Y Qu, “Multi-view knowledge graph embedding for entity alignment,” IJCAI, 2019 75 [80] T Dettmers, P Minervini, P Stenetorp, and S Riedel, “Convolutional 2d knowledge graph embeddings,” in Thirty-second AAAI conference on artificial intelligence, 2018, pp 1811–1818 [81] D Q Nguyen, T D Nguyen, D Q Nguyen, and D Q Phung, “A novel embedding model for knowledge base completion based on convolutional neural network,” pp 327–333, 2018 [82] S Vashishth, S Sanyal, V Nitin, and P Talukdar, “Composition-based multi-relational graph convolutional networks,” in International Conference on Learning Representations, 2019, pp 1–14 [83] Z Qiao, Z Ning, Y Du, and Y Zhou, “Context-enhanced entity and relation embedding for knowledge graph completion (student abstract),” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, 2021, pp 15 871–15 872 [84] F Che, D Zhang, J Tao, M Niu, and B Zhao, “Parame: Regarding neural network parameters as relation embeddings for knowledge graph completion,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, 2020, pp 2774–2781 [85] C Zhang, H Yao, C Huang, M Jiang, Z Li, and N V Chawla, “Few-shot knowledge graph completion,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, 2020, pp 3041–3048 76 ... challenging tasks, such as knowledge graph alignment, knowledge graph completion, and question answering Recently, deep learning methods using the representation of knowledge graph entities (nodes)... INRODUCTION 1.1 Knowledge Graphs (KGs) 1.2 Knowledge graph completion and knowledge graph alignment 1.2.1 Knowledge graph completion 1.2.2 Knowledge graph alignment ... representation learning methods for two major tasks The first task is knowledge graph completion, the task of filling missing triples into incomplete knowledge graphs The second task is knowledge graph