Personalize job recommendation system

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY MASTER THESIS Personalize Job Recommendation System DUC-THAI DO Thai.DD211260M@sis.hust.edu.vn Major: Data Science and Artificial Intelligence Thesis advisor Department Institute : Dr Tran Viet Trung : Computer Science : School of Information and Communication Technology Hanoi, 04-2023 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY MASTER THESIS Personalize Job Recommendation System DUC-THAI DO Thai.DD211260M@sis.hust.edu.vn Major: Data Science and Artificial Intelligence Thesis advisor : Dr Tran Viet Trung Signature of advisor Department : Computer Science Institute : School of Information and Communication Technology Hanoi, 04-2023 &Ӝ1*+Ñ$;­+Ӝ,&+Ӫ1*+Ƭ$9,ӊ71$0 ĈӝFOұS– 7ӵGR– +ҥQKSK~F %Ҧ1;È&1+Ұ1&+ӌ1+6Ӱ$ /8Ұ19Ă17+Ҥ&6Ƭ +ӑYjWrQWiFJLҧOXұQYăQĈӛĈӭF7KiL ĈӅWjLOXұQYăQ&iQKkQKRiKӋWKӕQJJӧLêYLӋFOjP Chuyên ngành: KRDKӑFGӳOLӋXvà TUtWXӋQKkQWҥR (Elitech) 0mVӕ69: 20211260M 7iFJLҧ1JѭӡLKѭӟQJGүQNKRDKӑFYj+ӝLÿӗQJFKҩPOXұQYăQ[iFQKұQWiFJLҧÿm VӱDFKӳDEәVXQJOXұQYăQWKHRELrQEҧQKӑS+ӝLÿӗQJQJj\ 22/04/2023 YӟLFiFQӝL dung sau: STT 1ӝLGXQJFKӍQKVӱD Trang %ӓÿiQKVӕFKѭѫQJWURQJFKѭѫQJPӣÿҫXYjNӃWOXұQ 9, 69 %әVXQJOjPU}PөFWLrXFӫDOXұQYăQWURQJSKҫQ0ӣÿҫXYjOjPU} KjPêFiQKkQKRi³SHUVRQDOL]LQJ´KӋWKӕQJWѭYҩQ 7URQJFKѭѫQJ EDQÿҫXEәVXQJOjPU}KѫQNKiLQLӋPFiFNӻWKXұW 20-21 “item popularity”, “user-item matching” ĈѭDQӝLGXQJFKѭѫQJ EDQÿҫX WUuQKEj\GDWDVHWVYjFiFKRҥWÿӝQJ 27-39 FөWKӇFKXҭQKRiKDLGDWDVHWVQKѭPөFFRQFӫDFKѭѫQJ EDQÿҫXYj FKӍQKWrQFKѭѫQJnày ÿӇOjPU}KѫQQӝLGXQJWUuQKEj\ Trình bày mơ KuQKWәQJWKӇFӫDEjLWRiQWѭYҩQYLӋFOjPYjOXұQJLҧL 27-28 FiFNƭWKXұWÿmOӵDFKӑQ/jPU}KѫQSLSHOLQHWKӇKLӋQSKѭѫQJSKiS ÿmWLӃQKjQKWKӵFKLӋQ 0{WҧNƭKѫQFiFKWLӃSFұQVӱGөQJSKѭѫQJSKiSNӃWKӧSK\EULG 50 0{WҧNƭKѫQYLӋFÿiQKJLiKӋWѭYҩQYӟLWұS5HF6\V 61 6ӱGөQJWKrPÿӝÿR0$3# 60-67 ĈiQKJLiVRViQKNӃWTXҧWKӵFQJKLӋPYӟLF{QJEӕNKiFVӱGөQJFQJ 68 datasets 10 3KkQWtFKNƭKѫQNӃWTXҧFӫDP{KuQKGӵDWUrQEҧQFKҩWGӳOLӋX 61-68 11 %әVXQJFiFWUtFKGүQFzQWKLӃX 21-22 12 %әVXQJWLrXÿӅYjWrQFiFWUөFFӫDFiFELӇXÿӗ 32-65 13 5jVRiWKLӋXFKӍQKFiFOӛLVRҥQWKҧR 1Jj\WKiQJQăP 2023 *LiRYLrQKѭӟQJGүQ 7iFJLҧOXұQYăQ &+Ӫ7ӎ&++Ӝ,ĈӖ1* 6Ĉ+47%0 %DQKjQKOҫQ1 ngày 11/11/2014 Graduation Thesis Assignment Name: Duc-Thai Do Phone: +84 902210496 Email: Thai.DD211260M@sis.hust.edu.vn; thai.dec1mo@gmail.com Class: 21A-IT-KHDL-E/CH2021A Affiliation: Hanoi University of Science and Technology I - Duc-Thai Do - hereby warrants that the work and presentation in this thesis were performed by myself under the supervision of Dr Tran Viet Trung All the results presented in this thesis are truthful and are not copied from any other works All references in this thesis including images, tables, figures, and quotes are clearly and fully documented in the bibliography I will take full responsibility for even one copy that violates school regulations Hanoi, April 2023 Author Duc-Thai Do ACKNOWLEDGMENTS Before presenting the main content of the thesis, I would like to dedicate these lines to send my most sincere thanks to the people who have helped and shaped the person I am today I thank my parents and grandparents for raising, being with, and supporting me unconditionally They have given me a family that could not be more wonderful, a place that always gives me motivation whenever I feel tired or stumble on the road of life In order to complete this graduation project, I would like to express my sincerest thanks to Mr Tran Viet Trung, who not only suggested and gave new ideas, but also closely guided and encouraged me to overcome this difficult period Working with you is one of the luckiest things I have had Thanks to your encouragement and enthusiastic guidance, I was able to pass and complete this graduation thesis At the same time, I would also like to thank all the teachers of Hanoi University of Science and Technology for always doing their best for us The teachers have brought us valuable knowledge and experience that we can’t get anywhere else I hope the teachers are always healthy to continue educating the next generation of Bach Khoa students I would like to thank my loved one for always being there, cheering, and helping me during my study and graduation thesis ABSTRACT Nowadays, online recruitment websites have become one of the main channels for people to search for jobs These platforms have saved a lot of time and money for both the candidate as well as the recruiting organization These platforms have saved a lot of time and money for both job seekers as well as recruiting organizations However, traditional information retrieval techniques such as searching for the desired job through a keyword using a search engine are not suitable The reason is that because the number of results returned to job seekers can be very large, they need to spend considerable time reading and reviewing their options, resulting in a very boring and difficult job search experience For that reason, this thesis aims at building an effective job recommendation system, increasing the personalization and relevance of job search results, and helping user experience and job searching journey become more easy and more exciting To achieve this, the thesis has analyzed data on job listings and job seeker characteristics and behavior from two labor market datasets: RecSys2016 and CareerBuilder2012 researched and utilized a combination of different techniques, tools, and models in the fields of natural language processing, machine learning, and recommendation systems to implement, and experiment various job recommendation algorithms: item popularity, user-item matching, content-based, user-based collaborative filtering, and graph neural network on the two labor dataset The effectiveness of the system has been evaluated based on two metrics: M ap@K and RSScore, showing the practical and positive results in recommendations in the field of the labor market Contents Introduction 10 Chapter Theoretical basis 16 1.1.Foundation algorithms 16 1.1.1 TF-IDF 16 1.1.2 Cosine Similarity 17 1.1.3 K-nearest neighbors 17 1.1.4 Overview of neural network 18 1.1.5 Overview of graph neural network 19 1.2.Overview of recommendation system 21 1.2.1 Item popularity recommendation 21 1.2.2 User-item matching recommendation 22 1.2.3 Content-based recommendation system 22 1.2.4 Collaborative filtering recommendation system 23 1.2.5 Hybrid recommendation system 25 1.3.Related work 26 Chapter Proposed approaches for job recommendation 28 2.1.Overall model construction of the job recommendation system 28 2.2.Dataset description and analysis 29 2.2.1 RecSys2016 dataset 29 2.2.2 CareerBuilder2012 dataset 40 2.3.Data labeling 46 2.4.Data preprocessing 48 2.5.Recommendation model implementation 49 2.5.1 Item popularity approach 49 2.5.2 User-item matching approach 50 2.5.3 Content-based with item popularity approach 51 2.5.4 Collaborative filtering with item popularity approach 52 2.5.5 Graph Neural Network with item popularity approach 53 Chapter Experiments and results 61 3.1.Evaluation metrics 61 3.1.1 Map@k 61 3.1.2 RecSys2016 Score 62 3.2.Results and discussion 62 Conclusion and Future work 69 List of Figures Search results from careerbuilder.vn at 08/04/2023 11 1.1 1.2 1.3 1.4 Neural network architecture Basic graph neural network illustration Content-based recommendation system Used-based vs Item-based in memory-based collaborative filtering 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25 User’s university degree distribution in the RecSys2016 dataset 33 User’s total experience year distribution in the RecSys2016 dataset 34 User’s current experience year distribution in the RecSys2016 dataset 34 User’s number of experience entry distribution in the RecSys2016 dataset 35 Job’s employment type distribution in the RecSys2016 dataset 35 Job’s active during test status distribution in the RecSys2016 dataset 36 User and job’s career level distribution in the RecSys2016 dataset 36 Top user and job’s discipline id distribution in the RecSys2016 dataset 37 Top user and job’s industry id distribution in the RecSys2016 dataset 38 User and job’s country distribution in the RecSys2016 dataset 38 Interaction type distribution in the RecSys2016 dataset 39 CareerBuilder2012 data layout 40 User’s degree type distribution in the CareerBuilder dataset 42 Top 20 user’s major distribution in the CareerBuilder dataset 42 Top 20 user’s job distribution in the CareerBuilder dataset 42 User’s years from graduation distribution in the CareerBuilder dataset 43 User’s total experience years (≤ 40) distribution in the CareerBuilder dataset 44 User’s CurrentlyEmployed status distribution in the CareerBuilder dataset 44 User’s ManagedOthers status distribution in the CareerBuilder dataset 45 Top 20 item’s job titles in the CareerBuilder dataset 45 User and item’s US state distribution in the CareerBuilder dataset 46 Item’s active status distribution in the CareerBuilder dataset 47 Popularity score calculation illustration 50 Simple collaborative filtering illustration 52 Bipartite graph demonstration in recommendation systems 53 3.1 3.2 RGCN’s loss curve during training in RecSys2016 65 RGCN’s loss curve during training in CareerBuilder2012 65 19 19 23 24 Table 2.2: Table Items in the RecSys2016 dataset Field name Type Description id title Int Text career level Int discipline id Int industry id Int country region latitude longitude employment Text Int Float Float Int Anonymized ID of the item Concepts (numeric IDs) that have been extracted from the job title of the job posting career level ID Meaning of numbers are the same as career level in the table User Anonymized IDs represent disciplines such as ”Consulting”, ”HR”, etc Anonymized IDs represent industries such as ”Internet”, ”Automotive”, ”Finance”, etc Code of the country in which the job is offered Specified for some users who have as country de latitude information (rounded to ca 10km) longitude information (rounded to ca 10km) The type of employment: • 0: unknown • 1: full-time • 2: part-time • 3: freelancer • 4: intern • 5: voluntary tags Text created at Int active during test Int Concepts that have been extracted from the tags, skills, or company name A Unix time stamp timestamp representing the time when the interaction got created if the item is still active (= recommendable) during the test period and if the item is not active anymore in the test period (= not recommendable) 57 Table 2.3: Table Interactions in the RecSys2016 dataset Field name Type Description user id item id Int Int interaction type Int created at Int ID of the user who performed the interaction ID of the item on which the interaction was performed The type of interaction that was performed on the item A Unix time stamp timestamp representing the time when the interaction got created Table 2.4: Table Impressions in the RecSys2016 dataset Field name Type Description user id items Int Int year week Text Int ID of the user A comma-separated list of items that were displayed to the user Year the job posting was displayed to the user Week of the year that job posting was displayed to the user Table 2.5: RecSys2016 dataset statistics Event type #events #users #items Click 7,183,038 769,396 998,424 Bookmark 206,191 59,063 142,908 Reply 422,026 107,463 190,099 Delete 1,015,423 44,595 215,844 Impression 201,872,093 2,755,167 846,814 - - 1,500,000 1,358,098 58 Table 2.6: Table users in CareerBuilder2012 dataset Field name Type Description UserID WindowID Split City State Country ZipCode DegreeType Major GraduationDate WorkHistoryCount TotalYearsExperience CurrentlyEmployed ManagedOthers ManagedHowMany Int Int String String String String Int String String Datetime Int Int Boolean Boolean Int ID of the user ID of the window time User belongs to training or test set User’s City User’s State User’s Country User’s postal code User’s Degree Type User’s Major User’s Graduation Date Amount of work the user has done Total years of user experience Is the user currently working? Is the user managing others Number of people being managed by the user Table 2.7: Table user history in CareerBuilder2012 dataset Field name Type Description UserID WindowID Split Sequence Int Int String Int JobTitle String ID of the user ID of the window time User belongs to training or test set Order of work done by a user, smaller order indicates more recent work Title of the job Table 2.8: Table jobs in CareerBuilder2012 dataset Field name Type Description JobID WindowID Title Description Requirements City State Country Zip5 StartDate Int Int String String String String String String Int Datetime EndDate Datetime ID of the job ID of the window time Title of the job Job description Job requirements City of job postings State of job postings Country of job posting Postal code of job posting Job postings start showing up on Careerbuilder.com Job posting time no longer shows on Careerbuilder.com 59 Table 2.9: Table apps in CareerBuilder2012 dataset Field name Type Description UserID WindowID Split ApplicationDate Int Int String Datetime JobID Int ID of the user ID of the window time User belongs to training or test set Time the user applied for the corresponding job advertisement ID of the job Table 2.10: Table window dates in CareerBuilder2012 dataset Field name Type Description Window Train Start Train End / Test Start Int Datetime Datetime Test End Datetime ID of the window time The start time of the training phase Time to end the training period and start the test period Test period end time Table 2.11: Anonymized textual fields processing example 60 title tags =⇒ text feature 4298526, 4316979 1471052, 2072458, 2512557 =⇒ 4298526 4316979 1471052 2072458 2512557 Chapter Experiments and results All different algorithms in this thesis are evaluated with M ap@k and RecSys2016 Score (RSScore), which are described in the following subsections 3.1 Evaluation metrics 3.1.1 Map@k M AP @k (Mean Average Precision at k) is a metric used to evaluate the performance of ranking algorithms, including information retrieval and recommendation systems It measures the average precision of the top-k-ranked items recommended by a system, where k is a positive integer U U k    M AP @k = APu @k = Pu @k × relu (k) U u=1 U u=1 min(n, k) k=1 (3.1.1) The M AP @k metric is calculated in 3.1.1, in which: • U is the number of users • n is the number of relevant items • Pu @k is the precision within the first top k items of user u  if k th item is relevant with user u • relu (k) = otherwise M AP @k is a useful metric because it takes into account both the order and the relevance of the recommended items, and it is sensitive to the number of relevant items in the query A higher M AP @k score indicates better performance of the ranking algorithm In this thesis, M AP @1, M AP @5, M AP @10, M AP @30, and M AP @150 are applied M AP @k with k ≤ 10 is a reasonable metric since an interface screen can intuitively contain up to 10 items for the user to interact M AP @150 was also used in the 61 Kaggle Job Recommendation Challenge competition [3] This metric is used to measure the performance of the model in recommending a big pool of potential items 3.1.2 RecSys2016 Score The RecSys2016 challenge designed an evaluation measure that reflects typical use cases at XING: users are presented with their top-k personalized recommendations, and user interaction with one of the top-k is counted as a success The task of the original challenge is to compute 30 recommendations (or less) for each of the 150,000 target users In particular, the algorithms have to predict those items that a user will interact with The original evaluation measure equation of the ACM RecSys Challenge 2016 sum all the individual user’s score in 3.1.2 RS2016EvalM easure = U   20 × (Pu @2 + Pu @4 + Ru @30 + Successu @30) u=1 (3.1.2) + 10 × (Pu @6 + Pu @20) in which: • Pu @k is the precision within the first top k items of user u • Ru @30 is the recall with all recommendation items of user u  if user u interact with item(s) from top 30 recommendation • Successu @30 = otherwise • U is the number of target users This is a comprehensive evaluation metric that aims at recommending the top 30 most relevant items in the ACM RecSys Challenge 2016 However, the original task of the challenge has a fixed 150,000 target users as the test set which is unable to be obtained For that reason, the original RecSys2016 evaluation measure is not suitable for this thesis since the numbers of target users in the test sets are different Therefore, this thesis further divides that scoring function by the total target users RSScore’s equation is described in 3.1.3 U   20 × (Pu @2 + Pu @4 + Ru @30 + Successu @30) RSScore = U u=1 + 10 × (Pu @6 + Pu @20) 3.2 (3.1.3) Results and discussion Before going into any results and discussion, it is important to notice the fact that the results of the recommendation system are affected by the items shown to users 62 Table 3.1: Performance of Matching model with different weights in the Recsys2016 dataset (w1, w2) Map@1 Map@5 Map@10 Map@30 Map@150 RSScore (0.0, 1.0) 0.0018 0.0018 0.0020 0.0024 0.0028 0.7715 (0.1, 0.9) 0.0018 0.0018 0.0020 0.0024 0.0028 0.7671 (0.2, 0.8) 0.0018 0.0018 0.0020 0.0024 0.0028 0.7677 (0.3, 0.7) 0.0017 0.0017 0.0019 0.0023 0.0026 0.7457 (0.4, 0.6) 0.0015 0.0014 0.0016 0.0019 0.0023 0.6551 (0.5, 0.5) 0.0011 0.0010 0.0012 0.0014 0.0016 0.4970 (0.6, 0.4) 0.0005 0.0005 0.0006 0.0007 0.0008 0.2715 (0.7, 0.3) 0.0002 0.0002 0.0002 0.0003 0.0003 0.1032 (0.8, 0.2) 0.0001 0.0001 0.0001 0.0001 0.0001 0.0376 (0.9, 0.1) 0.0001 0.0000 0.0000 0.0001 0.0001 0.0328 (1.0, 0.0) 0.0000 0.0000 0.0000 0.0000 0.0000 0.0067 which are called impressions since the users can only interact with a finite number of impressions If a job post never shows in the impressions of the system, it is not possible to collect any interaction between the user and the post In RecSys2016, impressions are generated by the existing recommendation system In CareerBuilder2012, there is no information about how impressions shown to users are made For that reason, the existing recommendation algorithms or baselines can drive the performance of the recommendation models in the thesis The user-item matching approach’s performance with different weights using the RecSys2016 and CareerBuilder2012 dataset are respectively shown in Table 3.1 and 3.2 Surprisingly, in RecSys2016, even utilizing some seem-to-be informative non-textual data such as: ”discipline id”, ”industry id”, and ”career level” beside geographical data like ”region” and ”country”, the model that was constructed based solely on textual data has the best performance in all evaluation metrics This might be due to the big gaps between users and items in ”discipline id”, ”industry id”, and ”career level” mentioned in the RecSys2016 dataset analysis In CareerBuilder2012, (w1, w2) = (0.5, 0.5) gives the highest performance, which means both non-textual and textual data are important for the recommendation although the non-textual data in CareerBuilder2012 only contains geographical data, including ”City”, ”State”, ”Country”, and not other seem-to-be informative data as RecSys2016’s The reason for the impact of geographical data on the performance in this dataset might be that the existing algorithms used in CareerBuilder that generate the impressions also utilized this data type, especially when the competition suggested using geographical data for its baseline approach Table 3.3 and 3.4 indicate the content-based recommendation system’s evaluation metrics with different weights The model constructed using the RecSys2016 dataset improves significantly compared to the user-item matching approach The optimal (w1, w2) pair is (0.1, 0.9), which is not so different from the one in the matching approach and still utilizes mostly textual data This reveals that there might be a semantic 63 Table 3.2: Performance of Matching model with different weights in the CareerBuilder2012 dataset (w1, w2) Map@1 Map@5 Map@10 Map@30 Map@150 RSScore (0.0, 1.0) 0.0008 0.0008 0.0010 0.0012 0.0015 0.5014 (0.1, 0.9) 0.0023 0.0020 0.0023 0.0027 0.0032 0.9731 (0.2, 0.8) 0.0049 0.0047 0.0053 0.0062 0.0073 2.1135 (0.3, 0.7) 0.0091 0.0092 0.0103 0.0121 0.0137 3.7859 (0.4, 0.6) 0.0120 0.0126 0.0141 0.0162 0.0181 4.7051 (0.5, 0.5) 0.0131 0.0138 0.0153 0.0173 0.0193 4.7470 (0.6, 0.4) 0.0129 0.0130 0.0144 0.0163 0.0183 4.4659 (0.7, 0.3) 0.0127 0.0127 0.0141 0.0159 0.0178 4.2817 (0.8, 0.2) 0.0126 0.0126 0.0138 0.0156 0.0175 4.1917 (0.9, 0.1) 0.0126 0.0126 0.0139 0.0156 0.0176 4.1839 (1.0, 0.0) 0.0027 0.0028 0.0033 0.0041 0.0050 1.6564 gap as discussed in the Introduction between users’ profiles and jobs’ descriptions in the dataset, which makes different entity type (user-item) comparisons less effective than using one kind of entity type itself (item-item) This is reasonable since the textual data in the RecSys2016 dataset is anonymized into ids so words with the same meaning but in different forms are treated as totally distinct On the other hand, in the CareerBuilder2012 dataset, the performance of the model drops quite dramatically The optimal (w1, w2) pair is (0.6, 0.4) which does not change much from (0.5, 0.5) in the matching model and the data fields used also persist This is because the contentbased method can only make recommendations for only 15200/35315 target users who applied for at least one job in the past in the CareerBuilder2012 dataset The remaining 20115 new target users are recommended with the item popularity algorithm which is not effective as the matching approach in this case The number of new target users in the RecSys2016 dataset (29%) is much smaller than in the CareerBuilder2012 dataset (57%) so it hurts the performance of the content-based approach less The loss curves during a 500-epoch training of the RGCN model in the RecSys2016 and CareerBuilder2012 dataset are respectively shown in Figure 3.1 and 3.2 The curves indicate that the two models have converged after 500 epochs In the RecSys2016 dataset, the content-based method which is generally the best model is further combined with the matching method instead of item popularity The overall results of different approaches are manifested in Table 3.5 At first glance, there is no specific algorithm dominating the others on every evaluation metric The item popularity method is not personalized but still achieves better results than the matching one The matching algorithm has a better M ap@1 than the item popularity approach but surprisingly the item popularity one is better with the other metrics Since the item popularity method is better than the matching one in general which might be due to the existing recommendation algorithms in the platform, this facilitates the corresponding hybrid approach That’s why hybrid approach using the item popularity method is 64 Figure 3.1: RGCN’s loss curve during training in RecSys2016 Figure 3.2: RGCN’s loss curve during training in CareerBuilder2012 65 Table 3.3: Performance of Content-based model with different weights in the Recsys2016 dataset (w1, w2) Map@1 Map@5 Map@10 Map@30 Map@150 RSScore (0.0, 1.0) 0.0539 0.0352 0.0346 0.0354 0.0360 4.5402 (0.1, 0.9) 0.0554 0.0356 0.0350 0.0358 0.0364 4.5414 (0.2, 0.8) 0.0552 0.0353 0.0348 0.0356 0.0362 4.4921 (0.3, 0.7) 0.0549 0.0349 0.0344 0.0351 0.0356 4.3887 (0.4, 0.6) 0.0543 0.0342 0.0337 0.0343 0.0347 4.1898 (0.5, 0.5) 0.0532 0.0331 0.0325 0.0330 0.0333 3.8986 (0.6, 0.4) 0.0515 0.0315 0.0308 0.0312 0.0315 3.5835 (0.7, 0.3) 0.0506 0.0304 0.0297 0.0301 0.0303 3.3722 (0.8, 0.2) 0.0503 0.0302 0.0295 0.0299 0.0301 3.3133 (0.9, 0.1) 0.0502 0.0301 0.0294 0.0298 0.0300 3.2988 (1.0, 0.0) 0.0028 0.0047 0.0052 0.0059 0.0062 1.5364 Table 3.4: Performance of Content-based model with different weights in the CareerBuilder2012 dataset (w1, w2) Map@1 Map@5 Map@10 Map@30 Map@150 RSScore (0.0, 1.0) 0.0029 0.0023 0.0024 0.0025 0.0027 0.4866 (0.1, 0.9) 0.0040 0.0033 0.0034 0.0037 0.0040 0.7775 (0.2, 0.8) 0.0049 0.0043 0.0045 0.0050 0.0055 1.1730 (0.3, 0.7) 0.0052 0.0048 0.0052 0.0058 0.0065 1.4730 (0.4, 0.6) 0.0056 0.0051 0.0056 0.0062 0.0069 1.5932 (0.5, 0.5) 0.0058 0.0054 0.0058 0.0064 0.0071 1.6388 (0.6, 0.4) 0.0061 0.0056 0.0061 0.0068 0.0075 1.7312 (0.7, 0.3) 0.0060 0.0056 0.0061 0.0068 0.0075 1.7314 (0.8, 0.2) 0.0060 0.0057 0.0062 0.0068 0.0076 1.7336 (0.9, 0.1) 0.0060 0.0057 0.0062 0.0068 0.0075 1.7258 (1.0, 0.0) 0.0007 0.0012 0.0014 0.0017 0.0021 0.6035 better than the one using matching approach The collaborative filtering algorithm achieves the best RSScore = 6.7410 but the content-based one is the best at other evaluation metrics, which might indicate that collaborative is better for recommending a 30-item list in general The graph neural network approach is slightly better than the matching algorithm, but unexpectedly falls behind every other model One common metric used to determine the graph’s sparseness is the edge density Edge density is defined as the ratio of the number of edges in the graph to the maximum number of edges possible in the graph To determine the strength of connectivity between nodes within a graph, the node’s degree metric which is the number of edges connected to it is used The reason the GNN model performs poorly for this task might be because the graph is too sparse and the connectivity between the user and item nodes is too weak since the edge density of the graph is 0.0011% which is too poor and the average user 66 Table 3.5: Performance of different models in the RecSys2016 dataset Model Map@1 Map@5 Map@10 Map@30 Map@150 RSScore Popularity 0.0001 0.0069 0.0074 0.0092 0.0098 2.7400 Matching 0.0018 0.0018 0.0020 0.0024 0.0028 0.7715 CB + Popularity 0.0554 0.0356 0.0350 0.0358 0.0364 4.5414 CF + Popularity 0.0327 0.0304 0.0317 0.0341 0.0355 6.7410 GNN + Popularity 0.0001 0.0024 0.0026 0.0033 0.0040 1.1577 CB + Matching 0.0561 0.0340 0.0333 0.0337 0.0342 3.9489 Table 3.6: Performance of different models in the CareerBuilder2012 dataset Model Map@1 Map@5 Map@10 Map@30 Map@150 RSScore Popularity 0.0001 0.0003 0.0003 0.0004 0.0005 0.1248 Matching 0.0131 0.0138 0.0153 0.0173 0.0193 4.7470 CB + Popularity 0.0061 0.0057 0.0062 0.0068 0.0075 1.7336 CF + Popularity 0.0168 0.0145 0.0153 0.0171 0.0184 3.9252 GNN + Popularity 0.0000 0.0000 0.0001 0.0001 0.0002 0.0381 CF + Matching 0.0262 0.0244 0.0262 0.0294 0.0318 7.0675 and item node’s degree respectively is 10.9 and 8.4 which is small Overall, the contentbased combined with the item popularity approach is considered the best algorithm for the recommendation system in this dataset The collaborative filtering technique also achieves good results on every metric and is just slightly behind the content-based one In the CareerBuilder2012 dataset, the collaborative filtering method which is the best model is further combined with the matching method instead of item popularity Table 3.6 shows the overall results of different methods in this dataset Different from the RecSys2016, the item popularity of this dataset performs poorly and the matching method is a significant improvement which might be due to the impact of the existing recommendation algorithms in CareerBuilder The content-based approach also performs more poorly than the matching one because the number of new target users is 57% which is big and hurts the performance of the content-based method as discussed above The graph neural networks method in the CareerBuilder2012 dataset performs worse than the one in the RecSys2016 dataset and becomes the worst among these algorithms This might be because the graph in this dataset is also sparse and the connectivity between the user and item nodes is even weaker since the edge density of the graph is 0.0014% and the average user and item node’s degree respectively is 5.0 and 4.2 which is minor The hybrid technique based on the collaborative filtering and matching approach dominates the others, including the hybrid technique based on the collaborative filtering and item popularity method on every evaluation metric To sum up, the best-performing model in the RecSys2016 and CareerBuilder2012 is respectively the hybrid model based on content-based combined with the item popularity 67 method and collaborative filtering combined with the matching method This aligns with the statement ”the performance of the different models differs considerably across datasets” [13] This implies that the results are dataset dependent As mentioned, the two datasets RecSys2016 and CareerBuilder2012 are the only publicly available datasets that are suitable for the job recommendation system problem the thesis could find Although these competitions’ datasets are commonly used after completion of the competition to train and validate job recommendation systems when no dataset is available, most studies in this field use their own private data sets Moreover, the test sets of these datasets are only accessible during the contest period and are restricted now despite the author’s attempts Most of the literature based on the RecSys2016 dataset was published in 2016 to serve the challenge and utilized the competition’s test set and evaluation measure as well For the CareerBuilder2012 dataset, the number of publications is small with different evaluation metrics Despite the difference in the test set, comparisons between the thesis’s results and the results from the original competition using corresponding evaluation metrics are still drawn With the RecSys2016 dataset, our collaborative filtering with item popularity approach reaches RSScore = 6.7410 which outperforms the top score in the original 2016 challenge (681,707.38 for 150,000 target users which is equivalent to RSScore = 4.5447) About the CareerBuilder2012, the best result in the thesis is M AP @150 = 0.0318 using collaborative filtering combined with the matching approach falls behind the top score for the public and private test set in the original contest is M AP @150 = 0.1815 for approximately 7,000 target users and M AP @150 = 0.1828 for about 16,000 users It is worth noting that the differences in test sets may have contributed significantly to the variations in performance metrics across the studies, especially when the original test set in the CareerBuilder2012 challenge used separate interactions during 13 weeks that split into windows Overall, the classical approaches for recommendation systems like user-item matching, content-based and collaborative filtering still show their strength and practical application on these two datasets Hybrid models from these methods also play a crucial role in overcoming the limitations of each individual technique: the cold-start problem in collaborative filtering where it is difficult to provide recommendations for new users or heavy dependency on item features, which may not capture the diversity of user preferences in content-based recommendation system By combining the approaches, a hybrid model can leverage the advantages of each technique and provide more accurate and diverse recommendations 68 Conclusion and Future work Conclusion Currently, Vietnam has not had much research on the problem of the job recommendation system Also, there is a lack of labor market datasets in general and test data in particular, this thesis could not compare the results with other authors’ but only make comparisons within the different algorithms implemented by the author themselves The main contributions of this thesis are as follows: • Analyze jobs and candidates’ behavior and profile from two labor market datasets: RecSys2016 and CareerBuilder2012, providing valuable insights for a better understanding of the job market • Various job recommendation algorithms such as item popularity, user-item matching, content-based, user-based collaborative filtering, and graph neural network were implemented and experimented on the two datasets • The performance and practical usability of these models in the context of job recommendation systems were assessed utilizing two metrics: M ap@K and RSScore The content-based combined with the item popularity approach is generally the best model in the RecSys2016 dataset at M ap@1 = 0.0554, M ap@5 = 0.0356, M ap@10 = 0.0350, M AP @30 = 0.0364, M ap@150 = 0.0364, RSScore = 4.5414 On the other hand, in the CareerBuilder2012 dataset, the collaborative filtering associated with the matching method outperforms the others and achieves the highest evaluation metrics: M ap@1 = 0.0262, M ap@5 = 0.0244, M ap@10 = 0.0262, M ap@30 = 0.0294, M ap@150 = 0.0318, RSScore = 7.0675 In addition to the achieved contribution of the thesis, there are still some shortcomings such as: • Lack of information extractor from textual data in the CareerBuilder2012 dataset, which might give some useful features • Lack of test data and comprehensive comparison to other research • Ethical concerns about privacy and data security, particularly if the system collects sensitive information from job seekers were not considered in this thesis 69 A part of the research in the thesis has been published at The 35th International Conference on Industrial, Engineering & Other Applications of Applied Intelligent Systems (IEA/AIE 2022) in Japan [5] Future work Finding a job is a complex process, influenced by both explicit and implicit factors Various recommendation algorithms were implemented, experimented and evaluated with much potential for future improvement Since building artificial intelligence models is an iterative process, this thesis suggests some future development directions as follows: • Collecting and incorporating more data sources Moreover, some NLP models, such as Named Entity Recognition, Skill Extractor, and Skill Standardizer, might be involved thereby enhancing the embedding representation in particular and the overall results of the recommendation system in general • Exploring more advanced machine learning models In this thesis, some classical recommendation models such as content-based and collaborative were utilized Graph neural networks-based approach was also experimented with but did not reach the expected performance This indicates that there is room for advanced models such as deep learning could be explored to improve the system’s performance • Designing an end-to-end recommendation system architecture Since a user often comes into the system with a CV/resume, the system can have an information extraction model implemented beforehand to get useful features from the user’s CV/resume and job description without expecting the user to fill in his/her profile on the platform, which advances the user’s experiment • Applying inductive learning to make recommendations for new, unseen users, items Since new entities appearing in the online job portal are increasing dramatically, inductive reasoning might help improve users’ engagement impressively 70 Bibliography [1] Fabian Abel et al “Recsys challenge 2016: Job recommendations” In: Proceedings of the 10th ACM conference on recommender systems 2016, pp 425–426 [2] Marko Balabanovi´c and Yoav Shoham “Fab: content-based, collaborative recommendation” In: Communications of the ACM 40.3 (1997), pp 66–72 [3] Wojciech Krupa Ben Hamner Road Warrior Job Recommendation Challenge 2012 url: https://kaggle.com/competitions/job-recommendation [4] Erion C ¸ ano and Maurizio Morisio “Hybrid recommender systems: A systematic literature review” In: Intelligent Data Analysis 21.6 (2017), pp 1487–1524 [5] Hai-Nam Cao et al “Synonym Prediction for Vietnamese Occupational Skills” In: Advances and Trends in Artificial Intelligence Theory and Practices in Artificial Intelligence: 35th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2022, Kitakyushu, Japan, July 19–22, 2022, Proceedings Springer 2022, pp 351–362 [6] Cheng Guo et al “How integration helps on cold-start recommendations” In: Proceedings of the Recommender Systems Challenge 2017 2017, pp 1–6 [7] Ken Lazarus Performance-Based Matching: Using Machine Learning to Quickly Find Recruiters with Proven Success Tech rep Scout Exchange, 2018 [8] Kuan Liu et al “Temporal learning and sequence modeling for a job recommender system” In: Proceedings of the Recommender Systems Challenge 2016, pp 1–4 [9] Saket Maheshwary and Hemant Misra “Matching resumes to jobs via deep siamese network” In: Companion Proceedings of the The Web Conference 2018 2018, pp 87–88 [10] Yoosof Mashayekhi et al “A challenge-based survey of e-recruitment recommendation systems” In: arXiv preprint arXiv:2209.05112 (2022) [11] Motebang Daniel Mpela and Tranos Zuva “A mobile proximity job employment recommender system” In: 2020 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD) IEEE 2020, pp 1–6 [12] Mirko Polato and Fabio Aiolli “A preliminary study on a recommender system for the job recommendation challenge” In: Proceedings of the Recommender Systems Challenge 2016, pp 1–4 71