How to Forecast the Students’ Learning Outcomes Based on Factors of Interactive Activities in a Blended Learning Course Minh-Duc Le Hoa-Huy Nguyen Duc-Loc Nguyen VNU University of VNU Center for Education VNU University of Engineering and Accreditation Engineering and Technology, Vietnam Technology, Vietnam Vietnam National University National University Hanoi, Hanoi, 144 Xuan Thuy, Cau National University Hanoi, 144 Xuan Thuy, Cau Giay, 144 Xuan Thuy, Cau Giay, Giay, Hanoi, Vietnam Hanoi, Vietnam Hanoi, Vietnam duclm@vnu.edu.vn This paper summarizes the research results of identifying the influencing factors in the online learning phase of a blended learning course From such factors, we propose a model for predicting student outcomes In our study, we have conducted several models in order to predict the student’s learning outcomes, using a course of 231 participants Obtained data from the logs file of an LMS system is analyzed using learning analytics and machine learning techniques, and the results propose that the four factors are the number of views, the number of posts, the number of forum views, and the number of on-time submitted assignments impact on the student’s learning outcomes For the forecast of the final exam grade based on the results of the formative assessment tests, Bayesian Ridge is the most accurate among the four conducted models (Linear Regression, KNR, SVM, Bayesian Ridge) Our study can be a useful material for lecturers and course designers in effectively organizing blended learning courses CCS Concepts •Applied computing➝Education➝E-learning Keywords Blended learning; forecast students’ learning outcomes; learning analytics; learning activities INTRODUCTION Currently, blended learning is a popular trend in modern education because of its superiority Blended learning is widely understood as a combination of traditional and online learning [1] It is also is considered as one of the most suitable solutions for institutions when they initiate to shift from traditional learning to a new form [2] Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page Copyrights for components of this work owned by others than ACM must be honored Abstracting with credit is permitted To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee Request permissions from Permissions@acm.org ICFET 2020, June 5–8, 2020, Tokyo, Japan © 2020 Association for Computing Machinery ACM ISBN 978-1-4503-7533-7/20/06…$15.00 https://doi.org/10.1145/3404709.3404711 The corresponding author VNU University of Engineering and Technology, Vietnam National University Hanoi, 144 Xuan Thuy, Cau Giay, Hanoi, Vietnam huynguyen@vnu.edu.v 17020871@vnu.edu.vn vietanh@vnu.edu.vn n ABSTRACT * Viet Anh Nguyen* In general, when students participate in online learning activities, they often encounter a major problem regarding the lack of tools that support them in course recommendations and forecast results Along with the strong development of machine learning and data mining techniques, Learning Analytics (LA) is a crucial topic in the educational research community [3] Overall, analytics learning technology consists of five steps: data collection, reporting, forecasting, action, and improvement [4] Thanks to the application of LA techniques, data analysis of learners when participating in online activities is tremendously facilitated [5] The results of this analysis not only provide useful information about the learning process and the relationship between learners but also can be used as the basis for recommendations to support learners to improve their learning results When learners participate in online learning activities, most of them not receive enough support from lecturers [6] Therefore, identifying the factors that influence learners' results when they are participating in online learning activities is significant in building time recommendations to help learners to regulate their learning process We conduct this study to identify the factors that influence student achievement when taking the online phase of the course, with a special focus on the interactive activities that the learner engaged These factors are the basis for predicting student learning outcomes and can also be considered as useful indicators for lecturers and course designers in designing the courses 1.1 Research Context University of Engineering and Technology, Vietnam National University, Hanoi has started to apply informative technology to support teaching and learning activities since 2003 At first, our university has applied some LMS / LCMS systems such as Moodle, Blackboard as support tools for these activities In this phase, these tools are used as an additional information channel to assist learners to access learning resources supplied by the lecturers Recently, there are about 10% of the courses have been implemented under the blended learning course in which most online activities combined with some traditional lessons in each course Taking the courses using this model, learners always play an active role throughout the learning process by interacting within the learning system During the learning process, one of the legitimate learner’s demand is how to effectively participate in these courses In other words, for the designer and lecturer, what are the necessary actions in order to create a course that is effective for learners colleagues showed that the long-short term memory (LSTM) model was most effective [16], [17] 1.2 Research Questions In addition, studies focusing on finding factors that impact the students’ learning outcomes are also concerned The results of Khalil's research [18] and colleagues have identified factors such as course length, student motivation, and interaction during learning as well as background knowledge that have an impact on student dropout rates The study to identify the factors that affect learners' retention in MOOCs shows that the interaction between students and faculty effectively increases retention [19] Learners' interaction with learning systems through click stream is seen as a predictor of academic performance [20] The results of the study indicate that this factor along with the demographic characteristics has a great influence on student performance The interaction between students and lecturers is also considered as a factor to the students' engagement with the course [21] The learners’ sentiments in forums when participating in the MOOCs system are also factors that affect student performance as well as results Research by Tucker and colleagues have shown a correlation between student’s sentiments and their performance [22] In such a context, our research aims to build an effective course model to support learners To gradually resolve this issue, we conducted this study to answer the following research questions: (1) What factors affect students' learning outcomes when participating in a blended learning model? (2) Is it possible to develop a model for predicting students’ learning outcomes based on those factors? 1.3 Paper Structure In the next section, we will present an overview of the recent studies in identifying the factors that affect learners' results as well as studies related to the models to predict the students’ learning outcomes In section 3, we describe in detail the organization of research implementation, including research subjects, procedures, data as well as models The results of the study are presented in detail in Part Some discussions are presented in Section Finally, Section presents some conclusions and research directions in the future LITERATURE REVIEW A learner early identification with unsatisfactory academic results and timely warnings will help the learners significantly improve results[7], [8], especially when they are participating in an elearning environment in which lacks close supervision compared to traditional learning Many studies have proposed several models and methods to predict learners' results in order to support learners Typically, models apply data mining techniques, machine learning to figure out problems [9] The dropout student forecast model based on a set of demographic attributes Research results show that the logistic regression model with the best results with AUC ranges from 0.66 to 0.73 [10] Along with the approach of using a logistic regression model, Marbouti and his colleagues implemented a predictive model in weeks 2, and of the courses The results of the study imply that the logistic regression model is reliable in predicting students at risk [11] In the blended-learning environment, we can analyze some information from students gathered from the learning management system Several approaches using learning analytics to predict students Lu and colleagues [12] used learning activities from online and offline learning and proposed 21 variables to predict student performance Their findings showed that four online factors impacted student results Predicting student success in blended learning, Macarini and colleagues had used data from LMS Moodle what collected student activities based on student interaction The results showed that factors can predict student success, and detect student at-risk even in the first week [13] In the study about the student dropout prediction which is based on the student's personal characteristics as well as academic performance [14], the authors used typical machine learning methods such as Artificial Neural Network, Decision Tree, Bayesian Networks for data sets from the Open University of China shows the Decision Tree method is the best Xing [15] and colleagues conducted a study on 3617 students who took the course has 11 modules over weeks in a MOOCs system to predict student dropout by using existing and past features to forecast based on Bayesian Network and Decision Tree (C4.5) models Research results show that these factors have a positive impact on the quality of intervention and support instructors When performing student dropout prediction studies in the MOOCs system, Fei and On the other hand, scores are also considered as the predictors of results as well as student performance, especially the results evaluated under formative assessment can be seen as the basis for predicting final results [23], [24] In order to predict the student’s grade for the average group of students, the regression model should be used, in contrast to predicting student’s performance for each student, the SVM model should be considered for better results [25] METHODS 3.1 Participants and Procedures The study was conducted with 23 first-year students who first took the blended learning course with the subject topic is “Introduction of Information Technology” The course is organized for 12 weeks There are two content types in this course In offline learning activities, the former is theoretical content which takes place in three weeks in 1st, 4th, and 10th, the latter is practical content which takes place in weeks 2nd, 5th, and 11th All learning content and interactive activities such as online learning, forum, online assignments, are implemented on the Moodle system at the website (https://onlinecourses.uet.vnu.edu.vn) The results are assessed by formative form It includes three online tests for theoretical content and three exercises that are submitted through the LMS system for practical content The final exam consists of two parts, the theory section consists of 40 questions for 30 minutes of online testing, and the practical section is a computerbased test for 60 minutes 3.2 Data Data is collected from the system's log databases which are collected when learners start attending the course until the end of the course between September and December 2019 From the data source which stores learners' interactive activities, we have developed a tool as a plugin for the Moodle system to analyze and statistic the learner's interaction activities The data of each student includes the number of content views, the number of postings, the number of forum posts, the number of forum views, the number of on-time submitted assignments, the number of late submitted assignments, the number of un submitted assignments, online test scores, and final online exam 3.3 Model In response to research questions, we conducted a number of students’ learning outcomes prediction models: (1) To forecast student’s learning outcomes based on factors that result from the learners’ interaction with the system, (2) To forecast the final exam results based on formative assessment tests’ grade To predict the students’ learning outcomes based on interactive activities, we build the model based on the Regression machine learning technique with the R tool In our model, the chosen activities as factors are independent variables, the predictor variable is the final grade of the course To select the parameters that best fit the model, we have performed two main steps are: selecting the best regression model and selecting the best factors for a model In order to forecast the final exam grade based on the results of the assignment’s results, we tested four predictive models: Linear Regression, SVR (Epsilon-Support Vector Regression), KNR (Regression based on KNN), and Bayesian Ridge RESULTS 4.1 Descriptive of the Interactive Learning Activities Table describes the descriptive statistics of the interaction of the learning activities The statistical results showed that each student interacts with the system to view learning resources 12 times per week on average A student who has the lowest number of interactions is more than times per week It is noticeable that students actively interact with the learning system On average, the students submitted about two assignments per five assignments that they need to submit Table Descriptive statistics the interaction of the learning activities Min Max Mean Std Deviations Number of views 55 340 155.7 67.64 Number of posts 13 58 29.3 13.27 Number of forum views 77 21.17 14.13 Number of forum posts 3.5 1.96 Number submitted assignments of 2.7 1.03 late 1.35 0.72 Variable Number of submitted assignments 4.2 To Forecast the Student’s Learning Outcomes Based on Statistics of Interactive Activities The appropriate parameters were selected for the model to predict the final exam results are independent variables: number of views, number of posts, number of forum views, number of forum posts, number of submitted assignments, number of late submitted assignments Experimenting by using all subsets regression method with six variables to select the best models for different parameters, we obtained several models predicting the score of the final exam: (1) Correlated with a factor, showing the correlation between final exam results and the number of posts The correlation coefficient of the post parameter is 2.015 means with a 1% increase rate of the number of posts, the student's score increased by 2.015%; (2) Correlated with two factors, showing the correlation between the end of the final exam result with the number of posts and the number of on-time submitted assignments The correlation coefficient of the final score with the post parameter is 2.31, the coefficient of correlation between the final score and the number of on-time submitted assignments is 2.14; (3) Correlated with three factors, the results show a correlation between the test results with the number of posts, the number of forum views, and the number of on-time submitted assignments; (4) Correlated with four factors, the results showed that a correlation between the final exam result and the number of views, the number of posts, the number of forum visits, and the number of on-time submitted assignments In more detail, the correlation coefficient of the final exam result with the number of exercises submitted on time, the number of posts, the number of times participating in the forum, the number of views is 2.65, 2.31, 1.71, 1.32 respectively We use the Akaike Information Criterion (AIC) method to make comparisons among models The correlation model between the final exam result and four factors has the smallest AIC value is 171.8, it can be considered as an optimal choice in this case to forecast students’ learning outcomes 4.3 To Forecast the Student’s Learning Outcomes based on the Formative Assessment Results In this experiment, students take three online assessment tests and one final exam The data in Figure shows that there is a correlation between the results of the tests and the final exam Figure The correlation between the results of the tests and the final exam In this study, we use three functions to evaluate the models: average error average (MAE), mean square error (MSE), determination coefficient (coefficient of determination r2) The specific results of these measures are shown in Table Table Evaluation results of test models Model MSE MAE R2_loss Linear Regression 1.076808 0.883435 0.893004 SVR 1.150097 0.932311 1.014602 KNR(k=20) 1.113869 0.912662 0.953646 Bayesian Ridge 1.075545 0.883538 0.890447 According to the visual assessment, we can see that we should use the Bayesian Ridge model to predict the result as the MAE error value of the model is the lowest also be considered because if it has more data to train, the accuracy will ameliorate and not over fit training data However, after looking at the learning curve of the models in Figure 2, although Bayesian Ridge is the most accurate model, its learning curve shows the overfitting of this model A similar situation is observed in the Linear Regression model Considering the KNR model (K = 20), it might be relatively effective when the training data is not much, but when the training data increases, the accuracy of the model does not improve The course that has a variety of interactive activities can create a more effectively predicting model because we can easily choose more factors for the model Empirical results can suggest that we should design a course with a variety of interactive activities and those need to be implemented regularly in each course According to the learning curve, we shall appreciate the Support Vector Regression model in which bias and variance of the relatively balanced model without overfitting Besides, the accuracy of the model tends to increase if more data is added 5.2 Implication It is possible to use the results of the formative assessment in a course as a criterion to predict the student’s final exam Building models for predicting the final grade based on component scores will help learners to identify the results in the course as an early warning This result may also be a suggestion for the course designers on how to design effective learning activities in the course CONCLUSION In this research, we have tried to identify the factors that affect the student’s learning outcomes in the blended learning courses and proposed some models using the machine learning techniques to forecast them Findings show that factors related to student-content interaction and student-student interaction are significant Moreover, the test results of the formative assessments are reliable in order to develop the model for predicting the students' final exam grades The results also show that the Bayesian Ridge method should be applied for this problem Although the research was conducted on a small scale, the results were positive and could be applied in many courses In further studies, we will develop the models for predicting student learning outcomes based on a combination of component scores in the learning process and online learning activities Figure The learning curve of four models ACKNOWLEDGMENTS DISCUSSION 5.1 The Research Questions’ Results This work has been supported by Vietnam National University, Hanoi under Project QG.20.57 For reporting results based on statistics of interactive activities: Considering in the online learning phase, ignoring factors such as background knowledge and learners’ level, activities that support learners to interact with the system that affects their learning outcomes The results obtained from the forecasting models also confirm that the interaction between learners - content, learners learners have a great impact on learning outcomes as previous studies [20], [21] However, the optimal model selection for predicting student’s learning outcomes based on these factors is still a detrimental issue that requires further studies due to the number of statistical activities is not much in the small dataset Furthermore, the data collected from these interactive activities completely depends on whether those can be carried out in the course or not If a learning activity is not in the course, we cannot use them as the factors for the predicting model REFERENCES For forecasting student’s learning outcomes based on formative assessment results in the course: In order to perform this, the course needs to deploy many exercises to evaluate periodically After the students have the assignment results, they can get a forecast value of the final exam result that is the basis for students to adjust their learning strategies in the future The results show that we should use the Bayesian Ridge model in case the data does not have too much variation or the difference between training data and actual data is small because its accuracy is highest compared to other models However, the Support Vector Regression method should [1] C J Bonk and C R Graham, “The Handbook of Blended Learning,” In, 2006 [2] J Kang and G A Seomun, “Evaluating Web-Based Nursing Education’s Effects: A Systematic Review and MetaAnalysis,” Western Journal of Nursing Research 2018 [3] A Chiappe, “Learning Analytics in 21st century education: a review,” pp 971–991, 2017 [4] J P Campbell, P B DeBlois, and D G Oblinger, “Academic Analytics,” Educ Rev., vol 42, no October, pp 40–57, 2007 [5] O Viberg, M Hatakka, O Bälter, and A Mavroudi, “The Current Landscape of Learning Analytics in Higher Education,” Comput Human Behav., vol 89, no October 2017, pp 98–110, 2018 [6] E Nkenke et al., “Acceptance of technology-enhanced learning for a theoretical radiological science course: A randomized controlled trial,” BMC Med Educ., 2012 [7] S Aud et al., “The condition of education 2013,” 2013 [8] M Hlosta, Z Zdrahal, and J Zendulka, “Ouroboros : early identification of at-risk students,” LAK17 - Seventh Int Learn Anal Knowl Conf., pp 6–15, 2017 [9] M Mayilvaganan and D Kalpanadevi, “Comparison of classification techniques for predicting the performance of students academic environment,” in 2014 International Conference on Communication and Network Technologies, ICCNT 2014, 2015 [10] L Aulck, N Velagapudi, J Blumenstock, and J West, “Predicting Student Dropout in Higher Education,” Proc ICML Work #Data4Good Mach Learn Soc Good Appl New York, NY, USA, 2016 [11] F Marbouti, H Diefes-Dux, and J Strobel, “Building Course-Specific Regression-based Models to Identify At-risk Students,” 2015 [12] O H T Lu, A Y Q Huang, J C H Huang, A J Q Lin, H Ogata, and S J H Yang, “Applying learning analytics for the early prediction of students’ academic performance in blended learning,” Educ Technol Soc., vol 21, no 2, pp 220–232, 2018 [13] L A B Macarini, C Cechinel, M F B Machado, V F C Ramos, and R Munoz, “Predicting students success in blended learning-Evaluating different interactions inside learning management systems,” Appl Sci., vol 9, no 24, 2019 [17] Y Jung and J Lee, “Learning Engagement and Persistence in Massive Open Online Courses (MOOCS),” Comput Educ., 2018 [18] H Khalil and M Ebner, “MOOCs Completion Rates and Possible Methods to Improve Retention - A Literature Review,” EdMedia World Conf Educ Media Technol., vol 2014, no 1, pp 1305–1313, 2014 [19] K S Hone and G R El Said, “Exploring the factors affecting MOOC retention: A survey study,” Comput Educ., vol 98, pp 157–168, 2016 [20] H Waheed, S U Hassan, N R Aljohani, J Hardman, S Alelyani, and R Nawaz, “Predicting academic performance of students from VLE big data using deep learning models,” Comput Human Behav., vol 104, no November 2018, p 106189, 2020 [21] S S Jaggars and D Xu, “How online course design features influence student performance?,” Comput Educ., 2016 [22] C Tucker, B K Pursel, and A Divinsky, “Mining studentgenerated textual data in MOOCs and quantifying their effects on student performance and learning outcomes,” Comput Educ J., vol 5, no 4, pp 84–95, 2014 [14] M Tan and P Shao, “Prediction of student dropout in Elearning program through the use of machine learning method,” Int J Emerg Technol Learn., vol 10, no 1, pp 11–17, 2015 [23] Q Jin, P K Imbrie, J J J Lin, and X Chen, “A multioutcome hybrid model for predicting student success in engineering,” in ASEE Annual Conference and Exposition, Conference Proceedings, 2011 [15] W Xing, X Chen, J Stein, and M Marcinkowski, “Temporal predication of dropouts in MOOCs: Reaching the low hanging fruit through stacking generalization,” Comput Human Behav., vol 58, pp 119–129, 2016 [24] F Marbouti, H A Diefes-dux, and K Madhavan, “Computers & Education Models for early prediction of atrisk students in a course using standards-based grading,” Comput Educ., vol 103, pp 1–15, 2016 [16] M Fei and D Y Yeung, “Temporal Models for Predicting Student Dropout in Massive Open Online Courses,” Proc 15th IEEE Int Conf Data Min Work ICDMW 2015, pp 256–263, 2016 [25] S Huang and N Fang, “Ac 2010-190: Regression Models for Predicting Student Academic Performance in an Engineering Dynamics Course,” Age, 2010 ... the factors that affect the student’s learning outcomes in the blended learning courses and proposed some models using the machine learning techniques to forecast them Findings show that factors. .. the students? ?? learning outcomes based on interactive activities, we build the model based on the Regression machine learning technique with the R tool In our model, the chosen activities as factors. .. interactive activities, we have developed a tool as a plugin for the Moodle system to analyze and statistic the learner's interaction activities The data of each student includes the number of