MOOC 评价系统中同伴互评概率模型研究

Probabilistic Models of Peer Assessment in MOOC System

2012年起，基于网络、针对大众人群的大规模开放在线课程呈井喷式发展。目前的MOOC虽然能支持视频课程、论坛、测试和评价等功能，但对于学习者学习效果的评价和给予反馈的能力仍受到限制。在线学习的学习效果评价方法中，选择题和判断题等客观类试题可以通过机器进行评判反馈，但一些主观类试题，比如数学演算、设计问题和论文等一些复杂和开放性的作业任务就无法通过机器评判反馈。针对这一情况，一些MOOC平台正逐步引入同伴互评机制。虽然同伴互评机制的设立使得主观类试题得到有效评价，但学习者对同伴建议的准确性和权威性表示怀疑。调查发现，94%学生更喜欢老师评语。如此，需要依据一定的理论或过程模型保证同伴互评的准确度、信服度和价值。本文构建了三种关联复杂度不同的同伴互评概率模型来提升MOOC评价系统中主观试题评分的客观性和准确性，并利用Cousera中“人机交互冶课程的相关数据组来评测各同伴互评概率模型的准确度。评测方法采用了吉布斯采样法和期望最大化法。文章通过对使用三种概率模型得到的评测结果与通过Cousera平台同伴互评系统所得到的相应结果进行了比较，结果显示，准确度有显著提高。本文构建的模型可以提升同伴互评系统整体效果，且最高达到30%。文章最后还对同伴互评概率模型的进一步改进方向和其在MOOC系统中的实际应用进行了探讨，包括增加新的关注参数，例如评分者在评分时投入的关注度等。

With the rapid development of information technology， more and more traditional industries have been transformed， and the field of education has also been changed under the influence of new technologies， such as the Internet. Since 2012， there has been an explosive growth of MOOCs ( Massive Open Online Courses). Currently， the MOOC supports video lectures， forum， testing and other instructional functions. However， the learning evaluation and feedback functions are still very limited. Among the methods of evaluating learning， multiple choice questions could be accomplished by computers. However， for subjective questions， like complex and open-ended tasks， it would be very difficult. In order to alleviate such issues， a peer review mechanism has been introduced into many MOOC platforms. The establishment of a peer evaluation mechanism both reduces the subjectivity of questions and also improves the learner's abilities of learning through reading other assignments. However， learners are still skeptical about the accuracy and authority of the grades acquired through peer review. A surveys found that 94% of students preferred teachers’ grading.Consequently， it is beneficial to add theoretic and procedural models into the peer review to maintain its accuracy， credibility and value. In this paper， we constructed three probability models to improve the objectivity and accuracy of scores to subjective questions through peer review in MOOC system. The records was used to evaluate the peer assessment models from Coursera’s HCI course. They were divided into two groups， HCI1 and HCI2， among which those records from Coursera’s peer assessment system were named HCI1. Then the peer grading system was refined in several ways. For example， graders were divided into different language groups， such as English and Spanish， to address concerns of assignments being graded by non-native speakers as well as the observed patriotic grading effect. After these optimizations， Coursera’s peer assessment system was worked again. Those new records gained were named HCI2. We used these two record groups to evaluate the accuracies of three probability models. There were two evaluation methods in this paper， Gibbs Sampling and Expectation Maximization (EM). We found that the results of two methods were close to each other， while EM was quicker and Gibbs Sampling was more natural. Those results from the evaluation process were compared to the Coursera’s peer assessment results. The accuracies were found to be greatly improved. Most of all， the improvement effect of Model 3 (PG3) was the best owing to considering more related parameters. Through PG3， the root-mean-square error (RMSE) was cut by 33% and 31% ， with regard to HCI1 and HCI2 successively， the quantity with deviation from the true value less than 5% was increased by 19% and 15% ， and the quantity with deviation less than 10% was increased by 14% and 9% successively. Meanwhile， the mean deviation was greatly reduced.The three models， by focusing on those principal parameters associated with graders like grader bias and grader reliability， would make the peer assessment results more reliable， accurate and efficient. Compared to the existing peer assessment system， those models in this paper would enhance the overall effect up to 30% . Finally， we discussed the further optimizations and practical applications of those models， including increasing the parameters， such as the attention rates of graders.