riiider commited on
Commit
ffc70af
1 Parent(s): 02a8244

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +29 -6
README.md CHANGED
@@ -1,5 +1,9 @@
1
  ---
2
  license: cc-by-nc-sa-4.0
 
 
 
 
3
  ---
4
 
5
  # How to use
@@ -8,16 +12,35 @@ This is a model used in the EMNLP 2022 work, "Evaluating the Knowledge Dependenc
8
  We fine-tuned the [roberta-large](https://huggingface.co/roberta-large) model on RACE dataset. <br>
9
  Please refer to our publication to check how to use the model to evaluate the question.
10
 
 
 
 
 
11
  Code | https://github.com/riiid/question-score
12
 
13
- Paper preprint | https://arxiv.org/abs/2211.11902
14
 
15
  # How to cite
16
  ```
17
- @inproceedings{moon2022eval,
18
- title={Evaluating the Knowledge Dependency of Questions,
19
- author={Moon, H., Yang, Y., Shin, J., Yu, H., Lee, S., Jeong, M., Park, J., Kim, M., & Choi, S.},
20
- booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
21
- year={2022}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  }
 
23
  ```
 
1
  ---
2
  license: cc-by-nc-sa-4.0
3
+ datasets:
4
+ - race
5
+ language:
6
+ - en
7
  ---
8
 
9
  # How to use
 
12
  We fine-tuned the [roberta-large](https://huggingface.co/roberta-large) model on RACE dataset. <br>
13
  Please refer to our publication to check how to use the model to evaluate the question.
14
 
15
+ RACE finetune:
16
+ - validation_loss: 0.488
17
+ - validation_acc : 0.836
18
+
19
  Code | https://github.com/riiid/question-score
20
 
21
+ Paper | https://aclanthology.org/2022.emnlp-main.718/
22
 
23
  # How to cite
24
  ```
25
+ @inproceedings{moon-etal-2022-evaluating,
26
+ title = "Evaluating the Knowledge Dependency of Questions",
27
+ author = "Moon, Hyeongdon and
28
+ Yang, Yoonseok and
29
+ Yu, Hangyeol and
30
+ Lee, Seunghyun and
31
+ Jeong, Myeongho and
32
+ Park, Juneyoung and
33
+ Shin, Jamin and
34
+ Kim, Minsam and
35
+ Choi, Seungtaek",
36
+ booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
37
+ month = dec,
38
+ year = "2022",
39
+ address = "Abu Dhabi, United Arab Emirates",
40
+ publisher = "Association for Computational Linguistics",
41
+ url = "https://aclanthology.org/2022.emnlp-main.718",
42
+ pages = "10512--10526",
43
+ abstract = "The automatic generation of Multiple Choice Questions (MCQ) has the potential to reduce the time educators spend on student assessment significantly. However, existing evaluation metrics for MCQ generation, such as BLEU, ROUGE, and METEOR, focus on the n-gram based similarity of the generated MCQ to the gold sample in the dataset and disregard their educational value.They fail to evaluate the MCQ{'}s ability to assess the student{'}s knowledge of the corresponding target fact. To tackle this issue, we propose a novel automatic evaluation metric, coined Knowledge Dependent Answerability (KDA), which measures the MCQ{'}s answerability given knowledge of the target fact. Specifically, we first show how to measure KDA based on student responses from a human survey.Then, we propose two automatic evaluation metrics, KDA{\_}disc and KDA{\_}cont, that approximate KDA by leveraging pre-trained language models to imitate students{'} problem-solving behavior.Through our human studies, we show that KDA{\_}disc and KDA{\_}soft have strong correlations with both (1) KDA and (2) usability in an actual classroom setting, labeled by experts. Furthermore, when combined with n-gram based similarity metrics, KDA{\_}disc and KDA{\_}cont are shown to have a strong predictive power for various expert-labeled MCQ quality measures.",
44
  }
45
+
46
  ```