README.md

Short Answer Classification Model

This is a finetuned MPnet model with a classification head designed to predict whether a short answer is correct.

Training

This model was trained on the MultiRC dataset, a corpus of 20,422 answers to 5,130 questions about 456 source texts. Before training, chatGPT was used to generate correct answers to each question given the source. This was used as the reference answer. The model takes the reference answer together with the target answer and predicts whether the target answer is correct. 15% of sources along with all of their questions and answers were withheld from the training data and used to evaluate the model. The model had an F1 of 0.81 on the test set.

Input sequences to the model should follow this pattern: {target_answer} </s> {reference_answer}