We introduce a novel Chinese LLM benchmark dataset called ANGO, aiming to provide more in-depth guidance for model training and evaluation. We introduce the format of a single-question multiple-keypoints dataset for the first time, which will provide the most complete description for each question, enabling the test results to comprehensively showcase the model's performance from multiple perspectives. Based on the single-question multiple-keypoints format, we design a more detailed and refined model capability classification system - the Keypoint Tree, which reflects the relationships between different keypoints. It includes a total of 171 specific model capabilities accumulated in 4 hierarchical levels. With the help of the KeyPoint Tree, the performance of models on multiple levels of capabilities can be quickly measured, and corresponding adjustments can be made. ANGO also involves two new question attributes: human accuracy and human error-prone options. Based on human accuracy, we propose a more detailed difficulty classification compared to previous benchmarks. By combining the human accuracy of the question itself, the human accuracy of the involved key points, and the actual score of the question, all questions are divided into 9 difficulty levels, providing a quantifiable reference for evaluating models of different difficulty.
+ +In addition to the innovative data, we propose a complete set of verification processes tailored for ANGO, which can provide fairer results compared to the current leaderboards. This includes conducting multiple experiments with option shuffling to mitigate the issue of data leakage, designing test set sampling strategies that fully utilize the characteristics of ANGO, and implementing elimination mechanisms for high-accuracy questions. Based on these, we establish a dynamic updating system for the test set, resembling a seasonal system. Thanks to these methods, ANGO can continually update the test results, ensuring the fairness and effectiveness of the leaderboard. By preserving the test results from multiple seasons, it can provide researchers with an overview of the current trends in optimizing models within the community.
+ +The data utilized in our study were exclusively obtained from the Administrative Proficiency Test, which serves as a significant component of the Chinese civil service examination.
+The Administrative Proficiency Test is entirely composed of multiple-choice questions and aims to evaluate the abilities and skills necessary for practical administrative work. This test covers a wide range of knowledge areas, including Expression& Comprehension , Data Analysis, Quantitative Relations, Judgement&Inference, and Common Knowledge. As a comprehensive assessment tool, it requires candidates to respond to a series of questions related to administrative work within a limited timeframe. These questions may involve policy formulation, problem-solving, personnel and resource management, as well as handling emergency situations. By formulating these questions, it facilitates the evaluation of candidates' analytical thinking, Judgement&Inference, problem-solving abilities, and language proficiency.
+The nature of the Administrative Proficiency Test necessitates candidates to tackle complex questions within a specified timeframe, making it an ideal testing environment for assessing the language capabilities of language models. Language models typically demonstrate excellent performance in generating and comprehending text, and this test provides concrete and intricate contexts that simulate real-world language communication and decision-making processes. By employing language models to answer these questions, we can evaluate their understanding of complex problems, Judgement&Inference abilities, as well as the accuracy and fluency of their language expressions.
+Furthermore, the Administrative Proficiency Test encompasses a broad coverage and diversity. It includes questions and scenarios from various administrative domains, such as government administration, social affairs, and economic development. This diversity aids in evaluating the language processing abilities of language models across different fields, thereby providing a more comprehensive understanding of their potential strengths and limitations in practical applications. Moreover, it offers valuable insights for future model improvements and applications.
+ANGO's data covers all 34 provinces in China and includes three different types of examinations conducted between 2008 and 2023, including formal and mock exams.
+In order to enhance the quality of our data, we employed a simple yet efficient preprocessing approach.
+Given that mock exams often include previous exam questions, our data contained numerous duplicates. To address this issue, we employed a straightforward strategy of removing duplicates based on the record ID obtained from the data source. As a result of this step, the size of our data was reduced to 88,799 instances.
+ +The data consisted of two types of images: formula pictures and other types (such as images containing graphics). However, since our primary focus was on Chinese Natural Language Processing (NLP) evaluation rather than the multi-modal domain, we opted to remove all records containing pure images. This resulted in the removal of 17,650 records.
+ +As mentioned earlier, our data still contained formula pictures, and we recognized the importance of including formulae to ensure diversity in our data. To address this, we extracted 8,144 unique formula images from a pool of 34,062 LaTeX formulas derived from 5,574 questions. These images were then processed using a Formula OCR (Optical Character Recognition) model, followed by manual verification to ensure formula accuracy. Ultimately, we obtained a clean data consisting of 71,149 instances.
+Here is an example record:
+ +
+ Question: Forward: Backward
+ Material: Please select the option that best resembles the relationship between the given words or phrases in the question stem.
+ Type: Single Choice
+ Options:
+ A. Urge: Advise
+ B. Ocean: Land
+ C. Vibration: Quiet
+ D. Extend: Compress
+ Choice: D
+ Difficulty: 4
+ KeyPoints: Semantic Relationship - Antonym
+ Human Accuracy: 79.564999
+ Human Count: 183494
+ Most Wrong: C
+ Solution: Step 1: Determine the logical relationship between the words in the question stem. The two words in the question stem are antonyms. Step 2: Determine the logical relationship between the options. The option that has the same logical relationship as the question stem is option D. Option A is a synonym relationship, option B is a parallel relationship, and in option C, the antonym of "quiet" should be "noisy" instead of "vibration". Therefore, the correct answer is D.
+ Source: 2011 Jiangsu Province Civil Service Recruitment Examination 'Administrative Aptitude Test' (Category A), Question 41
+ Formulas: 0
+
To mitigate the impact of data leakage during model pretraining on benchmark evaluations, we have employed multiple benchmark evaluation tricks to enhance fairness and real-time performance of the benchmarks.
+ +Sometimes, a model's correct answer to a specific question may not be due to mastering a certain ability or understanding the question, but rather because it has recognized patterns of token order in the training data. By shuffling the order of options in multiple-choice questions and making multiple predictions with the correct answer placed in different options, we can average the results to reduce the model's reliance on character order.
+ +Thanks to sampling strategies optimized for ANGO, we can periodically sample the test set and update the leaderboard. This prevents certain institutions or individuals from maliciously hacking ANGO to inflate the model's performance. However, due to the limited number of questions in some key areas, dynamic iteration may not be feasible for all questions.
+ +In addition to the aforementioned dynamic updating of season, a new question elimination mechanism has been proposed. This mechanism calculates the average accuracy of each question across all models for each iteration. Questions with accuracies exceeding a threshold are temporarily removed by ANGO to ensure reliable discrimination among questions in ANGO.