| common_init_from_params: setting dry_penalty_last_n to ctx_size = 768 | |
| common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) | |
| system_info: n_threads = 6 (n_threads_batch = 6) / 12 | Metal : EMBED_LIBRARY = 1 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | ACCELERATE = 1 | AARCH64_REPACK = 1 | | |
| multiple_choice_score: there are 817 tasks in prompt | |
| multiple_choice_score: selecting 750 random tasks from 817 tasks available | |
| multiple_choice_score: preparing task data...done | |
| multiple_choice_score : calculating TruthfulQA score over 750 tasks. | |
| task acc_norm | |
| 1 100.00000000 | |
| 2 50.00000000 | |
| 3 33.33333333 | |
| 4 25.00000000 | |
| 5 40.00000000 | |
| 6 33.33333333 | |
| 7 28.57142857 | |
| 8 37.50000000 | |
| 9 33.33333333 | |
| 10 30.00000000 | |
| 11 27.27272727 | |
| 12 25.00000000 | |
| 13 30.76923077 | |
| 14 28.57142857 | |
| 15 26.66666667 | |
| 16 25.00000000 | |
| 17 23.52941176 | |
| 18 22.22222222 | |
| 19 21.05263158 | |
| 20 20.00000000 | |
| 21 19.04761905 | |
| 22 18.18181818 | |
| 23 17.39130435 | |
| 24 20.83333333 | |
| 25 20.00000000 | |
| 26 19.23076923 | |
| 27 18.51851852 | |
| 28 17.85714286 | |
| 29 20.68965517 | |
| 30 23.33333333 | |
| 31 25.80645161 | |
| 32 28.12500000 | |
| 33 30.30303030 | |
| 34 29.41176471 | |
| 35 28.57142857 | |
| 36 27.77777778 | |
| 37 29.72972973 | |
| 38 28.94736842 | |
| 39 30.76923077 | |
| 40 30.00000000 | |
| 41 29.26829268 | |
| 42 28.57142857 | |
| 43 30.23255814 | |
| 44 31.81818182 | |
| 45 33.33333333 | |
| 46 32.60869565 | |
| 47 31.91489362 | |
| 48 33.33333333 | |
| 49 32.65306122 | |
| 50 34.00000000 | |
| 51 33.33333333 | |
| 52 32.69230769 | |
| 53 32.07547170 | |
| 54 31.48148148 | |
| 55 32.72727273 | |
| 56 32.14285714 | |
| 57 31.57894737 | |
| 58 32.75862069 | |
| 59 33.89830508 | |
| 60 35.00000000 | |
| 61 34.42622951 | |
| 62 35.48387097 | |
| 63 34.92063492 | |
| 64 34.37500000 | |
| 65 33.84615385 | |
| 66 33.33333333 | |
| 67 32.83582090 | |
| 68 32.35294118 | |
| 69 33.33333333 | |
| 70 32.85714286 | |
| 71 32.39436620 | |
| 72 33.33333333 | |
| 73 34.24657534 | |
| 74 33.78378378 | |
| 75 33.33333333 | |
| 76 34.21052632 | |
| 77 33.76623377 | |
| 78 34.61538462 | |
| 79 34.17721519 | |
| 80 35.00000000 | |
| 81 34.56790123 | |
| 82 35.36585366 | |
| 83 34.93975904 | |
| 84 34.52380952 | |
| 85 34.11764706 | |
| 86 33.72093023 | |
| 87 33.33333333 | |
| 88 32.95454545 | |
| 89 33.70786517 | |
| 90 33.33333333 | |
| 91 32.96703297 | |
| 92 32.60869565 | |
| 93 32.25806452 | |
| 94 31.91489362 | |
| 95 31.57894737 | |
| 96 32.29166667 | |
| 97 32.98969072 | |
| 98 32.65306122 | |
| 99 33.33333333 | |
| 100 33.00000000 | |
| 101 33.66336634 | |
| 102 33.33333333 | |
| 103 33.00970874 | |
| 104 33.65384615 | |
| 105 33.33333333 | |
| 106 33.96226415 | |
| 107 34.57943925 | |
| 108 34.25925926 | |
| 109 33.94495413 | |
| 110 34.54545455 | |
| 111 34.23423423 | |
| 112 34.82142857 | |
| 113 35.39823009 | |
| 114 35.96491228 | |
| 115 35.65217391 | |
| 116 35.34482759 | |
| 117 35.04273504 | |
| 118 34.74576271 | |
| 119 34.45378151 | |
| 120 35.00000000 | |
| 121 35.53719008 | |
| 122 35.24590164 | |
| 123 35.77235772 | |
| 124 36.29032258 | |
| 125 36.80000000 | |
| 126 37.30158730 | |
| 127 37.00787402 | |
| 128 36.71875000 | |
| 129 37.20930233 | |
| 130 37.69230769 | |
| 131 37.40458015 | |
| 132 37.12121212 | |
| 133 36.84210526 | |
| 134 36.56716418 | |
| 135 37.03703704 | |
| 136 36.76470588 | |
| 137 37.22627737 | |
| 138 37.68115942 | |
| 139 38.12949640 | |
| 140 37.85714286 | |
| 141 37.58865248 | |
| 142 37.32394366 | |
| 143 37.06293706 | |
| 144 37.50000000 | |
| 145 37.24137931 | |
| 146 36.98630137 | |
| 147 36.73469388 | |
| 148 36.48648649 | |
| 149 36.24161074 | |
| 150 36.00000000 | |
| 151 36.42384106 | |
| 152 36.18421053 | |
| 153 35.94771242 | |
| 154 35.71428571 | |
| 155 35.48387097 | |
| 156 35.89743590 | |
| 157 35.66878981 | |
| 158 35.44303797 | |
| 159 35.22012579 | |
| 160 35.00000000 | |
| 161 35.40372671 | |
| 162 35.18518519 | |
| 163 34.96932515 | |
| 164 34.75609756 | |
| 165 35.15151515 | |
| 166 35.54216867 | |
| 167 35.32934132 | |
| 168 35.71428571 | |
| 169 35.50295858 | |
| 170 35.29411765 | |
| 171 35.08771930 | |
| 172 34.88372093 | |
| 173 34.68208092 | |
| 174 35.05747126 | |
| 175 34.85714286 | |
| 176 35.22727273 | |
| 177 35.02824859 | |
| 178 34.83146067 | |
| 179 34.63687151 | |
| 180 34.44444444 | |
| 181 34.25414365 | |
| 182 34.06593407 | |
| 183 33.87978142 | |
| 184 33.69565217 | |
| 185 33.51351351 | |
| 186 33.87096774 | |
| 187 33.68983957 | |
| 188 33.51063830 | |
| 189 33.33333333 | |
| 190 33.15789474 | |
| 191 32.98429319 | |
| 192 32.81250000 | |
| 193 32.64248705 | |
| 194 32.98969072 | |
| 195 32.82051282 | |
| 196 32.65306122 | |
| 197 32.48730964 | |
| 198 32.32323232 | |
| 199 32.16080402 | |
| 200 32.50000000 | |
| 201 32.33830846 | |
| 202 32.67326733 | |
| 203 32.51231527 | |
| 204 32.35294118 | |
| 205 32.68292683 | |
| 206 32.52427184 | |
| 207 32.85024155 | |
| 208 32.69230769 | |
| 209 32.53588517 | |
| 210 32.38095238 | |
| 211 32.22748815 | |
| 212 32.07547170 | |
| 213 31.92488263 | |
| 214 31.77570093 | |
| 215 31.62790698 | |
| 216 31.94444444 | |
| 217 31.79723502 | |
| 218 32.11009174 | |
| 219 31.96347032 | |
| 220 31.81818182 | |
| 221 31.67420814 | |
| 222 31.53153153 | |
| 223 31.39013453 | |
| 224 31.69642857 | |
| 225 31.55555556 | |
| 226 31.41592920 | |
| 227 31.71806167 | |
| 228 31.57894737 | |
| 229 31.44104803 | |
| 230 31.30434783 | |
| 231 31.16883117 | |
| 232 31.03448276 | |
| 233 30.90128755 | |
| 234 31.19658120 | |
| 235 31.06382979 | |
| 236 31.35593220 | |
| 237 31.22362869 | |
| 238 31.51260504 | |
| 239 31.38075314 | |
| 240 31.25000000 | |
| 241 31.12033195 | |
| 242 30.99173554 | |
| 243 30.86419753 | |
| 244 30.73770492 | |
| 245 30.61224490 | |
| 246 30.48780488 | |
| 247 30.36437247 | |
| 248 30.24193548 | |
| 249 30.52208835 | |
| 250 30.40000000 | |
| 251 30.67729084 | |
| 252 30.95238095 | |
| 253 30.83003953 | |
| 254 30.70866142 | |
| 255 30.98039216 | |
| 256 30.85937500 | |
| 257 30.73929961 | |
| 258 30.62015504 | |
| 259 30.88803089 | |
| 260 30.76923077 | |
| 261 31.03448276 | |
| 262 30.91603053 | |
| 263 30.79847909 | |
| 264 31.06060606 | |
| 265 31.32075472 | |
| 266 31.20300752 | |
| 267 31.08614232 | |
| 268 30.97014925 | |
| 269 30.85501859 | |
| 270 30.74074074 | |
| 271 30.62730627 | |
| 272 30.51470588 | |
| 273 30.40293040 | |
| 274 30.29197080 | |
| 275 30.18181818 | |
| 276 30.07246377 | |
| 277 29.96389892 | |
| 278 29.85611511 | |
| 279 29.74910394 | |
| 280 30.00000000 | |
| 281 30.24911032 | |
| 282 30.14184397 | |
| 283 30.38869258 | |
| 284 30.28169014 | |
| 285 30.17543860 | |
| 286 30.41958042 | |
| 287 30.66202091 | |
| 288 30.90277778 | |
| 289 30.79584775 | |
| 290 30.68965517 | |
| 291 30.92783505 | |
| 292 31.16438356 | |
| 293 31.39931741 | |
| 294 31.29251701 | |
| 295 31.52542373 | |
| 296 31.75675676 | |
| 297 31.98653199 | |
| 298 31.87919463 | |
| 299 31.77257525 | |
| 300 31.66666667 | |
| 301 31.56146179 | |
| 302 31.45695364 | |
| 303 31.35313531 | |
| 304 31.25000000 | |
| 305 31.14754098 | |
| 306 31.04575163 | |
| 307 30.94462541 | |
| 308 31.16883117 | |
| 309 31.06796117 | |
| 310 30.96774194 | |
| 311 30.86816720 | |
| 312 31.08974359 | |
| 313 30.99041534 | |
| 314 30.89171975 | |
| 315 30.79365079 | |
| 316 30.69620253 | |
| 317 30.91482650 | |
| 318 30.81761006 | |
| 319 30.72100313 | |
| 320 30.62500000 | |
| 321 30.52959502 | |
| 322 30.74534161 | |
| 323 30.65015480 | |
| 324 30.55555556 | |
| 325 30.46153846 | |
| 326 30.36809816 | |
| 327 30.58103976 | |
| 328 30.79268293 | |
| 329 30.69908815 | |
| 330 30.90909091 | |
| 331 30.81570997 | |
| 332 31.02409639 | |
| 333 31.23123123 | |
| 334 31.13772455 | |
| 335 31.34328358 | |
| 336 31.54761905 | |
| 337 31.45400593 | |
| 338 31.36094675 | |
| 339 31.56342183 | |
| 340 31.47058824 | |
| 341 31.37829912 | |
| 342 31.57894737 | |
| 343 31.48688047 | |
| 344 31.68604651 | |
| 345 31.88405797 | |
| 346 31.79190751 | |
| 347 31.98847262 | |
| 348 31.89655172 | |
| 349 31.80515759 | |
| 350 31.71428571 | |
| 351 31.62393162 | |
| 352 31.81818182 | |
| 353 31.72804533 | |
| 354 31.63841808 | |
| 355 31.83098592 | |
| 356 31.74157303 | |
| 357 31.93277311 | |
| 358 31.84357542 | |
| 359 31.75487465 | |
| 360 31.66666667 | |
| 361 31.85595568 | |
| 362 31.76795580 | |
| 363 31.95592287 | |
| 364 32.14285714 | |
| 365 32.05479452 | |
| 366 31.96721311 | |
| 367 32.15258856 | |
| 368 32.33695652 | |
| 369 32.24932249 | |
| 370 32.16216216 | |
| 371 32.07547170 | |
| 372 32.25806452 | |
| 373 32.17158177 | |
| 374 32.35294118 | |
| 375 32.26666667 | |
| 376 32.18085106 | |
| 377 32.36074271 | |
| 378 32.27513228 | |
| 379 32.18997361 | |
| 380 32.36842105 | |
| 381 32.28346457 | |
| 382 32.46073298 | |
| 383 32.37597911 | |
| 384 32.29166667 | |
| 385 32.20779221 | |
| 386 32.12435233 | |
| 387 32.04134367 | |
| 388 32.21649485 | |
| 389 32.13367609 | |
| 390 32.05128205 | |
| 391 32.22506394 | |
| 392 32.14285714 | |
| 393 32.31552163 | |
| 394 32.48730964 | |
| 395 32.40506329 | |
| 396 32.32323232 | |
| 397 32.49370277 | |
| 398 32.41206030 | |
| 399 32.33082707 | |
| 400 32.25000000 | |
| 401 32.16957606 | |
| 402 32.33830846 | |
| 403 32.25806452 | |
| 404 32.17821782 | |
| 405 32.09876543 | |
| 406 32.01970443 | |
| 407 31.94103194 | |
| 408 31.86274510 | |
| 409 31.78484108 | |
| 410 31.70731707 | |
| 411 31.63017032 | |
| 412 31.79611650 | |
| 413 31.71912833 | |
| 414 31.88405797 | |
| 415 32.04819277 | |
| 416 31.97115385 | |
| 417 32.13429257 | |
| 418 32.05741627 | |
| 419 31.98090692 | |
| 420 31.90476190 | |
| 421 32.06650831 | |
| 422 31.99052133 | |
| 423 32.15130024 | |
| 424 32.31132075 | |
| 425 32.23529412 | |
| 426 32.15962441 | |
| 427 32.08430913 | |
| 428 32.00934579 | |
| 429 31.93473193 | |
| 430 31.86046512 | |
| 431 31.78654292 | |
| 432 31.94444444 | |
| 433 32.10161663 | |
| 434 32.25806452 | |
| 435 32.18390805 | |
| 436 32.33944954 | |
| 437 32.26544622 | |
| 438 32.42009132 | |
| 439 32.34624146 | |
| 440 32.50000000 | |
| 441 32.42630385 | |
| 442 32.57918552 | |
| 443 32.50564334 | |
| 444 32.65765766 | |
| 445 32.58426966 | |
| 446 32.51121076 | |
| 447 32.43847875 | |
| 448 32.36607143 | |
| 449 32.51670379 | |
| 450 32.66666667 | |
| 451 32.59423503 | |
| 452 32.52212389 | |
| 453 32.67108168 | |
| 454 32.59911894 | |
| 455 32.52747253 | |
| 456 32.45614035 | |
| 457 32.38512035 | |
| 458 32.31441048 | |
| 459 32.24400871 | |
| 460 32.17391304 | |
| 461 32.10412148 | |
| 462 32.25108225 | |
| 463 32.39740821 | |
| 464 32.54310345 | |
| 465 32.68817204 | |
| 466 32.61802575 | |
| 467 32.54817987 | |
| 468 32.69230769 | |
| 469 32.62260128 | |
| 470 32.55319149 | |
| 471 32.48407643 | |
| 472 32.62711864 | |
| 473 32.55813953 | |
| 474 32.48945148 | |
| 475 32.63157895 | |
| 476 32.56302521 | |
| 477 32.49475891 | |
| 478 32.42677824 | |
| 479 32.35908142 | |
| 480 32.50000000 | |
| 481 32.43243243 | |
| 482 32.57261411 | |
| 483 32.50517598 | |
| 484 32.64462810 | |
| 485 32.78350515 | |
| 486 32.71604938 | |
| 487 32.85420945 | |
| 488 32.78688525 | |
| 489 32.92433538 | |
| 490 32.85714286 | |
| 491 32.99389002 | |
| 492 33.13008130 | |
| 493 33.06288032 | |
| 494 33.19838057 | |
| 495 33.13131313 | |
| 496 33.06451613 | |
| 497 32.99798793 | |
| 498 32.93172691 | |
| 499 32.86573146 | |
| 500 32.80000000 | |
| 501 32.73453094 | |
| 502 32.66932271 | |
| 503 32.60437376 | |
| 504 32.73809524 | |
| 505 32.67326733 | |
| 506 32.60869565 | |
| 507 32.54437870 | |
| 508 32.67716535 | |
| 509 32.80943026 | |
| 510 32.74509804 | |
| 511 32.68101761 | |
| 512 32.61718750 | |
| 513 32.55360624 | |
| 514 32.49027237 | |
| 515 32.62135922 | |
| 516 32.75193798 | |
| 517 32.88201161 | |
| 518 32.81853282 | |
| 519 32.75529865 | |
| 520 32.69230769 | |
| 521 32.62955854 | |
| 522 32.56704981 | |
| 523 32.50478011 | |
| 524 32.44274809 | |
| 525 32.38095238 | |
| 526 32.31939163 | |
| 527 32.44781784 | |
| 528 32.38636364 | |
| 529 32.32514178 | |
| 530 32.26415094 | |
| 531 32.20338983 | |
| 532 32.33082707 | |
| 533 32.45778612 | |
| 534 32.39700375 | |
| 535 32.33644860 | |
| 536 32.27611940 | |
| 537 32.40223464 | |
| 538 32.34200743 | |
| 539 32.28200371 | |
| 540 32.22222222 | |
| 541 32.16266174 | |
| 542 32.10332103 | |
| 543 32.04419890 | |
| 544 31.98529412 | |
| 545 31.92660550 | |
| 546 32.05128205 | |
| 547 32.17550274 | |
| 548 32.29927007 | |
| 549 32.24043716 | |
| 550 32.18181818 | |
| 551 32.12341198 | |
| 552 32.24637681 | |
| 553 32.36889693 | |
| 554 32.31046931 | |
| 555 32.25225225 | |
| 556 32.19424460 | |
| 557 32.13644524 | |
| 558 32.07885305 | |
| 559 32.02146691 | |
| 560 32.14285714 | |
| 561 32.26381462 | |
| 562 32.20640569 | |
| 563 32.14920071 | |
| 564 32.26950355 | |
| 565 32.21238938 | |
| 566 32.33215548 | |
| 567 32.45149912 | |
| 568 32.57042254 | |
| 569 32.51318102 | |
| 570 32.45614035 | |
| 571 32.57443082 | |
| 572 32.51748252 | |
| 573 32.46073298 | |
| 574 32.40418118 | |
| 575 32.52173913 | |
| 576 32.63888889 | |
| 577 32.75563258 | |
| 578 32.69896194 | |
| 579 32.81519862 | |
| 580 32.75862069 | |
| 581 32.87435456 | |
| 582 32.81786942 | |
| 583 32.76157804 | |
| 584 32.87671233 | |
| 585 32.82051282 | |
| 586 32.93515358 | |
| 587 32.87904600 | |
| 588 32.82312925 | |
| 589 32.76740238 | |
| 590 32.88135593 | |
| 591 32.82571912 | |
| 592 32.77027027 | |
| 593 32.71500843 | |
| 594 32.65993266 | |
| 595 32.60504202 | |
| 596 32.55033557 | |
| 597 32.49581240 | |
| 598 32.44147157 | |
| 599 32.38731219 | |
| 600 32.33333333 | |
| 601 32.44592346 | |
| 602 32.55813953 | |
| 603 32.50414594 | |
| 604 32.45033113 | |
| 605 32.39669421 | |
| 606 32.50825083 | |
| 607 32.45469522 | |
| 608 32.40131579 | |
| 609 32.51231527 | |
| 610 32.45901639 | |
| 611 32.40589198 | |
| 612 32.51633987 | |
| 613 32.46329527 | |
| 614 32.41042345 | |
| 615 32.35772358 | |
| 616 32.30519481 | |
| 617 32.41491086 | |
| 618 32.36245955 | |
| 619 32.47172859 | |
| 620 32.41935484 | |
| 621 32.36714976 | |
| 622 32.47588424 | |
| 623 32.42375602 | |
| 624 32.37179487 | |
| 625 32.32000000 | |
| 626 32.26837061 | |
| 627 32.21690590 | |
| 628 32.32484076 | |
| 629 32.27344992 | |
| 630 32.22222222 | |
| 631 32.17115689 | |
| 632 32.12025316 | |
| 633 32.06951027 | |
| 634 32.01892744 | |
| 635 31.96850394 | |
| 636 31.91823899 | |
| 637 31.86813187 | |
| 638 31.81818182 | |
| 639 31.76838811 | |
| 640 31.87500000 | |
| 641 31.82527301 | |
| 642 31.93146417 | |
| 643 31.88180404 | |
| 644 31.83229814 | |
| 645 31.93798450 | |
| 646 31.88854489 | |
| 647 31.99381762 | |
| 648 32.09876543 | |
| 649 32.04930663 | |
| 650 32.15384615 | |
| 651 32.25806452 | |
| 652 32.36196319 | |
| 653 32.46554364 | |
| 654 32.41590214 | |
| 655 32.36641221 | |
| 656 32.46951220 | |
| 657 32.57229833 | |
| 658 32.52279635 | |
| 659 32.62518968 | |
| 660 32.57575758 | |
| 661 32.52647504 | |
| 662 32.47734139 | |
| 663 32.42835596 | |
| 664 32.37951807 | |
| 665 32.33082707 | |
| 666 32.43243243 | |
| 667 32.38380810 | |
| 668 32.33532934 | |
| 669 32.28699552 | |
| 670 32.38805970 | |
| 671 32.33979136 | |
| 672 32.29166667 | |
| 673 32.24368499 | |
| 674 32.19584570 | |
| 675 32.29629630 | |
| 676 32.39644970 | |
| 677 32.34859675 | |
| 678 32.30088496 | |
| 679 32.25331370 | |
| 680 32.20588235 | |
| 681 32.15859031 | |
| 682 32.25806452 | |
| 683 32.21083455 | |
| 684 32.30994152 | |
| 685 32.26277372 | |
| 686 32.21574344 | |
| 687 32.16885007 | |
| 688 32.12209302 | |
| 689 32.07547170 | |
| 690 32.02898551 | |
| 691 32.12735166 | |
| 692 32.22543353 | |
| 693 32.32323232 | |
| 694 32.27665706 | |
| 695 32.23021583 | |
| 696 32.32758621 | |
| 697 32.28120516 | |
| 698 32.23495702 | |
| 699 32.18884120 | |
| 700 32.28571429 | |
| 701 32.23965763 | |
| 702 32.19373219 | |
| 703 32.14793741 | |
| 704 32.10227273 | |
| 705 32.05673759 | |
| 706 32.15297450 | |
| 707 32.24893918 | |
| 708 32.20338983 | |
| 709 32.15796897 | |
| 710 32.11267606 | |
| 711 32.06751055 | |
| 712 32.02247191 | |
| 713 31.97755961 | |
| 714 31.93277311 | |
| 715 31.88811189 | |
| 716 31.84357542 | |
| 717 31.93863319 | |
| 718 32.03342618 | |
| 719 32.12795549 | |
| 720 32.08333333 | |
| 721 32.03883495 | |
| 722 31.99445983 | |
| 723 31.95020747 | |
| 724 31.90607735 | |
| 725 31.86206897 | |
| 726 31.95592287 | |
| 727 31.91196699 | |
| 728 31.86813187 | |
| 729 31.82441701 | |
| 730 31.78082192 | |
| 731 31.73734610 | |
| 732 31.83060109 | |
| 733 31.78717599 | |
| 734 31.74386921 | |
| 735 31.70068027 | |
| 736 31.65760870 | |
| 737 31.75033921 | |
| 738 31.70731707 | |
| 739 31.79972936 | |
| 740 31.89189189 | |
| 741 31.84885290 | |
| 742 31.80592992 | |
| 743 31.76312248 | |
| 744 31.85483871 | |
| 745 31.81208054 | |
| 746 31.90348525 | |
| 747 31.99464525 | |
| 748 31.95187166 | |
| 749 32.04272363 | |
| 750 32.13333333 | |
| Final result: 32.1333 +/- 1.7063 | |
| Random chance: 19.8992 +/- 1.4588 | |