Inquiry About Response Errors During eval
Hello, first of all, thank you for releasing the model.
For the evaluation of the performance of the model, certain characters/words are repeatedly appearing in the responses during eval. It seems that someone else has posted a similar issue in the discussion. The same problem appears to occur when the model is deployed on a GPU.
The environment I tested on is AWS SageMaker A100 40G. I would like to inquire if there is a way to operate without any issues in a GPU environment.
For reference, the tests were conducted with Transformers version 4.41.
Thank you!
Could you please provide the information below?
- input with special tokens:
[|system|]You are EXAONE model from LG AI Research, a helpful assistant.[|endofturn|]\n[|user|]λμ μμμ λ§ν΄λ΄\n[|assistant|]
- output with special tokens:
EXAONE 3.0 λͺ¨λΈλ‘μ, μ μ μ£Όλ λͺ©μ μ μ¬μ©μμκ² μ ννκ³ μ μ©ν μ 보λ₯Ό μ 곡νλ κ²μ
λλ€. μ λ λ€μν μ§λ¬Έμ λ΅λ³νκ³ , λ¬Έμ λ₯Ό ν΄κ²°νλ©°, νμ΅κ³Ό μ°κ΅¬λ₯Ό λκΈ° μν΄ μ€κ³λμμ΅λλ€. μ κ° λμΈ μ μλ λ€λ₯Έ λ°©λ²μ΄ μλ€λ©΄ μΈμ λ μ§ λ§μν΄ μ£ΌμΈμ![|endofturn|]
- generation configurations:
A100 40GB, torch==2.3.1, transformers==4.43.3, flash-attn==2.6.3, accelerate==0.33.0, greedy search (or top-k=?, top-p=?, ...), repetition penalty=1.0, ...
The above issue has been resolved.
However, could you provide the performance details for HumanEval-x?
Unfortunately, the HumanEval-x benchmark was not evaluated.
Ok. Thank you!