Post
893
📢 If you're interesting in quick application of target sentiment analysis towards your data, you might be insterested in using fine-tuned FlanT5-xl version. Reason is a quick performance: I've added batching support for series of sentiment analysis models in this card:
nicolay-r/sentiment-analysis-advances-665ba391e0eba729021ea101
The provider implementation:
https://github.com/nicolay-r/nlp-thirdgate/blob/master/llm/transformers_flan_t5.py
📺 How to quick launch:
https://github.com/nicolay-r/bulk-chain/blob/master/test/test_provider_batching.py
Reason for using? experimenting in out-of domain, the noticed the performance of xl version similar to LLaMA-3-3b-instruct.
🔑 Key takeaways of adaptaiont:
- paddings and truncation strategies for batching mode:
- https://huggingface.co./docs/transformers/en/pad_truncation
- add_special_tokens=False causes a drastic changes in the result behaviour (FlanT5 models).
💥 Crashes on pad_token_id=50256 during generation proces.
🔻 use_bf16 mode performs 3 times slower on CPU.
🚀 Performance for BASE sized model:
nicolay-r/flan-t5-tsa-thor-base
17.2 it/s (prompt) and 5.22 it/s (3-step CoT) (CPU Core i5-1140G7)
There are other domain-oriented models could be launched via the same provider:
nicolay-r/flan-t5-emotion-cause-thor-base
Reference: https://github.com/huggingface/transformers/issues/26061
nicolay-r/sentiment-analysis-advances-665ba391e0eba729021ea101
The provider implementation:
https://github.com/nicolay-r/nlp-thirdgate/blob/master/llm/transformers_flan_t5.py
📺 How to quick launch:
https://github.com/nicolay-r/bulk-chain/blob/master/test/test_provider_batching.py
Reason for using? experimenting in out-of domain, the noticed the performance of xl version similar to LLaMA-3-3b-instruct.
🔑 Key takeaways of adaptaiont:
- paddings and truncation strategies for batching mode:
- https://huggingface.co./docs/transformers/en/pad_truncation
- add_special_tokens=False causes a drastic changes in the result behaviour (FlanT5 models).
💥 Crashes on pad_token_id=50256 during generation proces.
🔻 use_bf16 mode performs 3 times slower on CPU.
🚀 Performance for BASE sized model:
nicolay-r/flan-t5-tsa-thor-base
17.2 it/s (prompt) and 5.22 it/s (3-step CoT) (CPU Core i5-1140G7)
There are other domain-oriented models could be launched via the same provider:
nicolay-r/flan-t5-emotion-cause-thor-base
Reference: https://github.com/huggingface/transformers/issues/26061
