Can it run faster than 2 tokens/second on one A100?

by aibarito-ua - opened Jul 26, 2023

Jul 26, 2023

Hello!
I am trying to run this model on one A100, but the speed is quite slow - 2 tokens/sec. Does anybody know how to make it faster?
I have tried 8-bit-mode and it is allocating twice less gpu memory, but the speed is not increasing.

Limerobot

upstage org Aug 2, 2023

https://www.google.com/search?q=llama+inference+speed+optimization&source=lnt&tbs=qdr:m&sa=X&ved=2ahUKEwjTw-3X5LyAAxULG4gKHa23BrcQpwV6BAgDEA8&biw=2168&bih=1065&dpr=2.36
I hope this helps you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment