torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 196.00 MiB. GPU 0 has a total capacty of 79.11 GiB of which 29.56 MiB is free
#21
by
butujuzipi
- opened
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 196.00 MiB. GPU 0 has a total capacty of 79.11 GiB of which 29.56 MiB is free
How should I solve this problem?
You need to load the model with bf16
You can hardly run this model on a single GPU. Try a smaller model or use model parallelism to run it on multiple GPUs instead.
Chester111
changed discussion status to
closed