Model Memory Requirements
#3
by
nvip1204
- opened
Can I run it on 4x3090?
number of parameter * size of each parameter in bytes = 72 * 10^9 * 2 bytes = 144 GB of VRAM if running in bf16, 72 GB for fp8, 36 for 4bit datatype so just add up the VRAM for your infrastructure and see if that falls under this