Add instructions to run R1-AWQ on SGLang

README.md

@@ -13,6 +13,8 @@ AWQ of DeepSeek R1.
 
 This quant modified some of the model code to fix an overflow issue when using float16.
 
+## Serving with vLLM
+
 To serve using vLLM with 8x 80GB GPUs, use the following command:
 ```sh
 VLLM_WORKER_MULTIPROC_METHOD=spawn python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 65536 --max-num-batched-tokens 65536 --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.97 --dtype float16 --served-model-name deepseek-reasoner --model cognitivecomputations/DeepSeek-R1-AWQ
@@ -26,3 +28,13 @@ Inference speed with batch size 1 and short prompt:
 Note:
 - Inference speed will be better than FP8 at low batch size but worse than FP8 at high batch size, this is the nature of low bit quantization.
 - vLLM supports MLA for AWQ now, you can run this model with full context length on just 8x 80GB GPUs.
+
+## Serving with SGLang
+
+```sh
+python3 -m sglang.launch_server --model cognitivecomputations/DeepSeek-R1-AWQ --tp 8 --trust-remote-code --dtype half
+```
+
+Note:
+- AWQ does not support BF16, so add the `--dtype half` flag if AWQ is used for quantization.
+- For more information about running DeepSeek-R1 using SGLang, feel free to check out their [documentation](https://docs.sglang.ai/references/deepseek.html).