Update README.md

README.md

@@ -127,6 +127,7 @@ Then you just need to run the TGI v2.2.0 (or higher) Docker container as follows
 
 ```bash
 docker run --gpus all --shm-size 1g -ti -p 8080:80 \
+  -v hf_cache:/data \
   -e MODEL_ID=hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
   -e NUM_SHARD=4 \
   -e QUANTIZE=awq \
@@ -139,7 +140,7 @@ docker run --gpus all --shm-size 1g -ti -p 8080:80 \
 > [!NOTE]
 > TGI will expose different endpoints, to see all the endpoints available check [TGI OpenAPI Specification](https://huggingface.github.io/text-generation-inference/#/).
 
-To send request to the deployed TGI endpoint compatible with [OpenAI specification](https://github.com/openai/openai-openapi) i.e. `/v1/chat/completions`:
+To send request to the deployed TGI endpoint compatible with [OpenAI OpenAPI specification](https://github.com/openai/openai-openapi) i.e. `/v1/chat/completions`:
 
 ```bash
 curl 0.0.0.0:8080/v1/chat/completions \
@@ -182,6 +183,59 @@ chat_completion = client.chat.completions.create(
 )
 ```
 
+### vLLM
+
+To run vLLM with Llama 3.1 70B Instruct AWQ in INT4, you will need to have Docker installed (see [installation notes](https://docs.docker.com/engine/install/)) and run the latest vLLM Docker container as follows:
+
+```bash
+docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000 \
+  -v hf_cache:/root/.cache/huggingface \
+  vllm/vllm-openai:latest \
+  --model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
+  --tensor-parallel-size 4 \
+  --max-model-len 4096
+```
+
+To send request to the deployed vLLM endpoint compatible with [OpenAI OpenAPI specification](https://github.com/openai/openai-openapi) i.e. `/v1/chat/completions`:
+
+```bash
+curl 0.0.0.0:8000/v1/chat/completions \
+  -X POST \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4",
+    "messages": [
+      {
+        "role": "system",
+        "content": "You are a helpful assistant."
+      },
+      {
+        "role": "user",
+        "content": "What is Deep Learning?"
+      }
+    ],
+    "max_tokens": 128
+  }'
+```
+
+Or programatically via the `openai` Python client (see [installation notes](https://github.com/openai/openai-python?tab=readme-ov-file#installation)) as follows:
+
+```python
+import os
+from openai import OpenAI
+
+client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key=os.getenv("VLLM_API_KEY", "-"))
+
+chat_completion = client.chat.completions.create(
+  model="hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4",
+  messages=[
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "What is Deep Learning?"},
+  ],
+  max_tokens=128,
+)
+```
+
 ## Quantization Reproduction
 
 > [!NOTE]