Setup MLC-LLM on CPU on UBUNTU 22.04 LTS
sudo apt update
sudo apt install ocl-icd-opencl-dev clinfo vulkan-tools
python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-ai-nightly
mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC
___________________________________________________________________________________________________________________________________________
$ mlc_llm --help
usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config,chat,serve,bench,package}
positional arguments:
{compile,convert_weight,gen_config,chat,serve,bench,package}
Subcommand to to run. (choices: compile, convert_weight, gen_config, chat, serve, bench, package)
options:
-h, --help show this help message and exit
____________________________________________________________________________________________________________________________________________
$ mlc_llm chat --help
usage: MLC LLM Chat CLI [-h] [--opt OPT] [--device DEVICE] [--overrides OVERRIDES] [--model-lib MODEL_LIB] model
positional arguments:
model A path to ``mlc-chat-config.json``, or an MLC model directory that contains `mlc-chat-config.json`.
It can also be a link to a HF repository pointing to an MLC compiled model. (required)
options:
-h, --help show this help message and exit
--opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags, denoted as O0, O1, O2,
O3, where O0 means no optimization, O2 means majority of them, and O3 represents extreme
optimization that could potentially break the system. Meanwhile, optimization flags could be
explicitly specified via details knobs, e.g. --opt="cublas_gemm=1;cudagraph=0". (default: "O2")
--device DEVICE The device used to deploy the model such as "cuda" or "cuda:0". Will detect from local available
GPUs if not specified. (default: "auto")
--overrides OVERRIDES
Chat configuration override. Configurations to override ChatConfig. Supports `conv_template`,
`context_window_size`, `prefill_chunk_size`, `sliding_window_size`, `attention_sink_size`,
`max_batch_size` and `tensor_parallel_shards`. Meanwhile, model chat could be explicitly specified
via details knobs, e.g. --overrides "context_window_size=1024;prefill_chunk_size=128". (default: "")
--model-lib MODEL_LIB
The full path to the model library file to use (e.g. a ``.so`` file). If unspecified, we will use
the provided ``model`` to search over possible paths. It the model lib is not found, it will be
compiled in a JIT manner. (default: "None")
------------------------------------------------------------------------------------------------------------------------------------------
$ mlc_llm compile --help
usage: mlc_llm compile [-h]
[--quantization {q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}]
[--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}]
[--device DEVICE] [--host HOST] [--opt OPT] [--system-lib-prefix SYSTEM_LIB_PREFIX] --output OUTPUT
[--overrides OVERRIDES] [--debug-dump DEBUG_DUMP]
model
positional arguments:
model A path to ``mlc-chat-config.json``, or an MLC model directory that contains `mlc-chat-config.json`.
It can also be a link to a HF repository pointing to an MLC compiled model. (required)
options:
-h, --help show this help message and exit
--quantization {q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}
The quantization mode we use to compile. If unprovided, will infer from `model`. (default: look up
mlc-chat-config.json, choices: q0f16, q0f32, q3f16_0, q3f16_1, q4f16_0, q4f16_1, q4f32_1, q4f16_2,
q4f16_autoawq, q4f16_ft, e5m2_e5m2_f16, e4m3_e4m3_f16)
--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}
Model architecture such as "llama". If not set, it is inferred from `mlc-chat-config.json`.
(default: "auto")
--device DEVICE The GPU device to compile the model to. If not set, it is inferred from GPUs available locally.
(default: "auto")
--host HOST The host LLVM triple to compile the model to. If not set, it is inferred from the local CPU and OS.
Examples of the LLVM triple: 1) iPhones: arm64-apple-ios; 2) ARM64 Android phones: aarch64-linux-
android; 3) WebAssembly: wasm32-unknown-unknown-wasm; 4) Windows: x86_64-pc-windows-msvc; 5) ARM
macOS: arm64-apple-darwin. (default: "auto")
--opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags, denoted as O0, O1, O2,
O3, where O0 means no optimization, O2 means majority of them, and O3 represents extreme
optimization that could potentially break the system. Meanwhile, optimization flags could be
explicitly specified via details knobs, e.g. --opt="cublas_gemm=1;cudagraph=0". (default: "O2")
--system-lib-prefix SYSTEM_LIB_PREFIX
Adding a prefix to all symbols exported. Similar to "objcopy --prefix-symbols". This is useful when
compiling multiple models into a single library to avoid symbol conflicts. Different from objcopy,
this takes no effect for shared library. (default: "auto")
--output OUTPUT, -o OUTPUT
The path to the output file. The suffix determines if the output file is a shared library or
objects. Available suffixes: 1) Linux: .so (shared), .tar (objects); 2) macOS: .dylib (shared), .tar
(objects); 3) Windows: .dll (shared), .tar (objects); 4) Android, iOS: .tar (objects); 5) Web: .wasm
(web assembly). (required)
--overrides OVERRIDES
Model configuration override. Configurations to override `mlc-chat-config.json`. Supports
`context_window_size`, `prefill_chunk_size`, `sliding_window_size`, `attention_sink_size`,
`max_batch_size` and `tensor_parallel_shards`. Meanwhile, model config could be explicitly specified
via details knobs, e.g. --overrides "context_window_size=1024;prefill_chunk_size=128". (default: "")
--debug-dump DEBUG_DUMP
Specifies the directory where the compiler will store its IRs for debugging purposes during various
phases of compilation. By default, this is set to `None`, indicating that debug dumping is disabled.
(default: None)
____________________________________________________________________________________________________________________________________________
$ mlc_llm convert_weight --help
usage: MLC AutoLLM Quantization Framework [-h] --quantization
{q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}
[--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}]
[--device DEVICE] [--source SOURCE]
[--source-format {auto,huggingface-torch,huggingface-safetensor,awq}] --output
OUTPUT
config
positional arguments:
config 1) Path to a HuggingFace model directory that contains a `config.json` or 2) Path to `config.json`
in HuggingFace format, or 3) The name of a pre-defined model architecture. A `config.json` file in
HuggingFace format defines the model architecture, including the vocabulary size, the number of
layers, the hidden size, number of attention heads, etc. Example:
https://huggingface.co./codellama/CodeLlama-7b-hf/blob/main/config.json. A HuggingFace directory
often contains a `config.json` which defines the model architecture, the non-quantized model weights
in PyTorch or SafeTensor format, tokenizer configurations, as well as an optional
`generation_config.json` provides additional default configuration for text generation. Example:
https://huggingface.co./codellama/CodeLlama-7b-hf/tree/main. (required)
options:
-h, --help show this help message and exit
--quantization {q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}
The quantization mode we use to compile. If unprovided, will infer from `model`. (required, choices:
q0f16, q0f32, q3f16_0, q3f16_1, q4f16_0, q4f16_1, q4f32_1, q4f16_2, q4f16_autoawq, q4f16_ft,
e5m2_e5m2_f16, e4m3_e4m3_f16)
--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}
Model architecture such as "llama". If not set, it is inferred from `mlc-chat-config.json`.
(default: "auto")
--device DEVICE The device used to do quantization such as "cuda" or "cuda:0". Will detect from local available GPUs
if not specified. (default: "auto")
--source SOURCE The path to original model weight, infer from `config` if missing. (default: "auto")
--source-format {auto,huggingface-torch,huggingface-safetensor,awq}
The format of source model weight, infer from `config` if missing. (default: "auto", choices: auto,
huggingface-torch, huggingface-safetensor, awq")
--output OUTPUT, -o OUTPUT
The output directory to save the quantized model weight. Will create `params_shard_*.bin` and
`ndarray-cache.json` in this directory. (required)
--------------------------------------------------------------------------------------------------------------------------------
$mlc_llm serve --help
usage: MLC LLM Serve CLI [-h] [--device DEVICE] [--model-lib MODEL_LIB] [--mode {local,interactive,server}]
[--additional-models [ADDITIONAL_MODELS ...]] [--max-batch-size MAX_BATCH_SIZE]
[--max-total-seq-length MAX_TOTAL_SEQ_LENGTH] [--prefill-chunk-size PREFILL_CHUNK_SIZE]
[--max-history-size MAX_HISTORY_SIZE] [--gpu-memory-utilization GPU_MEMORY_UTILIZATION]
[--speculative-mode {disable,small_draft,eagle,medusa}] [--spec-draft-length SPEC_DRAFT_LENGTH]
[--enable-tracing] [--host HOST] [--port PORT] [--allow-credentials]
[--allow-origins ALLOW_ORIGINS] [--allow-methods ALLOW_METHODS] [--allow-headers ALLOW_HEADERS]
model
positional arguments:
model A path to ``mlc-chat-config.json``, or an MLC model directory that contains `mlc-chat-config.json`.
It can also be a link to a HF repository pointing to an MLC compiled model. (required)
options:
-h, --help show this help message and exit
--device DEVICE The device used to deploy the model such as "cuda" or "cuda:0". Will detect from local available
GPUs if not specified. (default: "auto")
--model-lib MODEL_LIB
The full path to the model library file to use (e.g. a ``.so`` file). If unspecified, we will use
the provided ``model`` to search over possible paths. It the model lib is not found, it will be
compiled in a JIT manner. (default: "None")
--mode {local,interactive,server}
The engine mode in MLC LLM. We provide three preset modes: "local", "interactive" and "server". The
default mode is "local". The choice of mode decides the values of "--max-batch-size", "--max-total-
seq-length" and "--prefill-chunk-size" when they are not explicitly specified. 1. Mode "local"
refers to the local server deployment which has low request concurrency. So the max batch size will
be set to 4, and max total sequence length and prefill chunk size are set to the context window size
(or sliding window size) of the model. 2. Mode "interactive" refers to the interactive use of
server, which has at most 1 concurrent request. So the max batch size will be set to 1, and max
total sequence length and prefill chunk size are set to the context window size (or sliding window
size) of the model. 3. Mode "server" refers to the large server use case which may handle many
concurrent request and want to use GPU memory as much as possible. In this mode, we will
automatically infer the largest possible max batch size and max total sequence length. You can
manually specify arguments "--max-batch-size", "--max-total-seq-length" and "--prefill-chunk-size"
to override the automatic inferred values. (default: "local")
--additional-models [ADDITIONAL_MODELS ...]
The model paths and (optional) model library paths of additional models (other than the main model).
When engine is enabled with speculative decoding, additional models are needed. The way of
specifying additional models is: "--additional-models model_path_1 model_path_2 ..." or "--
additional-models model_path_1:model_lib_1 model_path_2 ...". When the model lib of a model is not
given, JIT model compilation will be activated to compile the model automatically.
--max-batch-size MAX_BATCH_SIZE
The maximum allowed batch size set for the KV cache to concurrently support.
--max-total-seq-length MAX_TOTAL_SEQ_LENGTH
The KV cache total token capacity, i.e., the maximum total number of tokens that the KV cache
support. This decides the GPU memory size that the KV cache consumes. If not specified, system will
automatically estimate the maximum capacity based on the vRAM size on GPU.
--prefill-chunk-size PREFILL_CHUNK_SIZE
The maximum number of tokens the model passes for prefill each time. It should not exceed the
prefill chunk size in model config. If not specified, this defaults to the prefill chunk size in
model config.
--max-history-size MAX_HISTORY_SIZE
The maximum history length for rolling back the RNN state. If unspecified, the default value is 1.
KV cache does not need this.
--gpu-memory-utilization GPU_MEMORY_UTILIZATION
A number in (0, 1) denoting the fraction of GPU memory used by the server in total. It is used to
infer to maximum possible KV cache capacity. When it is unspecified, it defaults to 0.85. Under mode
"local" or "interactive", the actual memory usage may be significantly smaller than this number.
Under mode "server", the actual memory usage may be slightly larger than this number.
--speculative-mode {disable,small_draft,eagle,medusa}
The speculative decoding mode. Right now three options are supported: - "disable", where speculative
decoding is not enabled, - "small_draft", denoting the normal speculative decoding (small draft)
style, - "eagle", denoting the eagle-style speculative decoding. The default mode is "disable".
(default: "disable")
--spec-draft-length SPEC_DRAFT_LENGTH
The number of draft tokens to generate in speculative proposal. The default values is 4.
--enable-tracing Enable Chrome Tracing for the server. After enabling, you can send POST request to the
"debug/dump_event_trace" entrypoint to get the Chrome Trace. For example, "curl -X POST
http://127.0.0.1:8000/debug/dump_event_trace -H "Content-Type: application/json" -d '{"model":
"dist/llama"}'"
--host HOST host name (default: "127.0.0.1")
--port PORT port (default: "8000")
--allow-credentials allow credentials
--allow-origins ALLOW_ORIGINS
allowed origins (default: "['*']")
--allow-methods ALLOW_METHODS
allowed methods (default: "['*']")
--allow-headers ALLOW_HEADERS
allowed headers (default: "['*']")
_________________________________________________________________________________________________________________________________________
$ mlc_llm gen_config --help
usage: MLC LLM Configuration Generator [-h] --quantization
{q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}
[--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}]
--conv-template
{llama-3,custom,open_hermes_mistral,vicuna_v1.1,gorilla,gorilla-openfunctions-v2,llava,gpt2,minigpt,stablecode_completion,conv_one_shot,llama-2,stablelm-3b,guanaco,LM,rwkv_world,gpt_bigcode,codellama_instruct,phi-2,phi-3,wizardlm_7b,stablelm-2,mistral_default,redpajama_chat,oasst,stablelm,llama_default,moss,gemma_instruction,neural_hermes_mistral,rwkv,stablecode_instruct,codellama_completion,wizard_coder_or_math,chatml,orion,glm,dolly}
[--context-window-size CONTEXT_WINDOW_SIZE]
[--sliding-window-size SLIDING_WINDOW_SIZE] [--prefill-chunk-size PREFILL_CHUNK_SIZE]
[--attention-sink-size ATTENTION_SINK_SIZE]
[--tensor-parallel-shards TENSOR_PARALLEL_SHARDS] [--max-batch-size MAX_BATCH_SIZE]
--output OUTPUT
config
positional arguments:
config 1) Path to a HuggingFace model directory that contains a `config.json` or 2) Path to `config.json`
in HuggingFace format, or 3) The name of a pre-defined model architecture. A `config.json` file in
HuggingFace format defines the model architecture, including the vocabulary size, the number of
layers, the hidden size, number of attention heads, etc. Example:
https://huggingface.co./codellama/CodeLlama-7b-hf/blob/main/config.json. A HuggingFace directory
often contains a `config.json` which defines the model architecture, the non-quantized model weights
in PyTorch or SafeTensor format, tokenizer configurations, as well as an optional
`generation_config.json` provides additional default configuration for text generation. Example:
https://huggingface.co./codellama/CodeLlama-7b-hf/tree/main. (required)
options:
-h, --help show this help message and exit
--quantization {q0f16,q0f32,q3f16_0,q3f16_1,q4f16_0,q4f16_1,q4f32_1,q4f16_2,q4f16_autoawq,q4f16_ft,e5m2_e5m2_f16,e4m3_e4m3_f16}
The quantization mode we use to compile. If unprovided, will infer from `model`. (required, choices:
q0f16, q0f32, q3f16_0, q3f16_1, q4f16_0, q4f16_1, q4f32_1, q4f16_2, q4f16_autoawq, q4f16_ft,
e5m2_e5m2_f16, e4m3_e4m3_f16)
--model-type {auto,llama,mistral,gemma,gpt2,mixtral,gpt_neox,gpt_bigcode,phi-msft,phi,phi3,qwen,qwen2,stablelm,baichuan,internlm,rwkv5,orion,llava,rwkv6,chatglm,eagle,bert,medusa}
Model architecture such as "llama". If not set, it is inferred from `mlc-chat-config.json`.
(default: "auto", choices: auto, llama, mistral, gemma, gpt2, mixtral, gpt_neox, gpt_bigcode, phi-
msft, phi, phi3, qwen, qwen2, stablelm, baichuan, internlm, rwkv5, orion, llava, rwkv6, chatglm,
eagle, bert, medusa)
--conv-template {llama-3,custom,open_hermes_mistral,vicuna_v1.1,gorilla,gorilla-openfunctions-v2,llava,gpt2,minigpt,stablecode_completion,conv_one_shot,llama-2,stablelm-3b,guanaco,LM,rwkv_world,gpt_bigcode,codellama_instruct,phi-2,phi-3,wizardlm_7b,stablelm-2,mistral_default,redpajama_chat,oasst,stablelm,llama_default,moss,gemma_instruction,neural_hermes_mistral,rwkv,stablecode_instruct,codellama_completion,wizard_coder_or_math,chatml,orion,glm,dolly}
Conversation template. It depends on how the model is tuned. Use "LM" for vanilla base model
(required, choices: llama-3, custom, open_hermes_mistral, vicuna_v1.1, gorilla, gorilla-
openfunctions-v2, llava, gpt2, minigpt, stablecode_completion, conv_one_shot, llama-2, stablelm-3b,
guanaco, LM, rwkv_world, gpt_bigcode, codellama_instruct, phi-2, phi-3, wizardlm_7b, stablelm-2,
mistral_default, redpajama_chat, oasst, stablelm, llama_default, moss, gemma_instruction,
neural_hermes_mistral, rwkv, stablecode_instruct, codellama_completion, wizard_coder_or_math,
chatml, orion, glm, dolly)
--context-window-size CONTEXT_WINDOW_SIZE
Option to provide the maximum sequence length supported by the model. This is usually explicitly
shown as context length or context window in the model card. If this option is not set explicitly,
by default, it will be determined by `context_window_size` or `max_position_embeddings` in
`config.json`, and the latter is usually inaccurate for some models. (default: "None")
--sliding-window-size SLIDING_WINDOW_SIZE
(Experimental) The sliding window size in sliding window attention (SWA). This optional field
overrides the `sliding_window_size` in config.json for those models that use SWA. Currently only
useful when compiling Mistral. This flag subjects to future refactoring. (default: "None")
--prefill-chunk-size PREFILL_CHUNK_SIZE
(Experimental) The chunk size during prefilling. By default, the chunk size is the same as sliding
window or max sequence length. This flag subjects to future refactoring. (default: "None")
--attention-sink-size ATTENTION_SINK_SIZE
(Experimental) The number of stored sinks. Only supported on Mistral yet. By default, the number of
sinks is 4. This flag subjects to future refactoring. (default: "None")
--tensor-parallel-shards TENSOR_PARALLEL_SHARDS
Number of shards to split the model into in tensor parallelism multi-gpu inference. (default:
"None")
--max-batch-size MAX_BATCH_SIZE
The maximum allowed batch size set for the KV cache to concurrently support. (default: "80")
--output OUTPUT, -o OUTPUT
The output directory for generated configurations, including `mlc-chat-config.json` and tokenizer
configuration. (required)
________________________________________________________________________________________________________________________________________
$ mlc_llm bench --help
usage: MLC LLM Chat CLI [-h] [--prompt PROMPT] [--opt OPT] [--device DEVICE] [--overrides OVERRIDES]
[--generate-length GENERATE_LENGTH] [--model-lib MODEL_LIB]
model
positional arguments:
model A path to ``mlc-chat-config.json``, or an MLC model directory that contains `mlc-chat-config.json`.
It can also be a link to a HF repository pointing to an MLC compiled model. (required)
options:
-h, --help show this help message and exit
--prompt PROMPT The prompt of the text generation. (default: "What is the meaning of life?")
--opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags, denoted as O0, O1, O2,
O3, where O0 means no optimization, O2 means majority of them, and O3 represents extreme
optimization that could potentially break the system. Meanwhile, optimization flags could be
explicitly specified via details knobs, e.g. --opt="cublas_gemm=1;cudagraph=0". (default: "O2")
--device DEVICE The device used to deploy the model such as "cuda" or "cuda:0". Will detect from local available
GPUs if not specified. (default: "auto")
--overrides OVERRIDES
Chat configuration override. Configurations to override ChatConfig. Supports `conv_template`,
`context_window_size`, `prefill_chunk_size`, `sliding_window_size`, `attention_sink_size`,
`max_batch_size` and `tensor_parallel_shards`. Meanwhile, model chat could be explicitly specified
via details knobs, e.g. --overrides "context_window_size=1024;prefill_chunk_size=128". (default: "")
--generate-length GENERATE_LENGTH
The target length of the text generation. (default: "256")
--model-lib MODEL_LIB
The full path to the model library file to use (e.g. a ``.so`` file). If unspecified, we will use
the provided ``model`` to search over possible paths. It the model lib is not found, it will be
compiled in a JIT manner. (default: "None")
__________________________________________________________________________________________________________________________________________
$ mlc_llm package --help
usage: MLC LLM Package CLI [-h] [--package-config PACKAGE_CONFIG] [--mlc-llm-home MLC_LLM_HOME] [--output OUTPUT]
options:
-h, --help show this help message and exit
--package-config PACKAGE_CONFIG
The path to "mlc-package-config.json" which is used for package build. See "https://github.com/mlc-
ai/mlc-llm/blob/main/ios/MLCChat/mlc-package-config.json" as an example. (default: "mlc-package-
config.json")
--mlc-llm-home MLC_LLM_HOME
The source code path to MLC LLM. (default: the $MLC_LLM_HOME environment variable)
--output OUTPUT, -o OUTPUT
The path of output directory for the package build outputs. (default: "dist")
