Installation
CUDA
bitsandbytes is only supported on CUDA GPUs for CUDA versions 11.0 - 12.5. However, there’s a multi-backend effort under way which is currently in alpha release, check the respective section below in case you’re interested to help us with early feedback.
The latest version of bitsandbytes builds on:
| OS | CUDA | Compiler |
|---|---|---|
| Linux | 11.7 - 12.3 | GCC 11.4 |
| 12.4+ | GCC 13.2 | |
| Windows | 11.7 - 12.4 | MSVC 19.38+ (VS2022 17.8.0+) |
MacOS support is still a work in progress! Subscribe to this issue to get notified about discussions and to track the integration progress.
For Linux systems, make sure your hardware meets the following requirements to use bitsandbytes features.
| Feature | Hardware requirement |
|---|---|
| LLM.int8() | NVIDIA Turing (RTX 20 series, T4) or Ampere (RTX 30 series, A4-A100) GPUs |
| 8-bit optimizers/quantization | NVIDIA Kepler (GTX 780 or newer) |
bitsandbytes >= 0.39.1 no longer includes Kepler binaries in pip installations. This requires manual compilation, and you should follow the general steps and use cuda11x_nomatmul_kepler for Kepler-targeted compilation.
To install from PyPI.
pip install bitsandbytes
Compile from source
For Linux and Windows systems, you can compile bitsandbytes from source. Installing from source allows for more build options with different CMake configurations.
To compile from source, you need CMake >= 3.22.1 and Python >= 3.8 installed. Make sure you have a compiler installed to compile C++ (gcc, make, headers, etc.). For example, to install a compiler and CMake on Ubuntu:
apt-get install -y build-essential cmake
You should also install CUDA Toolkit by following the NVIDIA CUDA Installation Guide for Linux guide from NVIDIA. The current expected CUDA Toolkit version is 11.1+ and it is recommended to install GCC >= 7.3 and required to have at least GCC >= 6.
Refer to the following table if you’re using another CUDA Toolkit version.
| CUDA Toolkit | GCC |
|---|---|
| >= 11.4.1 | >= 11 |
| >= 12.0 | >= 12 |
| >= 12.4 | >= 13 |
Now to install the bitsandbytes package from source, run the following commands:
git clone https://github.com/TimDettmers/bitsandbytes.git && cd bitsandbytes/
pip install -r requirements-dev.txt
cmake -DCOMPUTE_BACKEND=cuda -S .
make
pip install -e . # `-e` for "editable" install, when developing BNB (otherwise leave that out)If you have multiple versions of CUDA installed or installed it in a non-standard location, please refer to CMake CUDA documentation for how to configure the CUDA compiler.
PyTorch CUDA versions
Some bitsandbytes features may need a newer CUDA version than the one currently supported by PyTorch binaries from Conda and pip. In this case, you should follow these instructions to load a precompiled bitsandbytes binary.
- Determine the path of the CUDA version you want to use. Common paths include:
/usr/local/cuda/usr/local/cuda-XX.XwhereXX.Xis the CUDA version number
Then locally install the CUDA version you need with this script from bitsandbytes:
wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/install_cuda.sh
# Syntax cuda_install CUDA_VERSION INSTALL_PREFIX EXPORT_TO_BASH
# CUDA_VERSION in {110, 111, 112, 113, 114, 115, 116, 117, 118, 120, 121, 122, 123, 124, 125}
# EXPORT_TO_BASH in {0, 1} with 0=False and 1=True
# For example, the following installs CUDA 11.7 to ~/local/cuda-11.7 and exports the path to your .bashrc
bash install_cuda.sh 117 ~/local 1- Set the environment variables
BNB_CUDA_VERSIONandLD_LIBRARY_PATHby manually overriding the CUDA version installed by PyTorch.
It is recommended to add the following lines to the .bashrc file to make them permanent.
export BNB_CUDA_VERSION=<VERSION>
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<PATH>For example, to use a local install path:
export BNB_CUDA_VERSION=117
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/YOUR_USERNAME/local/cuda-11.7- Now when you launch bitsandbytes with these environment variables, the PyTorch CUDA version is overridden by the new CUDA version (in this example, version 11.7) and a different bitsandbytes library is loaded.
Multi-backend
This functionality is currently in preview and therefore not yet production-ready!
Please follow these steps to install bitsandbytes with device-specific backend support other than CUDA:
Pip install the pre-built wheel (recommended for most)
WIP (will be added in the coming days)
Compilation
AMD GPU
bitsandbytes is fully supported from ROCm 6.1 onwards (currently in alpha release).
If you would like to install ROCm and PyTorch on bare metal, skip Docker steps and refer to our official guides at ROCm installation overview and Installing PyTorch for ROCm (Step 3 of wheels build for quick installation). Please make sure to get PyTorch wheel for the installed ROCm version.
# Create a docker container with latest ROCm image, which includes ROCm libraries
docker pull rocm/dev-ubuntu-22.04:6.1.2-complete
docker run -it --device=/dev/kfd --device=/dev/dri --group-add video rocm/dev-ubuntu-22.04:6.1.2-complete
apt-get update && apt-get install -y git && cd home
# Install pytorch compatible with above ROCm version
pip install torch --index-url https://download.pytorch.org/whl/rocm6.1/
# Install bitsandbytes from PyPI
# (This is supported on Ubuntu 22.04, Python 3.10, ROCm 6.1.0/6.1.1/6.1.2 and gpu arch - gfx90a, gfx942, gfx1100
# Please install from source if your configuration doesn't match with these)
pip install bitsandbytes
# Install bitsandbytes from source
# Clone bitsandbytes repo, ROCm backend is currently enabled on multi-backend-refactor branch
git clone --depth 1 -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
# Install dependencies
pip install -r requirements-dev.txt
# Compile & install
apt-get install -y build-essential cmake # install build tools dependencies, unless present
cmake -DCOMPUTE_BACKEND=hip -S . # Use -DBNB_ROCM_ARCH="gfx90a;gfx942" to target specific gpu arch
make
pip install -e . # `-e` for "editable" install, when developing BNB (otherwise leave that out)