Hugging Face's logo

qualcomm
/
Segment-Anything-Model

Image Segmentation
PyTorch
TF Lite
ONNX
foundation
android
Model card Files Community

Segment-Anything-Model: Optimized for Mobile Deployment

High-quality segmentation mask generation around any object in an image with simple input prompt

Transformer based encoder-decoder where prompts specify what to segment in an image thereby allowing segmentation without the need for additional training. The image encoder generates embeddings and the lightweight decoder operates on the embeddings for point and mask based image segmentation.

This model is an implementation of Segment-Anything-Model found here.

This repository provides scripts to run Segment-Anything-Model on Qualcomm® devices. More details on model performance across various devices, can be found here.

Model Details

  • Model Type: Semantic segmentation
  • Model Stats:
    • Model checkpoint: vit_l
    • Input resolution: 720p (720x1280)
    • Number of parameters (SAMDecoder): 5.11M
    • Model size (SAMDecoder): 19.6 MB
Model Device Chipset Target Runtime Inference Time (ms) Peak Memory Range (MB) Precision Primary Compute Unit Target Model
SAMDecoder Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 7.442 ms 0 - 33 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 7.305 ms 4 - 21 MB FP16 NPU Segment-Anything-Model.so
SAMDecoder Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 10.954 ms 0 - 61 MB FP16 NPU Segment-Anything-Model.onnx
SAMDecoder Samsung Galaxy S24 Snapdragon® 8 Gen 3 TFLITE 5.197 ms 0 - 39 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder Samsung Galaxy S24 Snapdragon® 8 Gen 3 QNN 5.138 ms 40 - 83 MB FP16 NPU Segment-Anything-Model.so
SAMDecoder Samsung Galaxy S24 Snapdragon® 8 Gen 3 ONNX 8.273 ms 6 - 58 MB FP16 NPU Segment-Anything-Model.onnx
SAMDecoder Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 4.189 ms 0 - 38 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 4.836 ms 4 - 45 MB FP16 NPU Use Export Script
SAMDecoder Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 8.097 ms 6 - 52 MB FP16 NPU Segment-Anything-Model.onnx
SAMDecoder QCS8550 (Proxy) QCS8550 Proxy TFLITE 7.44 ms 0 - 33 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder QCS8550 (Proxy) QCS8550 Proxy QNN 6.839 ms 4 - 7 MB FP16 NPU Use Export Script
SAMDecoder SA7255P ADP SA7255P TFLITE 53.012 ms 0 - 33 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder SA7255P ADP SA7255P QNN 49.841 ms 1 - 11 MB FP16 NPU Use Export Script
SAMDecoder SA8255 (Proxy) SA8255P Proxy TFLITE 7.45 ms 0 - 32 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder SA8255 (Proxy) SA8255P Proxy QNN 6.933 ms 4 - 6 MB FP16 NPU Use Export Script
SAMDecoder SA8295P ADP SA8295P TFLITE 9.944 ms 0 - 36 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder SA8295P ADP SA8295P QNN 8.969 ms 0 - 14 MB FP16 NPU Use Export Script
SAMDecoder SA8650 (Proxy) SA8650P Proxy TFLITE 7.451 ms 0 - 34 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder SA8650 (Proxy) SA8650P Proxy QNN 6.992 ms 4 - 6 MB FP16 NPU Use Export Script
SAMDecoder SA8775P ADP SA8775P TFLITE 10.463 ms 0 - 33 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder SA8775P ADP SA8775P QNN 9.711 ms 2 - 12 MB FP16 NPU Use Export Script
SAMDecoder QCS8450 (Proxy) QCS8450 Proxy TFLITE 8.499 ms 0 - 36 MB FP16 NPU Segment-Anything-Model.tflite
SAMDecoder QCS8450 (Proxy) QCS8450 Proxy QNN 8.282 ms 4 - 43 MB FP16 NPU Use Export Script
SAMDecoder Snapdragon X Elite CRD Snapdragon® X Elite QNN 7.345 ms 4 - 4 MB FP16 NPU Use Export Script
SAMDecoder Snapdragon X Elite CRD Snapdragon® X Elite ONNX 14.961 ms 11 - 11 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart1 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 208.707 ms 12 - 79 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 203.4 ms 12 - 104 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart1 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 167.127 ms 25 - 182 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart1 Samsung Galaxy S24 Snapdragon® 8 Gen 3 TFLITE 147.832 ms 11 - 664 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 Samsung Galaxy S24 Snapdragon® 8 Gen 3 QNN 144.807 ms 12 - 651 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart1 Samsung Galaxy S24 Snapdragon® 8 Gen 3 ONNX 122.784 ms 23 - 696 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart1 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 144.47 ms 10 - 662 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 143.144 ms 12 - 660 MB FP16 NPU Use Export Script
SAMEncoderPart1 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 98.548 ms 23 - 668 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart1 QCS8550 (Proxy) QCS8550 Proxy TFLITE 208.97 ms 12 - 71 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 QCS8550 (Proxy) QCS8550 Proxy QNN 176.897 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart1 SA7255P ADP SA7255P TFLITE 1172.71 ms 0 - 644 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 SA7255P ADP SA7255P QNN 1103.665 ms 5 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart1 SA8255 (Proxy) SA8255P Proxy TFLITE 206.562 ms 12 - 67 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 SA8255 (Proxy) SA8255P Proxy QNN 179.137 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart1 SA8295P ADP SA8295P TFLITE 242.758 ms 12 - 640 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 SA8295P ADP SA8295P QNN 207.245 ms 0 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart1 SA8650 (Proxy) SA8650P Proxy TFLITE 206.266 ms 12 - 73 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 SA8650 (Proxy) SA8650P Proxy QNN 175.273 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart1 SA8775P ADP SA8775P TFLITE 249.944 ms 12 - 656 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 SA8775P ADP SA8775P QNN 211.755 ms 1 - 11 MB FP16 NPU Use Export Script
SAMEncoderPart1 QCS8450 (Proxy) QCS8450 Proxy TFLITE 231.125 ms 12 - 993 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart1 QCS8450 (Proxy) QCS8450 Proxy QNN 225.612 ms 12 - 965 MB FP16 NPU Use Export Script
SAMEncoderPart1 Snapdragon X Elite CRD Snapdragon® X Elite QNN 170.905 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart1 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 181.653 ms 38 - 38 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart2 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 663.885 ms 12 - 110 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 848.959 ms 12 - 111 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart2 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 752.707 ms 12 - 199 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart2 Samsung Galaxy S24 Snapdragon® 8 Gen 3 TFLITE 538.304 ms 12 - 1133 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 474.493 ms 11 - 1140 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 581.145 ms 12 - 1112 MB FP16 NPU Use Export Script
SAMEncoderPart2 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 437.687 ms 36 - 1409 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart2 QCS8550 (Proxy) QCS8550 Proxy TFLITE 676.949 ms 12 - 107 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 QCS8550 (Proxy) QCS8550 Proxy QNN 741.824 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart2 SA7255P ADP SA7255P QNN 1879.956 ms 3 - 13 MB FP16 NPU Use Export Script
SAMEncoderPart2 SA8255 (Proxy) SA8255P Proxy TFLITE 673.933 ms 12 - 103 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 SA8255 (Proxy) SA8255P Proxy QNN 740.075 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart2 SA8295P ADP SA8295P TFLITE 707.086 ms 12 - 1174 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 SA8295P ADP SA8295P QNN 783.918 ms 0 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart2 SA8650 (Proxy) SA8650P Proxy TFLITE 658.949 ms 12 - 119 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 SA8650 (Proxy) SA8650P Proxy QNN 739.207 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart2 SA8775P ADP SA8775P TFLITE 702.232 ms 0 - 1144 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart2 Snapdragon X Elite CRD Snapdragon® X Elite QNN 641.303 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart2 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 744.868 ms 52 - 52 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart3 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 672.567 ms 12 - 110 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 843.337 ms 12 - 107 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart3 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 745.502 ms 24 - 207 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart3 Samsung Galaxy S24 Snapdragon® 8 Gen 3 TFLITE 534.698 ms 5 - 1125 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 471.028 ms 11 - 1141 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 575.953 ms 12 - 1111 MB FP16 NPU Use Export Script
SAMEncoderPart3 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 458.142 ms 36 - 1408 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart3 QCS8550 (Proxy) QCS8550 Proxy TFLITE 677.866 ms 12 - 111 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 QCS8550 (Proxy) QCS8550 Proxy QNN 733.613 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart3 SA7255P ADP SA7255P QNN 1878.822 ms 4 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart3 SA8255 (Proxy) SA8255P Proxy QNN 738.856 ms 13 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart3 SA8295P ADP SA8295P TFLITE 707.285 ms 12 - 1174 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 SA8295P ADP SA8295P QNN 783.748 ms 1 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart3 SA8650 (Proxy) SA8650P Proxy TFLITE 662.19 ms 12 - 106 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 SA8650 (Proxy) SA8650P Proxy QNN 731.685 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart3 SA8775P ADP SA8775P TFLITE 700.772 ms 0 - 1147 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart3 SA8775P ADP SA8775P QNN 741.409 ms 1 - 11 MB FP16 NPU Use Export Script
SAMEncoderPart3 Snapdragon X Elite CRD Snapdragon® X Elite QNN 635.362 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart3 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 745.734 ms 51 - 51 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart4 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 657.654 ms 12 - 109 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 856.098 ms 12 - 114 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart4 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 743.085 ms 19 - 212 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart4 Samsung Galaxy S24 Snapdragon® 8 Gen 3 ONNX 604.577 ms 24 - 1422 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart4 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 475.414 ms 11 - 1141 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 574.057 ms 12 - 1112 MB FP16 NPU Use Export Script
SAMEncoderPart4 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 462.476 ms 36 - 1408 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart4 QCS8550 (Proxy) QCS8550 Proxy TFLITE 656.285 ms 12 - 111 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 QCS8550 (Proxy) QCS8550 Proxy QNN 721.087 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart4 SA7255P ADP SA7255P QNN 1879.639 ms 4 - 13 MB FP16 NPU Use Export Script
SAMEncoderPart4 SA8255 (Proxy) SA8255P Proxy TFLITE 678.333 ms 12 - 103 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 SA8255 (Proxy) SA8255P Proxy QNN 733.002 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart4 SA8295P ADP SA8295P TFLITE 705.127 ms 12 - 1172 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 SA8295P ADP SA8295P QNN 781.449 ms 0 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart4 SA8650 (Proxy) SA8650P Proxy TFLITE 663.855 ms 12 - 110 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 SA8650 (Proxy) SA8650P Proxy QNN 730.597 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart4 SA8775P ADP SA8775P TFLITE 704.507 ms 0 - 1145 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart4 SA8775P ADP SA8775P QNN 740.223 ms 2 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart4 Snapdragon X Elite CRD Snapdragon® X Elite QNN 633.268 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart4 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 739.909 ms 51 - 51 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart5 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 670.811 ms 12 - 104 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 840.684 ms 12 - 118 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart5 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 745.91 ms 12 - 210 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart5 Samsung Galaxy S24 Snapdragon® 8 Gen 3 QNN 648.946 ms 12 - 1109 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart5 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 430.043 ms 11 - 1143 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 575.631 ms 12 - 1112 MB FP16 NPU Use Export Script
SAMEncoderPart5 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 433.917 ms 36 - 1404 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart5 QCS8550 (Proxy) QCS8550 Proxy TFLITE 648.794 ms 12 - 95 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 QCS8550 (Proxy) QCS8550 Proxy QNN 727.832 ms 12 - 16 MB FP16 NPU Use Export Script
SAMEncoderPart5 SA7255P ADP SA7255P QNN 1883.878 ms 12 - 21 MB FP16 NPU Use Export Script
SAMEncoderPart5 SA8255 (Proxy) SA8255P Proxy TFLITE 688.453 ms 12 - 112 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 SA8255 (Proxy) SA8255P Proxy QNN 745.534 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart5 SA8295P ADP SA8295P TFLITE 707.293 ms 12 - 1176 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 SA8295P ADP SA8295P QNN 784.759 ms 0 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart5 SA8650 (Proxy) SA8650P Proxy TFLITE 648.646 ms 12 - 118 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 SA8650 (Proxy) SA8650P Proxy QNN 735.432 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart5 SA8775P ADP SA8775P TFLITE 707.889 ms 0 - 1146 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart5 SA8775P ADP SA8775P QNN 740.576 ms 1 - 11 MB FP16 NPU Use Export Script
SAMEncoderPart5 Snapdragon X Elite CRD Snapdragon® X Elite QNN 644.302 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart5 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 734.599 ms 51 - 51 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart6 Samsung Galaxy S23 Snapdragon® 8 Gen 2 TFLITE 679.861 ms 4 - 94 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 Samsung Galaxy S23 Snapdragon® 8 Gen 2 QNN 844.027 ms 12 - 107 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart6 Samsung Galaxy S23 Snapdragon® 8 Gen 2 ONNX 744.683 ms 12 - 198 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart6 Samsung Galaxy S24 Snapdragon® 8 Gen 3 QNN 672.024 ms 12 - 1115 MB FP16 NPU Segment-Anything-Model.so
SAMEncoderPart6 Samsung Galaxy S24 Snapdragon® 8 Gen 3 ONNX 613.484 ms 20 - 1419 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart6 Snapdragon 8 Elite QRD Snapdragon® 8 Elite TFLITE 429.205 ms 12 - 1141 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 Snapdragon 8 Elite QRD Snapdragon® 8 Elite QNN 577.174 ms 12 - 1113 MB FP16 NPU Use Export Script
SAMEncoderPart6 Snapdragon 8 Elite QRD Snapdragon® 8 Elite ONNX 465.664 ms 36 - 1412 MB FP16 NPU Segment-Anything-Model.onnx
SAMEncoderPart6 QCS8550 (Proxy) QCS8550 Proxy TFLITE 647.378 ms 12 - 109 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 QCS8550 (Proxy) QCS8550 Proxy QNN 731.1 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart6 SA7255P ADP SA7255P QNN 1877.714 ms 2 - 10 MB FP16 NPU Use Export Script
SAMEncoderPart6 SA8255 (Proxy) SA8255P Proxy TFLITE 661.485 ms 12 - 113 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 SA8255 (Proxy) SA8255P Proxy QNN 738.693 ms 12 - 14 MB FP16 NPU Use Export Script
SAMEncoderPart6 SA8295P ADP SA8295P TFLITE 706.823 ms 12 - 1176 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 SA8295P ADP SA8295P QNN 782.241 ms 0 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart6 SA8650 (Proxy) SA8650P Proxy TFLITE 622.148 ms 14 - 113 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 SA8650 (Proxy) SA8650P Proxy QNN 729.56 ms 12 - 15 MB FP16 NPU Use Export Script
SAMEncoderPart6 SA8775P ADP SA8775P TFLITE 704.738 ms 0 - 1144 MB FP16 NPU Segment-Anything-Model.tflite
SAMEncoderPart6 SA8775P ADP SA8775P QNN 741.461 ms 1 - 11 MB FP16 NPU Use Export Script
SAMEncoderPart6 Snapdragon X Elite CRD Snapdragon® X Elite QNN 636.013 ms 12 - 12 MB FP16 NPU Use Export Script
SAMEncoderPart6 Snapdragon X Elite CRD Snapdragon® X Elite ONNX 723.315 ms 53 - 53 MB FP16 NPU Segment-Anything-Model.onnx

Installation

Install the package via pip:

pip install "qai-hub-models[sam]"

Configure Qualcomm® AI Hub to run this model on a cloud-hosted device

Sign-in to Qualcomm® AI Hub with your Qualcomm® ID. Once signed in navigate to Account -> Settings -> API Token.

With this API token, you can configure your client to run models on the cloud hosted devices.

qai-hub configure --api_token API_TOKEN

Navigate to docs for more information.

Demo off target

The package contains a simple end-to-end demo that downloads pre-trained weights and runs this model on a sample input.

python -m qai_hub_models.models.sam.demo

The above demo runs a reference implementation of pre-processing, model inference, and post processing.

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.sam.demo

Run model on a cloud-hosted device

In addition to the demo, you can also run the model on a cloud-hosted Qualcomm® device. This script does the following:

  • Performance check on-device on a cloud-hosted device
  • Downloads compiled assets that can be deployed on-device for Android.
  • Accuracy check between PyTorch and on-device outputs.
python -m qai_hub_models.models.sam.export

Profiling Results
------------------------------------------------------------
SAMDecoder
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 7.4                    
Estimated peak memory usage (MB): [0, 33]                
Total # Ops                     : 845                    
Compute Unit(s)                 : NPU (845 ops)          

------------------------------------------------------------
SAMEncoderPart1
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 208.7                  
Estimated peak memory usage (MB): [12, 79]               
Total # Ops                     : 584                    
Compute Unit(s)                 : NPU (584 ops)          

------------------------------------------------------------
SAMEncoderPart2
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 663.9                  
Estimated peak memory usage (MB): [12, 110]              
Total # Ops                     : 572                    
Compute Unit(s)                 : NPU (572 ops)          

------------------------------------------------------------
SAMEncoderPart3
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 672.6                  
Estimated peak memory usage (MB): [12, 110]              
Total # Ops                     : 572                    
Compute Unit(s)                 : NPU (572 ops)          

------------------------------------------------------------
SAMEncoderPart4
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 657.7                  
Estimated peak memory usage (MB): [12, 109]              
Total # Ops                     : 572                    
Compute Unit(s)                 : NPU (572 ops)          

------------------------------------------------------------
SAMEncoderPart5
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 670.8                  
Estimated peak memory usage (MB): [12, 104]              
Total # Ops                     : 572                    
Compute Unit(s)                 : NPU (572 ops)          

------------------------------------------------------------
SAMEncoderPart6
Device                          : Samsung Galaxy S23 (13)
Runtime                         : TFLITE                 
Estimated inference time (ms)   : 679.9                  
Estimated peak memory usage (MB): [4, 94]                
Total # Ops                     : 572                    
Compute Unit(s)                 : NPU (572 ops)

How does this work?

This export script leverages Qualcomm® AI Hub to optimize, validate, and deploy this model on-device. Lets go through each step below in detail:

Step 1: Compile model for on-device deployment

To compile a PyTorch model for on-device deployment, we first trace the model in memory using the jit.trace and then call the submit_compile_job API.

import torch

import qai_hub as hub
from qai_hub_models.models.sam import Model

# Load the model
model = Model.from_pretrained()
decoder_model = model.decoder
encoder_splits[0]_model = model.encoder_splits[0]
encoder_splits[1]_model = model.encoder_splits[1]
encoder_splits[2]_model = model.encoder_splits[2]
encoder_splits[3]_model = model.encoder_splits[3]
encoder_splits[4]_model = model.encoder_splits[4]
encoder_splits[5]_model = model.encoder_splits[5]

# Device
device = hub.Device("Samsung Galaxy S23")

# Trace model
decoder_input_shape = decoder_model.get_input_spec()
decoder_sample_inputs = decoder_model.sample_inputs()

traced_decoder_model = torch.jit.trace(decoder_model, [torch.tensor(data[0]) for _, data in decoder_sample_inputs.items()])

# Compile model on a specific device
decoder_compile_job = hub.submit_compile_job(
    model=traced_decoder_model ,
    device=device,
    input_specs=decoder_model.get_input_spec(),
)

# Get target model to run on-device
decoder_target_model = decoder_compile_job.get_target_model()
# Trace model
encoder_splits[0]_input_shape = encoder_splits[0]_model.get_input_spec()
encoder_splits[0]_sample_inputs = encoder_splits[0]_model.sample_inputs()

traced_encoder_splits[0]_model = torch.jit.trace(encoder_splits[0]_model, [torch.tensor(data[0]) for _, data in encoder_splits[0]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[0]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[0]_model ,
    device=device,
    input_specs=encoder_splits[0]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[0]_target_model = encoder_splits[0]_compile_job.get_target_model()
# Trace model
encoder_splits[1]_input_shape = encoder_splits[1]_model.get_input_spec()
encoder_splits[1]_sample_inputs = encoder_splits[1]_model.sample_inputs()

traced_encoder_splits[1]_model = torch.jit.trace(encoder_splits[1]_model, [torch.tensor(data[0]) for _, data in encoder_splits[1]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[1]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[1]_model ,
    device=device,
    input_specs=encoder_splits[1]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[1]_target_model = encoder_splits[1]_compile_job.get_target_model()
# Trace model
encoder_splits[2]_input_shape = encoder_splits[2]_model.get_input_spec()
encoder_splits[2]_sample_inputs = encoder_splits[2]_model.sample_inputs()

traced_encoder_splits[2]_model = torch.jit.trace(encoder_splits[2]_model, [torch.tensor(data[0]) for _, data in encoder_splits[2]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[2]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[2]_model ,
    device=device,
    input_specs=encoder_splits[2]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[2]_target_model = encoder_splits[2]_compile_job.get_target_model()
# Trace model
encoder_splits[3]_input_shape = encoder_splits[3]_model.get_input_spec()
encoder_splits[3]_sample_inputs = encoder_splits[3]_model.sample_inputs()

traced_encoder_splits[3]_model = torch.jit.trace(encoder_splits[3]_model, [torch.tensor(data[0]) for _, data in encoder_splits[3]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[3]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[3]_model ,
    device=device,
    input_specs=encoder_splits[3]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[3]_target_model = encoder_splits[3]_compile_job.get_target_model()
# Trace model
encoder_splits[4]_input_shape = encoder_splits[4]_model.get_input_spec()
encoder_splits[4]_sample_inputs = encoder_splits[4]_model.sample_inputs()

traced_encoder_splits[4]_model = torch.jit.trace(encoder_splits[4]_model, [torch.tensor(data[0]) for _, data in encoder_splits[4]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[4]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[4]_model ,
    device=device,
    input_specs=encoder_splits[4]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[4]_target_model = encoder_splits[4]_compile_job.get_target_model()
# Trace model
encoder_splits[5]_input_shape = encoder_splits[5]_model.get_input_spec()
encoder_splits[5]_sample_inputs = encoder_splits[5]_model.sample_inputs()

traced_encoder_splits[5]_model = torch.jit.trace(encoder_splits[5]_model, [torch.tensor(data[0]) for _, data in encoder_splits[5]_sample_inputs.items()])

# Compile model on a specific device
encoder_splits[5]_compile_job = hub.submit_compile_job(
    model=traced_encoder_splits[5]_model ,
    device=device,
    input_specs=encoder_splits[5]_model.get_input_spec(),
)

# Get target model to run on-device
encoder_splits[5]_target_model = encoder_splits[5]_compile_job.get_target_model()

Step 2: Performance profiling on cloud-hosted device

After compiling models from step 1. Models can be profiled model on-device using the target_model. Note that this scripts runs the model on a device automatically provisioned in the cloud. Once the job is submitted, you can navigate to a provided job URL to view a variety of on-device performance metrics.

decoder_profile_job = hub.submit_profile_job(
    model=decoder_target_model,
    device=device,
)
encoder_splits[0]_profile_job = hub.submit_profile_job(
    model=encoder_splits[0]_target_model,
    device=device,
)
encoder_splits[1]_profile_job = hub.submit_profile_job(
    model=encoder_splits[1]_target_model,
    device=device,
)
encoder_splits[2]_profile_job = hub.submit_profile_job(
    model=encoder_splits[2]_target_model,
    device=device,
)
encoder_splits[3]_profile_job = hub.submit_profile_job(
    model=encoder_splits[3]_target_model,
    device=device,
)
encoder_splits[4]_profile_job = hub.submit_profile_job(
    model=encoder_splits[4]_target_model,
    device=device,
)
encoder_splits[5]_profile_job = hub.submit_profile_job(
    model=encoder_splits[5]_target_model,
    device=device,
)

Step 3: Verify on-device accuracy

To verify the accuracy of the model on-device, you can run on-device inference on sample input data on the same cloud hosted device.

decoder_input_data = decoder_model.sample_inputs()
decoder_inference_job = hub.submit_inference_job(
    model=decoder_target_model,
    device=device,
    inputs=decoder_input_data,
)
decoder_inference_job.download_output_data()
encoder_splits[0]_input_data = encoder_splits[0]_model.sample_inputs()
encoder_splits[0]_inference_job = hub.submit_inference_job(
    model=encoder_splits[0]_target_model,
    device=device,
    inputs=encoder_splits[0]_input_data,
)
encoder_splits[0]_inference_job.download_output_data()
encoder_splits[1]_input_data = encoder_splits[1]_model.sample_inputs()
encoder_splits[1]_inference_job = hub.submit_inference_job(
    model=encoder_splits[1]_target_model,
    device=device,
    inputs=encoder_splits[1]_input_data,
)
encoder_splits[1]_inference_job.download_output_data()
encoder_splits[2]_input_data = encoder_splits[2]_model.sample_inputs()
encoder_splits[2]_inference_job = hub.submit_inference_job(
    model=encoder_splits[2]_target_model,
    device=device,
    inputs=encoder_splits[2]_input_data,
)
encoder_splits[2]_inference_job.download_output_data()
encoder_splits[3]_input_data = encoder_splits[3]_model.sample_inputs()
encoder_splits[3]_inference_job = hub.submit_inference_job(
    model=encoder_splits[3]_target_model,
    device=device,
    inputs=encoder_splits[3]_input_data,
)
encoder_splits[3]_inference_job.download_output_data()
encoder_splits[4]_input_data = encoder_splits[4]_model.sample_inputs()
encoder_splits[4]_inference_job = hub.submit_inference_job(
    model=encoder_splits[4]_target_model,
    device=device,
    inputs=encoder_splits[4]_input_data,
)
encoder_splits[4]_inference_job.download_output_data()
encoder_splits[5]_input_data = encoder_splits[5]_model.sample_inputs()
encoder_splits[5]_inference_job = hub.submit_inference_job(
    model=encoder_splits[5]_target_model,
    device=device,
    inputs=encoder_splits[5]_input_data,
)
encoder_splits[5]_inference_job.download_output_data()

With the output of the model, you can compute like PSNR, relative errors or spot check the output with expected output.

Note: This on-device profiling and inference requires access to Qualcomm® AI Hub. Sign up for access.

Run demo on a cloud-hosted device

You can also run the demo on-device.

python -m qai_hub_models.models.sam.demo --on-device

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.sam.demo -- --on-device

Deploying compiled model to Android

The models can be deployed using multiple runtimes:

  • TensorFlow Lite (.tflite export): This tutorial provides a guide to deploy the .tflite model in an Android application.

  • QNN (.so export ): This sample app provides instructions on how to use the .so shared library in an Android application.

View on Qualcomm® AI Hub

Get more details on Segment-Anything-Model's performance across various devices here. Explore all available models on Qualcomm® AI Hub

License

  • The license for the original implementation of Segment-Anything-Model can be found here.
  • The license for the compiled assets for on-device deployment can be found here

References

Community

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Examples
Image Segmentation
Inference API (serverless) does not yet support pytorch models for this pipeline type.