File size: 7,029 Bytes
fab5e76 e6da6d9 89075fd fab5e76 72c90ad fab5e76 1c3c008 fab5e76 72c90ad fab5e76 1c3c008 72c90ad fab5e76 72c90ad fab5e76 1c3c008 72c90ad fab5e76 b7a988e fab5e76 19838f3 fab5e76 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 |
---
license: apache-2.0
datasets:
- NeelNanda/pile-10k
base_model:
- allenai/Molmo-7B-D-0924
---
## Model Details
This model is an int4 model with group_size 128 and symmetric quantization of [allenai/Molmo-7B-D-0924](https://huggingface.co./allenai/Molmo-7B-D-0924) generated by [intel/auto-round](https://github.com/intel/auto-round). Load the model with revision="e64d453" to use AutoGPTQ format.
## How To Use
### INT4 Inference
```python
from auto_round import AutoRoundConfig ## must import for auto-round format
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests
quantized_model_path = "OPEA/Molmo-7B-D-0924-int4-sym-inc"
# load the processor
processor = AutoProcessor.from_pretrained(
quantized_model_path,
trust_remote_code=True,
torch_dtype='auto',
device_map='auto'
)
# load the model
model = AutoModelForCausalLM.from_pretrained(
quantized_model_path,
trust_remote_code=True,
torch_dtype='auto',
device_map='auto',
##revision="e64d453" ##AutoGPTQ format
)
image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
text = "Describe this image."
# process the image and text
inputs = processor.process(
images=[Image.open(requests.get(image_url, stream=True).raw)],
text=text
)
# move inputs to the correct device and make a batch of size 1
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
inputs["images"] = inputs["images"].to(model.dtype)
# generate output; maximum 200 new tokens; stop generation when <|endoftext|> is generated
output = model.generate_from_batch(
inputs,
GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
tokenizer=processor.tokenizer
)
# only get generated tokens; decode them to text
generated_tokens = output[0,inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
# print the generated text
print(generated_text)
##INT4:
## In this serene beach scene, a woman with long, dark hair sits on the sandy shore, facing left. She is dressed in a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly as she reaches out to high-five a large, light brown dog. The dog, possibly a Labrador or a similar breed, sits on its hind legs with its front paws raised, eagerly engaging in the friendly gesture. The dog is adorned with a blue harness featuring a pattern of pink and blue flowers, and a red leash lies on the sand beside it. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, warm glow over the scene, enhancing the tranquil and joyful atmosphere.
##FP32:
## In this serene beach scene, a woman and her dog share a tender moment. The woman, with long dark hair, is seated on the sandy shore, her legs crossed as she faces the ocean. She is wearing a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly at her canine companion. The dog, a light brown Labrador, sits beside her with its front paws raised, eagerly reaching out to touch her hand. The dog is adorned with a blue harness decorated with pink and green flowers, and a red leash lies on the sand nearby. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, golden glow over the scene, enhancing the peaceful and joyful atmosphere.
image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
text = "How many people are there on the baseball field in the picture??"
##INT4:
## Counting the <points x1="46.5" y1="37.1" x2="58.6" y2="48.3" x3="76.5" y3="33.0" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.
##FP32:
## Counting the <points x1="46.5" y1="37.6" x2="58.5" y2="49.0" x3="76.0" y3="33.1" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.
image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
text = "Which company does this image represent?"
##INT4:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, followed by "INSIDE" in uppercase letters. This iconic logo design is instantly recognizable and has been a symbol of Intel's brand for many years.
##FP32:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, with "INSIDE" in uppercase letters below it. This iconic logo design is instantly recognizable and associated with Intel's brand in the computer industry.
```
## Evaluation the model
pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb.
```bash
auto-round-mllm --eval --model OPEA/Molmo-7B-D-0924-int4-sym-inc --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"
```
|Metric |16bits| INT4 |
|:------------------|:-----|:------|
|avg |69.22 |68.90 |
|MMBench_DEV_EN_V11 |70.89 |69.20 |
|ScienceQA_VAL |35.81 |36.34 |
|TextVQA_VAL |81.16 |80.64 |
|POPE |89.02 |89.40 |
### Generate the model
Here is the sample command to reproduce the model.
```bash
pip install auto-round
auto-round-mllm
--model allenai/Molmo-7B-D-0924 \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 1000 \
--nsamples 512 \
--seqlen 2048 \
--format 'auto_gptq,auto_round' \
--output_dir "./tmp_autoround"
```
## Ethical Considerations and Limitations
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
## Caveats and Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
## Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
## Cite
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round) |