OPEA
/

Safetensors
molmo
custom_code
4-bit precision
intel/auto-round
File size: 7,029 Bytes
fab5e76
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e6da6d9
89075fd
fab5e76
 
 
 
 
 
 
72c90ad
fab5e76
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1c3c008
fab5e76
 
 
 
 
 
72c90ad
fab5e76
1c3c008
72c90ad
fab5e76
 
 
 
 
72c90ad
fab5e76
1c3c008
72c90ad
 
fab5e76
 
b7a988e
 
 
 
 
 
 
 
 
 
 
 
 
fab5e76
 
 
 
 
 
 
 
 
 
19838f3
fab5e76
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
---
license: apache-2.0
datasets:
- NeelNanda/pile-10k
base_model:
- allenai/Molmo-7B-D-0924
---

## Model Details

This model is an int4 model with group_size 128 and symmetric quantization of [allenai/Molmo-7B-D-0924](https://huggingface.co./allenai/Molmo-7B-D-0924) generated by [intel/auto-round](https://github.com/intel/auto-round). Load the model with revision="e64d453" to use AutoGPTQ format.

## How To Use
### INT4 Inference
```python
from auto_round import AutoRoundConfig ## must import for auto-round format
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests

quantized_model_path = "OPEA/Molmo-7B-D-0924-int4-sym-inc"

# load the processor
processor = AutoProcessor.from_pretrained(
    quantized_model_path,
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto'
)

# load the model
model = AutoModelForCausalLM.from_pretrained(
    quantized_model_path,
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto',
    ##revision="e64d453" ##AutoGPTQ format
)

image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
text = "Describe this image."
# process the image and text
inputs = processor.process(
    images=[Image.open(requests.get(image_url, stream=True).raw)],
    text=text
)

# move inputs to the correct device and make a batch of size 1
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
inputs["images"] = inputs["images"].to(model.dtype)

# generate output; maximum 200 new tokens; stop generation when <|endoftext|> is generated
output = model.generate_from_batch(
    inputs,
    GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
    tokenizer=processor.tokenizer
)

# only get generated tokens; decode them to text
generated_tokens = output[0,inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)

# print the generated text
print(generated_text)
##INT4:
## In this serene beach scene, a woman with long, dark hair sits on the sandy shore, facing left. She is dressed in a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly as she reaches out to high-five a large, light brown dog. The dog, possibly a Labrador or a similar breed, sits on its hind legs with its front paws raised, eagerly engaging in the friendly gesture. The dog is adorned with a blue harness featuring a pattern of pink and blue flowers, and a red leash lies on the sand beside it. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, warm glow over the scene, enhancing the tranquil and joyful atmosphere.

##FP32:
## In this serene beach scene, a woman and her dog share a tender moment. The woman, with long dark hair, is seated on the sandy shore, her legs crossed as she faces the ocean. She is wearing a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly at her canine companion. The dog, a light brown Labrador, sits beside her with its front paws raised, eagerly reaching out to touch her hand. The dog is adorned with a blue harness decorated with pink and green flowers, and a red leash lies on the sand nearby. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, golden glow over the scene, enhancing the peaceful and joyful atmosphere.


image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
text = "How many people are there on the baseball field in the picture??"
##INT4:
## Counting the <points x1="46.5" y1="37.1" x2="58.6" y2="48.3" x3="76.5" y3="33.0" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.

##FP32:
## Counting the <points x1="46.5" y1="37.6" x2="58.5" y2="49.0" x3="76.0" y3="33.1" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.


image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
text = "Which company does this image represent?"
##INT4:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, followed by "INSIDE" in uppercase letters. This iconic logo design is instantly recognizable and has been a symbol of Intel's brand for many years.

##FP32:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, with "INSIDE" in uppercase letters below it. This iconic logo design is instantly recognizable and associated with Intel's brand in the computer industry.

```

## Evaluation the model
pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb.
```bash
auto-round-mllm --eval --model OPEA/Molmo-7B-D-0924-int4-sym-inc --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"
```
|Metric             |16bits| INT4  | 
|:------------------|:-----|:------|
|avg                |69.22 |68.90  |
|MMBench_DEV_EN_V11 |70.89 |69.20  |
|ScienceQA_VAL      |35.81 |36.34  |
|TextVQA_VAL        |81.16 |80.64  |
|POPE               |89.02 |89.40  |

### Generate the model
Here is the sample command to reproduce the model.
```bash
pip install auto-round
auto-round-mllm
--model allenai/Molmo-7B-D-0924 \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 1000 \
--nsamples 512 \
--seqlen 2048 \
--format 'auto_gptq,auto_round' \
--output_dir "./tmp_autoround"
```

## Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

## Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)

## Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

## Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)