File size: 1,840 Bytes
7e1729b 97385c4 7e1729b 97385c4 7e1729b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
license: mit
pipeline_tag: image-text-to-text
library_name: phantom
---
Paper: https://huggingface.co./papers/2409.14713
# Two steps only need.
First step. (git clone and install required packages)
```bash
# Download Project Code
git clone https://github.com/ByungKwanLee/Phantom
# Virtual Environment
conda create -n trol python=3.11 -y
conda activate trol
# install torch
pip3 install torch torchvision
# install requiresments
pip install -r requirements.txt
# flash attention
pip install flash-attn --no-build-isolation
# all cache deleted
conda clean -a && pip cache purge
```
Second step. (open, edit, and run `demo.py`)
```python
# model selection
size = '0.5b' # [Select One] '0.5b' (transformers more recent version) | '1.8b' | '3.8b' (transformers==4.37.2) | '7b'
# User prompt
prompt_type="with_image" # Select one option "text_only", "with_image"
img_path='figures/demo.png'
question="Describe the image in detail"
# loading model
model, tokenizer = load_model(size=size)
# prompt type -> input prompt
if prompt_type == 'with_image':
# Image Load
image = pil_to_tensor(Image.open(img_path).convert("RGB"))
inputs = [{'image': image, 'question': question}]
elif prompt_type=='text_only':
inputs = [{'question': question}]
# cpu -> gpu
for param in model.parameters():
if not param.is_cuda:
param.data = param.cuda()
# Generate
with torch.inference_mode():
# Model
_inputs = model.eval_process(inputs=inputs,
data='demo',
tokenizer=tokenizer,
device='cuda:0')
generate_ids = model.generate(**_inputs, do_sample=False, max_new_tokens=256)
answer = tokenizer.batch_decode(generate_ids, skip_special_tokens=True)[0]
print(answer)
```
So easy to run the code Let's shout Phantom! |