Create README.md

README.md

@@ -0,0 +1,113 @@
+---
+datasets:
+- liuhaotian/LLaVA-Pretrain
+- liuhaotian/LLaVA-Instruct-150K
+language:
+- en
+tags:
+- llava
+- phi
+---
+
+# LLaVA-3b Model Card
+
+## Model details
+
+LLaVA-3b is a model fine-tuned from [Dolphin 2.6 Phi](https://huggingface.co/cognitivecomputations/dolphin-2_6-phi-2) in a LLaVA fashion using vision tower from
+[SigLIP 400M](https://huggingface.co/timm/ViT-SO400M-14-SigLIP-384). There are a couple of things different from the original LLaVA architecture:
+
+1. Multiple image tokens. The multimodal projector generates embeddings of shape [5, 2560] instead of [1, 2560] for images. The idea is that using more tokens
+   allows to get more info from the image into the language model.
+2. The model uses the output from the latest layer of the vision encoder instead of intermediate one.
+
+As Dolphin 2.6 Phi, LLaVA-3b uses ChatML prompt format:
+
+```
+<|im_start|>system
+You are Dolphin, a helpful AI assistant.<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant
+```
+
+## How to use
+
+**Install dependencies**
+
+```
+!pip install -q open_clip_torch timm einops
+```
+
+**Download modeling files**
+
+```
+from huggingface_hub import hf_hub_download
+
+hf_hub_download(repo_id="visheratin/LLaVA-3b", filename="configuration_llava.py", local_dir="./", force_download=True)
+hf_hub_download(repo_id="visheratin/LLaVA-3b", filename="configuration_phi.py", local_dir="./", force_download=True)
+hf_hub_download(repo_id="visheratin/LLaVA-3b", filename="modeling_llava.py", local_dir="./", force_download=True)
+hf_hub_download(repo_id="visheratin/LLaVA-3b", filename="modeling_phi.py", local_dir="./", force_download=True)
+hf_hub_download(repo_id="visheratin/LLaVA-3b", filename="processing_llava.py", local_dir="./", force_download=True)
+```
+
+**Create a model**
+
+```
+from modeling_llava import LlavaForConditionalGeneration
+import torch
+
+model = LlavaForConditionalGeneration.from_pretrained("visheratin/LLaVA-3b", torch_dtype=torch.float16)
+model = model.to("cuda")
+```
+
+**Create processors**
+
+```
+from transformers import AutoTokenizer
+from processing_llava import LlavaProcessor, OpenCLIPImageProcessor
+
+tokenizer = AutoTokenizer.from_pretrained("visheratin/LLaVA-3b")
+image_processor = OpenCLIPImageProcessor(model.config.preprocess_config)
+processor = LlavaProcessor(image_processor, tokenizer)
+```
+
+**Set image and text**
+
+```
+from PIL import Image
+import requests
+
+image_file = "https://images.unsplash.com/photo-1439246854758-f686a415d9da"
+raw_image = Image.open(requests.get(image_file, stream=True).raw)
+
+prompt = """<|im_start|>system
+A chat between a curious human and an artificial intelligence assistant.
+The assistant gives helpful, detailed, and polite answers to the human's questions.
+The assistant does not hallucinate and pays very close attention to the details.<|im_end|>
+<|im_start|>user
+<image>
+Describe the image.<|im_end|>
+<|im_start|>assistant
+"""
+```
+
+**Process inputs**
+
+```
+inputs = processor(prompt, raw_image, model, return_tensors='pt')
+
+inputs['input_ids'] = inputs['input_ids'].to(model.device)
+inputs['attention_mask'] = inputs['attention_mask'].to(model.device)
+```
+
+**Generate the data**
+
+```
+output = model.generate(**inputs, max_new_tokens=200, do_sample=True, top_p=0.5, temperature=1.2, eos_token_id=tokenizer.eos_token_id)
+```
+
+## License
+This model is based on Phi-2 and is governed by Microsoft's microsoft-research-license which prohibits commercial use.
+
+**Where to send questions or comments about the model:**
+https://twitter.com/visheratin