File size: 3,132 Bytes
62b7847 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
---
license: apache-2.0
language:
- en
pipeline_tag: video-text-to-text
tags:
- video
- video-understanding
- vision
- multimodal
- conversational
- custom_code
- instruction-tuning
library_name: transformers
---
# Apollo: An Exploration of Video Understanding in Large Multimodal Models
Apollo is a family of Large Multimodal Models (LMMs) that push the state-of-the-art in video understanding. It supports tasks including:
- Long-form video comprehension
- Temporal reasoning
- Complex video question-answering
- Multi-turn conversations grounded in video content
Apollo models excel at handling hour-long videos, balancing speed and accuracy through strategic design decisions. Our models outperform most 7B competitors at just 3B parameters and even rival 30B-scale models.
**Key Highlights:**
- **7B model varient**
- **32 tokens/frame**
## Quick Start
**Installation:**
```bash
pip install -e .
pip install flash-attn --no-build-isolation
```
**Inference Example:**
```python
import torch
from transformers import AutoModelForCausalLM
from apollo.mm_utils import (
KeywordsStoppingCriteria,
tokenizer_mm_token,
ApolloMMLoader
)
from apollo.conversations import conv_templates, SeparatorStyle
from huggingface_hub import snapshot_download
model_url = "Apollo-LMMs/Apollo-3B-t32"
model_path = snapshot_download(model_url, repo_type="model")
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
low_cpu_mem_usage=True
).to(device=device, dtype=torch.bfloat16)
tokenizer = model.tokenizer
vision_processors = model.vision_tower.vision_processor
config = model.config
num_repeat_token = config.mm_connector_cfg['num_output_tokens']
mm_processor = ApolloMMLoader(
vision_processors,
config.clip_duration,
frames_per_clip=4,
clip_sampling_ratio=0.65,
model_max_length=config.model_max_length,
device=device,
num_repeat_token=num_repeat_token
)
video_path = "path/to/video.mp4"
question = "Describe this video in detail"
mm_data, replace_string = mm_processor.load_video(video_path)
conv = conv_templates["qwen_2"].copy()
conv.append_message(conv.roles[0], replace_string + "\n\n" + question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
input_ids = tokenizer_mm_token(prompt, tokenizer, return_tensors="pt").unsqueeze(0).to(device)
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
stopping_criteria = KeywordsStoppingCriteria([stop_str], tokenizer, input_ids)
with torch.inference_mode():
output_ids = model.generate(
input_ids,
vision_input=[mm_data],
data_types=['video'],
do_sample=True,
temperature=0.4,
max_new_tokens=256,
top_p=0.7,
use_cache=True,
num_beams=1,
stopping_criteria=[stopping_criteria]
)
pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(pred)
```
For more details, visit the [project website](https://apollo-lmms.github.io) or check out the [paper](https://arxiv.org/abs/2412.10360). |