nisten

/

obsidian-3b-multimodal-q6-gguf

like 69

Using this image:

and this llama server:

 ./server -m models/obsidian-f16.gguf --mmproj models/mmproj-obsidian-f16.gguf --host 0.0.0.0  -ngl 42

and this python code:

import requests
import json
import base64

image_path = '/home/nawal/IMG_20231210_220649_640x480.png'

with open(image_path, 'rb') as image_file:
    encoded_string = base64.b64encode(image_file.read()).decode('utf-8')

url = 'http://192.168.0.52:8080/completion'
headers = { 'Content-Type': 'application/json' }
data = {
    'prompt': '''
<|im_start|>user
What does this image contain?\n[img-1]
###
<|im_start|>assistant''',
    'n_predict': 512,
    'top_k': 40,
    'temperature': 0.2,
    'image_data': [{'data': encoded_string, 'id': 1}]
}
response = requests.post(url, headers=headers, data=json.dumps(data))

# Check the response
if response.status_code == 200:
    print('Request successful!')
    print(response.json())  # This will print the response content
else:
    print('Request failed with status code:', response.status_code)

The model keeps talking and eventually barfs out ###\n - any idea how to make it stop while it's ahead? Any hints on prompts to make it less verbose or less likely to put 'assistant:' in the output stream several times?

{'content': "\n: The image features a wooden table with three Legos pieces on it.\n###\n###\nassistant\n: Two of the Legos are red, while the third one is blue. They are arranged in a way that makes them look like they are standing or sitting next to each other.\n###\n###\nassistant\n: The Legos are placed on top of the table, creating an interesting and playful scene.\n###\n###\n###\nassistant\n: The Legos' arrangement suggests that they might be part of a creative display or decoration project. This could be for various purposes such as home decor, a themed event, or even a toy store display.\n###\n###\n###\n###\n###\nassistant\n: The Legos are placed in a way that makes it easy to see the different colors and shapes of each piece. This arrangement creates an eye-catching display that can evoke a sense of playfulness and imagination, making the scene more engaging for viewers.\n###\n###\n###\n###\n###\n###\n###\n###\nassistant\n: The Legos are arranged in such a way that they create a visually appealing composition on the table. Their contrasting colors and different shapes add depth to the display, making it more interesting and dynamic. This type of arrangement can be used for various purposes, such as home decoration or artistic projects, where the unique combination of colors and shapes can evoke emotions and spark creativity.\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###\n###", 'generation_settings': {'frequency_penalty': 0.0, 'grammar': '', 'ignore_eos': False, 'logit_bias': [], 'min_p': 0.05000000074505806, 'mirostat': 0, 'mirostat_eta': 0.10000000149011612, 'mirostat_tau': 5.0, 'model': 'models/obsidian-f16.gguf', 'n_ctx': 2048, 'n_keep': 0, 'n_predict': 512, 'n_probs': 0, 'penalize_nl': True, 'presence_penalty': 0.0, 'repeat_last_n': 64, 'repeat_penalty': 1.100000023841858, 'seed': 4294967295, 'stop': [], 'stream': False, 'temp': 0.20000000298023224, 'tfs_z': 1.0, 'top_k': 40, 'top_p': 0.949999988079071, 'typical_p': 1.0}, 'model': 'models/obsidian-f16.gguf', 'prompt': '', 'slot_id': 0, 'stop': True, 'stopped_eos': False, 'stopped_limit': True, 'stopped_word': False, 'stopping_word': '', 'timings': {'predicted_ms': 40839.297, 'predicted_n': 512, 'predicted_per_second': 12.536944502252329, 'predicted_per_token_ms': 79.764251953125, 'prompt_ms': 4771.674, 'prompt_n': 0, 'prompt_per_second': 0.0, 'prompt_per_token_ms': None}, 'tokens_cached': 1105, 'tokens_evaluated': 0, 'tokens_predicted': 512, 'truncated': False}