peiyi9979

/

math-shepherd-mistral-7b-prm

like 29

We are using VLLM for accelerated inference of math-shepherd.
The results are slightly different in values compared to hf version.
Wonder if this is due to VLLM implementation, and if this difference is acceptable.
VLLM Output:

[0.9662998914718628, 0.9618092775344849, 0.9654793739318848, 0.9461783170700073]
[0.9662998914718628, 0.9618092775344849, 0.9654793739318848, 0.16878120601177216]

HF Output:

# tensor([0.9955, 0.9958, 0.9983, 0.9957])
# tensor([0.9955, 0.9958, 0.9983, 0.0240])

vllm version: 0.5.5

import vllm
import torch
import numpy as np
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(
    logprobs=20,
    prompt_logprobs=20,
)

llm = LLM(model='/root/huggingface/math-shepherd-mistral-7b-prm', dtype='float32', tensor_parallel_size=1, gpu_memory_utilization=0.9, max_model_len=4096)
question = """Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?"""
output1 = """Step 1: Janet's ducks lay 16 eggs per day. ки\nStep 2: She eats three for breakfast every morning, so she has 16 - 3 = 13 eggs left. ки\nStep 3: She bakes muffins for her friends every day with four eggs, so she has 13 - 4 = 9 eggs left. ки\nStep 4: She sells the remainder at the farmers' market daily for $2 per fresh duck egg, so she makes 9 * $2 = $18 every day at the farmers' market. The answer is: 18 ки""" # 18 is right
output2 = """Step 1: Janet's ducks lay 16 eggs per day. ки\nStep 2: She eats three for breakfast every morning, so she has 16 - 3 = 13 eggs left. ки\nStep 3: She bakes muffins for her friends every day with four eggs, so she has 13 - 4 = 9 eggs left. ки\nStep 4: She sells the remainder at the farmers' market daily for $2 per fresh duck egg, so she makes 9 * $2 = $17 every day at the farmers' market. The answer is: 17 ки""" # 17 is wrong

good_token = '+'
bad_token = '-'
step_tag = 'ки'

tokenizer = llm.get_tokenizer()
[good_token_id, bad_token_id, step_tag_id] = tokenizer.encode(f"{good_token} {bad_token} {step_tag}")[1:] # [648, 387]

full_prompt = [f"{question} {output}" for output in [output1, output2]]
response = llm.generate(full_prompt, sampling_params)

for res in response:
    prompt_logprobs = res.prompt_logprobs
    all_tokens = res.prompt_token_ids
    tag_token_index = [i for i, token in enumerate(all_tokens) if token == step_tag_id]
    results = []
    for token_index in tag_token_index:
        logprobs = prompt_logprobs[token_index]
        good_score = 0
        bad_score = 0
        if good_token_id in logprobs:
            good_score = logprobs[good_token_id].logprob
        if bad_token_id in logprobs:
            bad_score = logprobs[bad_token_id].logprob
            
        normalized_good_score = torch.softmax(torch.tensor([good_score, bad_score]), dim=0)[0].item()
        results.append(normalized_good_score)
    print(results)