Crystalcareai commited on
Commit
af50241
·
verified ·
1 Parent(s): 0d3349c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +68 -0
README.md CHANGED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - mistralai/Mistral-Small-24B-Base-2501
5
+ ---
6
+
7
+ # Arcee Blitz Caller Beta
8
+
9
+ This repository contains a beta version of the Arcee Blitz Caller, a powerful tool for automated function calling and tool selection. Below you'll find instructions on how to launch the service using vllm as well as detailed performance metrics.
10
+
11
+ ## Overview
12
+
13
+ Arcee Blitz Caller is built on top of the Mistral-Small-24B-Base-2501 model and is designed for efficient and accurate tool calling operations.
14
+
15
+ ## Base Model
16
+
17
+ - [mistralai/Mistral-Small-24B-Base-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Base-2501)
18
+
19
+ ## Quick Start
20
+
21
+ To launch the service using vllm, run the following command:
22
+
23
+ ```bash
24
+ vllm serve arcee-ai/Arcee-Blitz-Caller \
25
+ --enable-auto-tool-choice \
26
+ --tool-call-parser hermes
27
+ ```
28
+
29
+ ## Performance Metrics
30
+
31
+ ### Non-Live Performance
32
+ | Metric | Score |
33
+ |--------|--------|
34
+ | AST Accuracy | 85.15% |
35
+ | Simple AST | 70.08% |
36
+ | Multiple AST | 93.50% |
37
+ | Parallel AST | 89.50% |
38
+ | Parallel Multiple AST | 87.50% |
39
+
40
+ ### Live Performance
41
+ | Metric | Score |
42
+ |--------|--------|
43
+ | Accuracy | 74.19% |
44
+ | Simple AST | 71.71% |
45
+ | Multiple AST | 75.02% |
46
+ | Parallel AST | 43.75% |
47
+ | Parallel Multiple AST | 66.67% |
48
+
49
+ ### Multi-Turn Performance
50
+ | Metric | Score |
51
+ |--------|--------|
52
+ | Accuracy | 0.25% |
53
+ | Base | 0.50% |
54
+ | Miss Function | 0.00% |
55
+ | Miss Parameter | 0.00% |
56
+ | Long Context | 0.50% |
57
+ | Relevance Detection | 61.11% |
58
+ | Irrelevance Detection | 77.68% |
59
+
60
+ > **Note:** The model demonstrates strong performance in single-turn interactions but currently shows limitations in multi-turn scenarios. We are actively developing new training data to enhance multi-turn capabilities.
61
+
62
+ ## Current Status
63
+
64
+ This is a beta release and is under active development. While the model shows promising results in many areas, users should be aware of its current limitations, particularly in multi-turn interactions.
65
+
66
+ ## License
67
+
68
+ This project is licensed under the Apache-2.0 License - see the [LICENSE](LICENSE) file for details.