Hugging Face's logo

legraphista
/
Phi-3-medium-128k-instruct-IMat-GGUF

Text Generation
GGUF
multilingual
quantized
GGUF
imatrix
quantization
imat
static
conversational
Model card Files Community
legraphista commited on
Commit
5e5961b
•
1 Parent(s): 371a94f

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +167 -0
imatrix.log ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ main: build = 3003 (d298382a)
2
+ main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
3
+ main: seed = 1716755471
4
+ llama_model_loader: loaded meta data with 27 key-value pairs and 245 tensors from Phi-3-medium-128k-instruct-IMat-GGUF/Phi-3-medium-128k-instruct.gguf (version GGUF V3 (latest))
5
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
6
+ llama_model_loader: - kv 0: general.architecture str = phi3
7
+ llama_model_loader: - kv 1: general.name str = Phi3
8
+ llama_model_loader: - kv 2: phi3.context_length u32 = 131072
9
+ llama_model_loader: - kv 3: phi3.rope.scaling.original_context_length u32 = 4096
10
+ llama_model_loader: - kv 4: phi3.embedding_length u32 = 5120
11
+ llama_model_loader: - kv 5: phi3.feed_forward_length u32 = 17920
12
+ llama_model_loader: - kv 6: phi3.block_count u32 = 40
13
+ llama_model_loader: - kv 7: phi3.attention.head_count u32 = 40
14
+ llama_model_loader: - kv 8: phi3.attention.head_count_kv u32 = 10
15
+ llama_model_loader: - kv 9: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010
16
+ llama_model_loader: - kv 10: phi3.rope.dimension_count u32 = 128
17
+ llama_model_loader: - kv 11: phi3.rope.freq_base f32 = 10000.000000
18
+ llama_model_loader: - kv 12: general.file_type u32 = 0
19
+ llama_model_loader: - kv 13: phi3.rope.scaling.attn_factor f32 = 1.190238
20
+ llama_model_loader: - kv 14: tokenizer.ggml.model str = llama
21
+ llama_model_loader: - kv 15: tokenizer.ggml.pre str = default
22
+ llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
23
+ llama_model_loader: - kv 17: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00...
24
+ llama_model_loader: - kv 18: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
25
+ llama_model_loader: - kv 19: tokenizer.ggml.bos_token_id u32 = 1
26
+ llama_model_loader: - kv 20: tokenizer.ggml.eos_token_id u32 = 32000
27
+ llama_model_loader: - kv 21: tokenizer.ggml.unknown_token_id u32 = 0
28
+ llama_model_loader: - kv 22: tokenizer.ggml.padding_token_id u32 = 32000
29
+ llama_model_loader: - kv 23: tokenizer.ggml.add_bos_token bool = false
30
+ llama_model_loader: - kv 24: tokenizer.ggml.add_eos_token bool = false
31
+ llama_model_loader: - kv 25: tokenizer.chat_template str = {% for message in messages %}{% if (m...
32
+ llama_model_loader: - kv 26: general.quantization_version u32 = 2
33
+ llama_model_loader: - type f32: 245 tensors
34
+ llm_load_vocab: special tokens definition check successful ( 323/32064 ).
35
+ llm_load_print_meta: format = GGUF V3 (latest)
36
+ llm_load_print_meta: arch = phi3
37
+ llm_load_print_meta: vocab type = SPM
38
+ llm_load_print_meta: n_vocab = 32064
39
+ llm_load_print_meta: n_merges = 0
40
+ llm_load_print_meta: n_ctx_train = 131072
41
+ llm_load_print_meta: n_embd = 5120
42
+ llm_load_print_meta: n_head = 40
43
+ llm_load_print_meta: n_head_kv = 10
44
+ llm_load_print_meta: n_layer = 40
45
+ llm_load_print_meta: n_rot = 128
46
+ llm_load_print_meta: n_embd_head_k = 128
47
+ llm_load_print_meta: n_embd_head_v = 128
48
+ llm_load_print_meta: n_gqa = 4
49
+ llm_load_print_meta: n_embd_k_gqa = 1280
50
+ llm_load_print_meta: n_embd_v_gqa = 1280
51
+ llm_load_print_meta: f_norm_eps = 0.0e+00
52
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-05
53
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
54
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
55
+ llm_load_print_meta: f_logit_scale = 0.0e+00
56
+ llm_load_print_meta: n_ff = 17920
57
+ llm_load_print_meta: n_expert = 0
58
+ llm_load_print_meta: n_expert_used = 0
59
+ llm_load_print_meta: causal attn = 1
60
+ llm_load_print_meta: pooling type = 0
61
+ llm_load_print_meta: rope type = 2
62
+ llm_load_print_meta: rope scaling = linear
63
+ llm_load_print_meta: freq_base_train = 10000.0
64
+ llm_load_print_meta: freq_scale_train = 1
65
+ llm_load_print_meta: n_yarn_orig_ctx = 4096
66
+ llm_load_print_meta: rope_finetuned = unknown
67
+ llm_load_print_meta: ssm_d_conv = 0
68
+ llm_load_print_meta: ssm_d_inner = 0
69
+ llm_load_print_meta: ssm_d_state = 0
70
+ llm_load_print_meta: ssm_dt_rank = 0
71
+ llm_load_print_meta: model type = 14B
72
+ llm_load_print_meta: model ftype = all F32
73
+ llm_load_print_meta: model params = 13.96 B
74
+ llm_load_print_meta: model size = 52.01 GiB (32.00 BPW)
75
+ llm_load_print_meta: general.name = Phi3
76
+ llm_load_print_meta: BOS token = 1 '<s>'
77
+ llm_load_print_meta: EOS token = 32000 '<|endoftext|>'
78
+ llm_load_print_meta: UNK token = 0 '<unk>'
79
+ llm_load_print_meta: PAD token = 32000 '<|endoftext|>'
80
+ llm_load_print_meta: LF token = 13 '<0x0A>'
81
+ llm_load_print_meta: EOT token = 32007 '<|end|>'
82
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
83
+ ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
84
+ ggml_cuda_init: found 1 CUDA devices:
85
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
86
+ llm_load_tensors: ggml ctx size = 0.28 MiB
87
+ llm_load_tensors: offloading 16 repeating layers to GPU
88
+ llm_load_tensors: offloaded 16/41 layers to GPU
89
+ llm_load_tensors: CPU buffer size = 53254.08 MiB
90
+ llm_load_tensors: CUDA0 buffer size = 20800.63 MiB
91
+ .......................................................................................
92
+ llama_new_context_with_model: n_ctx = 512
93
+ llama_new_context_with_model: n_batch = 512
94
+ llama_new_context_with_model: n_ubatch = 512
95
+ llama_new_context_with_model: flash_attn = 0
96
+ llama_new_context_with_model: freq_base = 10000.0
97
+ llama_new_context_with_model: freq_scale = 1
98
+ llama_kv_cache_init: CUDA_Host KV buffer size = 60.00 MiB
99
+ llama_kv_cache_init: CUDA0 KV buffer size = 40.00 MiB
100
+ llama_new_context_with_model: KV self size = 100.00 MiB, K (f16): 50.00 MiB, V (f16): 50.00 MiB
101
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB
102
+ llama_new_context_with_model: CUDA0 compute buffer size = 840.00 MiB
103
+ llama_new_context_with_model: CUDA_Host compute buffer size = 20.98 MiB
104
+ llama_new_context_with_model: graph nodes = 1606
105
+ llama_new_context_with_model: graph splits = 220
106
+
107
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
108
+ compute_imatrix: tokenizing the input ..
109
+ compute_imatrix: tokenization took 133.033 ms
110
+ compute_imatrix: computing over 234 chunks with batch_size 512
111
+ compute_imatrix: 2.40 seconds per pass - ETA 9.37 minutes
112
+ [1]4.4810,[2]3.4854,[3]3.4301,[4]3.6863,[5]4.1180,[6]4.2207,[7]3.8129,[8]4.1860,[9]4.4113,
113
+ save_imatrix: stored collected data after 10 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
114
+ [10]4.6985,[11]4.7333,[12]4.4193,[13]4.5511,[14]4.4712,[15]4.8532,[16]4.9821,[17]5.2736,[18]5.4072,[19]5.5905,
115
+ save_imatrix: stored collected data after 20 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
116
+ [20]5.7241,[21]5.8095,[22]5.9993,[23]5.7705,[24]5.6384,[25]5.6408,[26]5.3666,[27]5.1654,[28]4.9147,[29]4.8876,
117
+ save_imatrix: stored collected data after 30 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
118
+ [30]4.9822,[31]5.0317,[32]5.0841,[33]5.0646,[34]5.0884,[35]5.0893,[36]4.9139,[37]4.7968,[38]4.7524,[39]4.7424,
119
+ save_imatrix: stored collected data after 40 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
120
+ [40]4.7279,[41]4.6708,[42]4.7080,[43]4.7361,[44]4.7795,[45]4.8462,[46]4.9090,[47]4.9759,[48]5.0986,[49]5.1941,
121
+ save_imatrix: stored collected data after 50 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
122
+ [50]5.2907,[51]5.3709,[52]5.4552,[53]5.4307,[54]5.3597,[55]5.3080,[56]5.3784,[57]5.4179,[58]5.4352,[59]5.4922,
123
+ save_imatrix: stored collected data after 60 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
124
+ [60]5.5560,[61]5.5745,[62]5.6139,[63]5.6415,[64]5.6889,[65]5.7133,[66]5.7433,[67]5.7732,[68]5.8083,[69]5.8716,
125
+ save_imatrix: stored collected data after 70 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
126
+ [70]5.9098,[71]5.9411,[72]5.9714,[73]5.9286,[74]5.8875,[75]5.7936,[76]5.7094,[77]5.6668,[78]5.5887,[79]5.5385,
127
+ save_imatrix: stored collected data after 80 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
128
+ [80]5.4780,[81]5.4016,[82]5.3362,[83]5.2999,[84]5.2918,[85]5.3183,[86]5.3337,[87]5.3620,[88]5.3804,[89]5.3639,
129
+ save_imatrix: stored collected data after 90 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
130
+ [90]5.2972,[91]5.3001,[92]5.2878,[93]5.2948,[94]5.2938,[95]5.3006,[96]5.3134,[97]5.3192,[98]5.3045,[99]5.2758,
131
+ save_imatrix: stored collected data after 100 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
132
+ [100]5.2885,[101]5.3082,[102]5.3067,[103]5.2856,[104]5.2458,[105]5.2354,[106]5.2487,[107]5.2657,[108]5.2514,[109]5.2471,
133
+ save_imatrix: stored collected data after 110 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
134
+ [110]5.2362,[111]5.2459,[112]5.2581,[113]5.2588,[114]5.2725,[115]5.2767,[116]5.2727,[117]5.2718,[118]5.2804,[119]5.2674,
135
+ save_imatrix: stored collected data after 120 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
136
+ [120]5.2713,[121]5.2633,[122]5.2464,[123]5.2637,[124]5.2612,[125]5.2696,[126]5.2574,[127]5.2593,[128]5.2702,[129]5.2540,
137
+ save_imatrix: stored collected data after 130 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
138
+ [130]5.2362,[131]5.2287,[132]5.2281,[133]5.1918,[134]5.1895,[135]5.1684,[136]5.1480,[137]5.1234,[138]5.0985,[139]5.0746,
139
+ save_imatrix: stored collected data after 140 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
140
+ [140]5.0538,[141]5.0340,[142]5.0132,[143]5.0052,[144]4.9975,[145]4.9791,[146]4.9583,[147]4.9526,[148]4.9368,[149]4.9266,
141
+ save_imatrix: stored collected data after 150 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
142
+ [150]4.9150,[151]4.9036,[152]4.8937,[153]4.8784,[154]4.8674,[155]4.8813,[156]4.8594,[157]4.8542,[158]4.8593,[159]4.8556,
143
+ save_imatrix: stored collected data after 160 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
144
+ [160]4.8535,[161]4.8535,[162]4.8522,[163]4.8599,[164]4.8650,[165]4.8772,[166]4.8805,[167]4.8787,[168]4.8838,[169]4.8911,
145
+ save_imatrix: stored collected data after 170 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
146
+ [170]4.9040,[171]4.9000,[172]4.9053,[173]4.9210,[174]4.9268,[175]4.9438,[176]4.9541,[177]4.9646,[178]4.9726,[179]4.9948,
147
+ save_imatrix: stored collected data after 180 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
148
+ [180]5.0044,[181]5.0426,[182]5.0583,[183]5.0790,[184]5.0869,[185]5.0948,[186]5.1043,[187]5.1094,[188]5.1017,[189]5.1070,
149
+ save_imatrix: stored collected data after 190 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
150
+ [190]5.1140,[191]5.1248,[192]5.1314,[193]5.1552,[194]5.1474,[195]5.1256,[196]5.1589,[197]5.1894,[198]5.2157,[199]5.2588,
151
+ save_imatrix: stored collected data after 200 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
152
+ [200]5.2955,[201]5.3054,[202]5.3112,[203]5.2801,[204]5.2823,[205]5.2896,[206]5.3126,[207]5.3107,[208]5.3150,[209]5.3179,
153
+ save_imatrix: stored collected data after 210 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
154
+ [210]5.3271,[211]5.3399,[212]5.3411,[213]5.3404,[214]5.3484,[215]5.3635,[216]5.3794,[217]5.3830,[218]5.3796,[219]5.3782,
155
+ save_imatrix: stored collected data after 220 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
156
+ [220]5.3737,[221]5.3749,[222]5.3753,[223]5.3841,[224]5.3695,[225]5.3727,[226]5.3613,[227]5.3894,[228]5.4175,[229]5.4522,
157
+ save_imatrix: stored collected data after 230 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
158
+ [230]5.4826,[231]5.4968,[232]5.4777,[233]5.4565,[234]5.4323,
159
+ save_imatrix: stored collected data after 234 chunks in Phi-3-medium-128k-instruct-IMat-GGUF/imatrix.dat
160
+
161
+ llama_print_timings: load time = 5039.95 ms
162
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
163
+ llama_print_timings: prompt eval time = 543719.95 ms / 119808 tokens ( 4.54 ms per token, 220.35 tokens per second)
164
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
165
+ llama_print_timings: total time = 547185.12 ms / 119809 tokens
166
+
167
+ Final estimate: PPL = 5.4323 +/- 0.05178