File size: 20,397 Bytes
2baf2e9 77b33ab 3535b0d 2baf2e9 77b33ab a941f8d 77b33ab a649ad3 77b33ab a649ad3 77b33ab a941f8d 77b33ab a649ad3 77b33ab a941f8d 77b33ab 30285cc a649ad3 5dcd1cc a649ad3 5dcd1cc a649ad3 77b33ab 5dcd1cc 77b33ab |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 |
---
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
- Hadith
- Islam
- Arabic
license: apache-2.0
datasets:
- FDSRashid/hadith_info
language:
- ar
library_name: sentence-transformers
---
# QulBERT
This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
This model originates from the [Camel-Bert_Classical Arabic](https://huggingface.co./CAMeL-Lab/bert-base-arabic-camelbert-ca) model. It was then trained on the Jawami' Kalim dataset,
specifically a dataset of 440,000 matns and their corresponding taraf labels.
Taraf labels indicate two hadith are about the same report, and as such, are more semantically similar.
## Usage (Sentence-Transformers)
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
```
pip install -U sentence-transformers
```
Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer
sentences = ["أنا أحب القراءة والكتابة.", "الطيور تحلق في السماء."]
model = SentenceTransformer('FDSRashid/QulBERT')
embeddings = model.encode(sentences)
print(embeddings)
```
## Usage (HuggingFace Transformers)
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
```python
from transformers import AutoTokenizer, AutoModel
import torch
#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output[0] #First element of model_output contains all token embeddings
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
# Sentences we want sentence embeddings for
sentences = ["أنا أحب القراءة والكتابة.", "الطيور تحلق في السماء."]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('FDSRashid/QulBERT')
model = AutoModel.from_pretrained('FDSRashid/QulBERT')
# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
model_output = model(**encoded_input)
# Perform pooling. In this case, mean pooling.
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
print("Sentence embeddings:")
print(sentence_embeddings)
```
## Evaluation Results
The dataset was split into 75% training, 15% eval, 10% test.
Validation Results during Training:
Binary Classification Evaluation:
| epoch | steps | cossim_accuracy | cossim_accuracy_threshold | cossim_f1 | cossim_precision | cossim_recall | cossim_f1_threshold | cossim_ap | manhattan_accuracy | manhattan_accuracy_threshold | manhattan_f1 | manhattan_precision | manhattan_recall | manhattan_f1_threshold | manhattan_ap | euclidean_accuracy | euclidean_accuracy_threshold | euclidean_f1 | euclidean_precision | euclidean_recall | euclidean_f1_threshold | euclidean_ap | dot_accuracy | dot_accuracy_threshold | dot_f1 | dot_precision | dot_recall | dot_f1_threshold | dot_ap |
|-------|-------|-----------------|---------------------------|--------------------|--------------------|---------------|---------------------|--------------------|--------------------|------------------------------|--------------------|---------------------|------------------|------------------------|--------------------|--------------------|------------------------------|--------------------|---------------------|------------------|------------------------|--------------------|--------------|------------------------|--------------------|--------------------|------------|--------------------|--------------------|
| 0 | 10000 | 0.87335 | 0.5980355739593506 | 0.866067203028869 | 0.9132749251413996 | 0.8235 | 0.5871663689613342 | 0.9466943574346693 | 0.87115 | 415.270751953125 | 0.8638917195787933 | 0.9047671172896941 | 0.82655 | 422.6612548828125 | 0.945152683467575 | 0.871875 | 18.699993133544922 | 0.8645460950343135 | 0.9041095890410958 | 0.8283 | 19.169795989990234 | 0.945247114112153 | 0.8731 | 262.36114501953125 | 0.8656574463026075 | 0.9177637107164864 | 0.81915 | 260.2767333984375 | 0.9463618096371682 |
| 0 | 20000 | 0.8655 | 0.5025224685668945 | 0.859078237761509 | 0.8968612304846869 | 0.82435 | 0.4888851046562195 | 0.943873652860419 | 0.866775 | 477.4580078125 | 0.860756186146168 | 0.9014832247824421 | 0.82355 | 485.6708984375 | 0.9442439376416185 | 0.8676 | 21.476741790771484 | 0.8606938065955735 | 0.8945580065800118 | 0.8293 | 22.168407440185547 | 0.9444315640627436 | 0.863225 | 241.51820373535156 | 0.8566324020610548 | 0.88835186080232 | 0.8271 | 230.02301025390625 | 0.9423405098129569 |
| 0 | -1 | 0.8866 | 0.7285321950912476 | 0.8816885280033313 | 0.919398610508033 | 0.84695 | 0.7145423889160156 | 0.9558629287413469 | 0.885275 | 355.03125 | 0.8803685772294236 | 0.918177869475513 | 0.84555 | 357.0611572265625 | 0.9550033563717418 | 0.8856 | 16.121074676513672 | 0.8809697221933201 | 0.918130557362828 | 0.8467 | 16.198532104492188 | 0.9552434220598536 | 0.8866 | 333.26568603515625 | 0.8812167536022311 | 0.9111929936986009 | 0.85315 | 325.6474304199219 | 0.9551592673018441 |
| 1 | 10000 | 0.88225 | 0.5847429037094116 | 0.8791732103956634 | 0.8909538967039737 | 0.8677 | 0.5608978271484375 | 0.9553668396978772 | 0.879975 | 404.1671447753906 | 0.8754545454545455 | 0.8843877551020408 | 0.8667 | 420.20391845703125 | 0.9539648051031446 | 0.879775 | 18.318096160888672 | 0.8759632369883004 | 0.8975394785163423 | 0.8554 | 18.77162742614746 | 0.9541900283694951 | 0.878325 | 242.8575897216797 | 0.8763834841057261 | 0.8859200980893022 | 0.86705 | 229.83326721191406 | 0.9521114062744855 |
| 1 | 20000 | 0.865425 | 0.483412504196167 | 0.8604195660017525 | 0.8878310817998085 | 0.83465 | 0.47202983498573303 | 0.9437698616032332 | 0.867725 | 490.8877868652344 | 0.8626237623762377 | 0.8905451448040886 | 0.8364 | 498.3052062988281 | 0.945935000502437 | 0.867725 | 21.84794044494629 | 0.8626810749177227 | 0.8954220237775028 | 0.83225 | 22.427053451538086 | 0.9460338001929801 | 0.862825 | 234.37701416015625 | 0.857083710699961 | 0.8882274068114776 | 0.82805 | 229.8949432373047 | 0.9405896665434951 |
| 1 | -1 | 0.866575 | 0.6635169982910156 | 0.8608573256557902 | 0.88173001310616 | 0.84095 | 0.6324930191040039 | 0.9452499579769719 | 0.866875 | 412.3456726074219 | 0.8617781992464822 | 0.8840511121628017 | 0.8406 | 428.6363525390625 | 0.9456397883265427 | 0.867275 | 18.474044799804688 | 0.8617669654289373 | 0.883254593175853 | 0.8413 | 19.42306900024414 | 0.9458234307667238 | 0.8645 | 340.140380859375 | 0.8589694801735291 | 0.8718648606890869 | 0.84645 | 320.98138427734375 | 0.9439794500521119 |
| 2 | 10000 | 0.85825 | 0.521987795829773 | 0.8545418167266907 | 0.8548839071257006 | 0.8542 | 0.4656229019165039 | 0.9388312444848291 | 0.85815 | 477.94244384765625 | 0.8541671894998369 | 0.8574595656774323 | 0.8509 | 508.4425048828125 | 0.9391298371675241 | 0.858625 | 21.995594024658203 | 0.854181558255897 | 0.8726267473398707 | 0.8365 | 22.506919860839844 | 0.9392082300175095 | 0.856875 | 262.9927673339844 | 0.8540649892527501 | 0.8640061396776669 | 0.84435 | 240.31259155273438 | 0.936665567408799 |
| 2 | 20000 | 0.861025 | 0.4792778789997101 | 0.8557936427338275 | 0.8693005983082319 | 0.8427 | 0.4365364611148834 | 0.9417166077380268 | 0.861325 | 490.29339599609375 | 0.8565778465126891 | 0.8688474000925783 | 0.84465 | 521.2939453125 | 0.9421024298390495 | 0.861225 | 22.693565368652344 | 0.8568891594997083 | 0.8697533089560694 | 0.8444 | 23.553585052490234 | 0.9422682260686701 | 0.859775 | 237.65704345703125 | 0.8545056078380817 | 0.8821400053233963 | 0.82855 | 224.57196044921875 | 0.9406093768234505 |
| 2 | -1 | 0.84645 | 0.7098060250282288 | 0.8385932801673421 | 0.8789257330775555 | 0.8018 | 0.702235221862793 | 0.932382298001216 | 0.849825 | 371.1478271484375 | 0.8419526841642077 | 0.8728131372759472 | 0.8132 | 385.735107421875 | 0.9344418607926894 | 0.8498 | 17.05820083618164 | 0.8418963040355231 | 0.8813781788351107 | 0.8058 | 17.261516571044922 | 0.9345154644039888 | 0.83745 | 359.3741455078125 | 0.8301335348954395 | 0.8366683595733875 | 0.8237 | 335.80609130859375 | 0.9256669298415723 |
| 3 | 10000 | 0.8692 | 0.6066867113113403 | 0.8639819190466407 | 0.8882551753274187 | 0.841 | 0.5866260528564453 | 0.9479885087178834 | 0.870575 | 437.38861083984375 | 0.8650388914644825 | 0.8920110485498778 | 0.83965 | 447.34051513671875 | 0.9484228602702792 | 0.870575 | 19.797679901123047 | 0.8655175071287281 | 0.8900512495376974 | 0.8423 | 20.318492889404297 | 0.9487290465239262 | 0.866525 | 297.63665771484375 | 0.862023653088042 | 0.8892669182924884 | 0.8364 | 295.425048828125 | 0.9460553171567032 |
| 3 | 20000 | 0.8723 | 0.5461836457252502 | 0.8661531678726109 | 0.8997790829247265 | 0.83495 | 0.5138773322105408 | 0.9483721005411583 | 0.872775 | 465.31109619140625 | 0.8667593021460553 | 0.8929063726009967 | 0.8421 | 492.2287292480469 | 0.9486788228598396 | 0.87305 | 21.46672821044922 | 0.8673375089844954 | 0.891221776746149 | 0.8447 | 22.356992721557617 | 0.9489411054456987 | 0.87085 | 268.9063720703125 | 0.8649067921503737 | 0.8955399689457622 | 0.8363 | 255.71820068359375 | 0.9471728845921085 |
| 3 | -1 | 0.8801 | 0.5941712260246277 | 0.8756740022187249 | 0.9045893076062044 | 0.84855 | 0.5840033292770386 | 0.9545450783524295 | 0.87755 | 432.21533203125 | 0.8737334773440313 | 0.8995022768188076 | 0.8494 | 439.35577392578125 | 0.9532505174511154 | 0.87805 | 19.783367156982422 | 0.873815256929146 | 0.8958924256749659 | 0.8528 | 20.03304100036621 | 0.953443356122637 | 0.880975 | 282.0526123046875 | 0.8761084893429446 | 0.9099429063880211 | 0.8447 | 279.80755615234375 | 0.9545987838548831 |
| 4 | 10000 | 0.850325 | 0.5770859718322754 | 0.8458372263326683 | 0.8544462017244018 | 0.8374 | 0.5550715923309326 | 0.9367499212412196 | 0.85215 | 446.5159606933594 | 0.8481793290514087 | 0.8686513968237329 | 0.82865 | 461.0674133300781 | 0.9378884193257083 | 0.85235 | 20.780521392822266 | 0.8487315362363361 | 0.8695903058280439 | 0.82885 | 20.85832977294922 | 0.9379865284776105 | 0.846775 | 297.2707214355469 | 0.8422069666920926 | 0.8568028970512157 | 0.8281 | 282.2066955566406 | 0.9342792490823187 |
| 4 | 20000 | 0.885725 | 0.5763461589813232 | 0.8810150085099798 | 0.9096815422302694 | 0.8541 | 0.5624827742576599 | 0.9567680001721202 | 0.8861 | 449.548095703125 | 0.881673031087419 | 0.9044113780955886 | 0.86005 | 462.72589111328125 | 0.9571293388400879 | 0.88635 | 20.378496170043945 | 0.8821848696234137 | 0.9091198472067483 | 0.8568 | 20.860164642333984 | 0.9573813976283176 | 0.883225 | 285.4012451171875 | 0.8786437246963561 | 0.8894467213114754 | 0.8681 | 268.5011291503906 | 0.9549886227962548 |
| 4 | -1 | 0.883425 | 0.5326807498931885 | 0.878749968085378 | 0.8978452548651328 | 0.86045 | 0.4872320890426636 | 0.956368376823993 | 0.88455 | 480.01300048828125 | 0.8794581927741869 | 0.9067063133860777 | 0.8538 | 497.9632568359375 | 0.9566012690704293 | 0.8845 | 21.905109405517578 | 0.8798647229125566 | 0.9022647259734118 | 0.85855 | 22.690349578857422 | 0.9567001435137067 | 0.881875 | 258.05084228515625 | 0.8778676433185817 | 0.8933637022466093 | 0.8629 | 243.83050537109375 | 0.9554938129957324 |
| 5 | 10000 | 0.893375 | 0.46282997727394104 | 0.8898161026116519 | 0.9175608201423563 | 0.8637 | 0.4469180405139923 | 0.9617751510273491 | 0.89385 | 512.1046752929688 | 0.8904936907301277 | 0.9161334672941674 | 0.86625 | 515.0869750976562 | 0.9619645895583173 | 0.894275 | 23.09744644165039 | 0.890798553215504 | 0.9146604856977295 | 0.86815 | 23.39638900756836 | 0.9622504494079881 | 0.892075 | 230.25645446777344 | 0.8889115628905951 | 0.9073582252773004 | 0.8712 | 213.14920043945312 | 0.9608017350146727 |
| 5 | 20000 | 0.905125 | 0.4999743402004242 | 0.9022725529793706 | 0.923060829541294 | 0.8824 | 0.4821454584598541 | 0.9677318333926658 | 0.905375 | 477.36669921875 | 0.9025055438024112 | 0.9205012218582644 | 0.8852 | 493.6698913574219 | 0.9681063663719243 | 0.90565 | 21.852725982666016 | 0.9027959303964531 | 0.9260291257031702 | 0.8807 | 22.224273681640625 | 0.9681537834478611 | 0.9035 | 237.94554138183594 | 0.9007219292406943 | 0.9228832231665093 | 0.8796 | 233.02957153320312 | 0.9664151462381492 |
| 5 | -1 | 0.908825 | 0.4167391061782837 | 0.9067954713895064 | 0.9274400125463955 | 0.88705 | 0.4167391061782837 | 0.9692830626530475 | 0.908575 | 511.60858154296875 | 0.9058531974144758 | 0.9261794054647092 | 0.8864 | 521.9729614257812 | 0.9691215287508383 | 0.9088 | 23.511920928955078 | 0.9063943343939237 | 0.9309508749736454 | 0.8831 | 23.511920928955078 | 0.9692657736763628 | 0.907875 | 195.81820678710938 | 0.9062697749765865 | 0.9177218434408161 | 0.8951 | 192.03176879882812 | 0.9688303836479663 |
| 6 | 10000 | 0.9117 | 0.43377184867858887 | 0.9091142688285324 | 0.9351377068245493 | 0.8845 | 0.42168402671813965 | 0.9705081136434329 | 0.911125 | 503.63323974609375 | 0.9086830163666956 | 0.9269738895246021 | 0.8911 | 515.934814453125 | 0.9705936795264274 | 0.911625 | 23.21420669555664 | 0.90933805237106 | 0.9334948133326313 | 0.8864 | 23.21420669555664 | 0.970644014417841 | 0.910625 | 190.39918518066406 | 0.9088312549409635 | 0.9274449591422474 | 0.89095 | 190.39918518066406 | 0.9698163020951304 |
| 6 | 20000 | 0.912575 | 0.4052755534648895 | 0.9098029112456524 | 0.938453361679511 | 0.88285 | 0.4011077880859375 | 0.9710678702761814 | 0.9119 | 513.6328125 | 0.9090537815555045 | 0.9285602544715024 | 0.89035 | 525.4749755859375 | 0.970994069548643 | 0.91225 | 23.212299346923828 | 0.9088935972301172 | 0.9404341781627633 | 0.8794 | 23.397891998291016 | 0.9711426496517335 | 0.911375 | 189.27462768554688 | 0.9083260657671984 | 0.9325819024544401 | 0.8853 | 182.22271728515625 | 0.9704334542723605 |
| 6 | -1 | 0.91105 | 0.38402271270751953 | 0.9082011127137852 | 0.9365703357416064 | 0.8815 | 0.3781573176383972 | 0.9708056816629487 | 0.9107 | 519.240966796875 | 0.9077167452346792 | 0.9308007566204287 | 0.88575 | 528.8313598632812 | 0.9708053595341734 | 0.910775 | 23.534488677978516 | 0.9077579997942176 | 0.9347849120576394 | 0.88225 | 23.839462280273438 | 0.9709448175722556 | 0.90935 | 175.73391723632812 | 0.9069991873222268 | 0.9216040462427746 | 0.89285 | 162.47686767578125 | 0.9701583328129889 |
Triplet Evaluation:
| epoch | steps | accuracy_cosinus | accuracy_manhattan | accuracy_euclidean |
|-------|-------|------------------|--------------------|--------------------|
| 0 | 10000 | 0.9344 | 0.9323 | 0.9322 |
| 0 | 20000 | 0.9279 | 0.9271 | 0.9271 |
| 0 | -1 | 0.9481 | 0.9466 | 0.9468 |
| 1 | 10000 | 0.9403 | 0.9378 | 0.9385 |
| 1 | 20000 | 0.9307 | 0.9306 | 0.9312 |
| 1 | -1 | 0.9364 | 0.9373 | 0.9369 |
| 2 | 10000 | 0.9235 | 0.9239 | 0.9242 |
| 2 | 20000 | 0.929 | 0.9287 | 0.928 |
| 2 | -1 | 0.9267 | 0.927 | 0.928 |
| 3 | 10000 | 0.9431 | 0.9422 | 0.9434 |
| 3 | 20000 | 0.9356 | 0.9376 | 0.9367 |
| 3 | -1 | 0.9484 | 0.9481 | 0.9473 |
| 4 | 10000 | 0.9347 | 0.935 | 0.9351 |
| 4 | 20000 | 0.9517 | 0.9511 | 0.9516 |
| 4 | -1 | 0.9465 | 0.9473 | 0.9469 |
| 5 | 10000 | 0.9521 | 0.9517 | 0.9521 |
| 5 | 20000 | 0.9615 | 0.9618 | 0.9615 |
| 5 | -1 | 0.9638 | 0.9639 | 0.9635 |
| 6 | 10000 | 0.9629 | 0.9644 | 0.9641 |
| 6 | 20000 | 0.9673 | 0.967 | 0.9665 |
| 6 | -1 | 0.9666 | 0.9658 | 0.9666 |
## Training
The model was trained with the parameters:
**DataLoader**:
`torch.utils.data.dataloader.DataLoader` of length 27693 with parameters:
```
{'batch_size': 12, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
```
**Loss**:
`sentence_transformers.losses.BatchHardTripletLoss.BatchHardTripletLoss`
Parameters of the fit()-Method:
```
{
"epochs": 7,
"evaluation_steps": 10000,
"evaluator": "sentence_transformers.evaluation.SequentialEvaluator.SequentialEvaluator",
"max_grad_norm": 1,
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
"optimizer_params": {
"lr": 2e-05
},
"scheduler": "WarmupLinear",
"steps_per_epoch": null,
"warmup_steps": 10000,
"weight_decay": 0.01
}
```
## Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)
```
## Citing & Authors
<!--- Describe where people can find more information --> |