metadata
base_model: google/electra-large-discriminator
datasets:
- PiC/phrase_similarity
language:
- en
library_name: sentence-transformers
metrics:
- cosine_accuracy
- cosine_accuracy_threshold
- cosine_f1
- cosine_f1_threshold
- cosine_precision
- cosine_recall
- cosine_ap
- dot_accuracy
- dot_accuracy_threshold
- dot_f1
- dot_f1_threshold
- dot_precision
- dot_recall
- dot_ap
- manhattan_accuracy
- manhattan_accuracy_threshold
- manhattan_f1
- manhattan_f1_threshold
- manhattan_precision
- manhattan_recall
- manhattan_ap
- euclidean_accuracy
- euclidean_accuracy_threshold
- euclidean_f1
- euclidean_f1_threshold
- euclidean_precision
- euclidean_recall
- euclidean_ap
- max_accuracy
- max_accuracy_threshold
- max_f1
- max_f1_threshold
- max_precision
- max_recall
- max_ap
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:7004
- loss:SoftmaxLoss
widget:
- source_sentence: >-
Google SEO expert Matt Cutts had a similar experience, of the eight
magazines and newspapers Cutts tried to order, he received zero.
sentences:
- >-
He dissolved the services of her guards and her court attendants and
seized an expansive reach of properties belonging to her.
- >-
Google SEO expert Matt Cutts had a comparable occurrence, of the eight
magazines and newspapers Cutts tried to order, he received zero.
- >-
bill's newest solo play, "all over the map", premiered off broadway in
april 2016, produced by all for an individual cinema.
- source_sentence: >-
Shula said that Namath "beat our blitz" with his fast release, which let
him quickly dump the football off to a receiver.
sentences:
- >-
Shula said that Namath "beat our blitz" with his quick throw, which let
him quickly dump the football off to a receiver.
- >-
it elects a single component of parliament (mp) by the first past the
post system of election.
- >-
Matt Groening said that West was one of the most widely known group to
ever come to the studio.
- source_sentence: >-
When Angel calls out her name, Cordelia suddenly appears from the opposite
side of the room saying, "Yep, that chick's in rough shape.
sentences:
- >-
The ruined row of text, part of the Florida East Coast Railway, was
repaired by 2014 renewing freight train access to the port.
- >-
When Angel calls out her name, Cordelia suddenly appears from the
opposite side of the room saying, "Yep, that chick's in approximate
form.
- >-
Chaplin's films introduced a moderated kind of comedy than the typical
Keystone farce, and he developed a large fan base.
- source_sentence: >-
The following table shows the distances traversed by National Route 11 in
each different department, showing cities and towns that it passes by (or
near).
sentences:
- >-
The following table shows the distances traversed by National Route 11
in each separate city authority, showing cities and towns that it passes
by (or near).
- >-
Similarly, indigenous communities and leaders practice as the main rule
of law on local native lands and reserves.
- >-
later, sylvan mixed gary numan's albums "replicas" (with numan's
previous band tubeway army) and "the quest for instant gratification".
- source_sentence: She wants to write about Keima but suffers a major case of writer's block.
sentences:
- >-
In some countries, new extremist parties on the extreme opposite of left
of the political spectrum arose, motivated through issues of
immigration, multiculturalism and integration.
- >-
specific medical status of movement and the general condition of
movement both are conditions under which contradictions can move.
- >-
She wants to write about Keima but suffers a huge occurrence of writer's
block.
model-index:
- name: SentenceTransformer based on google/electra-large-discriminator
results:
- task:
type: binary-classification
name: Binary Classification
dataset:
name: quora duplicates dev
type: quora-duplicates-dev
metrics:
- type: cosine_accuracy
value: 0.748
name: Cosine Accuracy
- type: cosine_accuracy_threshold
value: 0.9737387895584106
name: Cosine Accuracy Threshold
- type: cosine_f1
value: 0.7604846225535881
name: Cosine F1
- type: cosine_f1_threshold
value: 0.9574624300003052
name: Cosine F1 Threshold
- type: cosine_precision
value: 0.7120418848167539
name: Cosine Precision
- type: cosine_recall
value: 0.816
name: Cosine Recall
- type: cosine_ap
value: 0.786909093121924
name: Cosine Ap
- type: dot_accuracy
value: 0.667
name: Dot Accuracy
- type: dot_accuracy_threshold
value: 275.4551696777344
name: Dot Accuracy Threshold
- type: dot_f1
value: 0.733229329173167
name: Dot F1
- type: dot_f1_threshold
value: 266.14727783203125
name: Dot F1 Threshold
- type: dot_precision
value: 0.6010230179028133
name: Dot Precision
- type: dot_recall
value: 0.94
name: Dot Recall
- type: dot_ap
value: 0.5935392159238977
name: Dot Ap
- type: manhattan_accuracy
value: 0.746
name: Manhattan Accuracy
- type: manhattan_accuracy_threshold
value: 87.73857116699219
name: Manhattan Accuracy Threshold
- type: manhattan_f1
value: 0.7614678899082568
name: Manhattan F1
- type: manhattan_f1_threshold
value: 131.43374633789062
name: Manhattan F1 Threshold
- type: manhattan_precision
value: 0.7033898305084746
name: Manhattan Precision
- type: manhattan_recall
value: 0.83
name: Manhattan Recall
- type: manhattan_ap
value: 0.7904964653279406
name: Manhattan Ap
- type: euclidean_accuracy
value: 0.747
name: Euclidean Accuracy
- type: euclidean_accuracy_threshold
value: 4.5833892822265625
name: Euclidean Accuracy Threshold
- type: euclidean_f1
value: 0.7610121836925962
name: Euclidean F1
- type: euclidean_f1_threshold
value: 5.5540361404418945
name: Euclidean F1 Threshold
- type: euclidean_precision
value: 0.7160493827160493
name: Euclidean Precision
- type: euclidean_recall
value: 0.812
name: Euclidean Recall
- type: euclidean_ap
value: 0.789806008641207
name: Euclidean Ap
- type: max_accuracy
value: 0.748
name: Max Accuracy
- type: max_accuracy_threshold
value: 275.4551696777344
name: Max Accuracy Threshold
- type: max_f1
value: 0.7614678899082568
name: Max F1
- type: max_f1_threshold
value: 266.14727783203125
name: Max F1 Threshold
- type: max_precision
value: 0.7160493827160493
name: Max Precision
- type: max_recall
value: 0.94
name: Max Recall
- type: max_ap
value: 0.7904964653279406
name: Max Ap
SentenceTransformer based on google/electra-large-discriminator
This is a sentence-transformers model finetuned from google/electra-large-discriminator on the PiC/phrase_similarity dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: google/electra-large-discriminator
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 1024 tokens
- Similarity Function: Cosine Similarity
- Training Dataset:
- Language: en
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ElectraModel
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("Deehan1866/Electra")
# Run inference
sentences = [
"She wants to write about Keima but suffers a major case of writer's block.",
"She wants to write about Keima but suffers a huge occurrence of writer's block.",
'specific medical status of movement and the general condition of movement both are conditions under which contradictions can move.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Evaluation
Metrics
Binary Classification
- Dataset:
quora-duplicates-dev
- Evaluated with
BinaryClassificationEvaluator
Metric | Value |
---|---|
cosine_accuracy | 0.748 |
cosine_accuracy_threshold | 0.9737 |
cosine_f1 | 0.7605 |
cosine_f1_threshold | 0.9575 |
cosine_precision | 0.712 |
cosine_recall | 0.816 |
cosine_ap | 0.7869 |
dot_accuracy | 0.667 |
dot_accuracy_threshold | 275.4552 |
dot_f1 | 0.7332 |
dot_f1_threshold | 266.1473 |
dot_precision | 0.601 |
dot_recall | 0.94 |
dot_ap | 0.5935 |
manhattan_accuracy | 0.746 |
manhattan_accuracy_threshold | 87.7386 |
manhattan_f1 | 0.7615 |
manhattan_f1_threshold | 131.4337 |
manhattan_precision | 0.7034 |
manhattan_recall | 0.83 |
manhattan_ap | 0.7905 |
euclidean_accuracy | 0.747 |
euclidean_accuracy_threshold | 4.5834 |
euclidean_f1 | 0.761 |
euclidean_f1_threshold | 5.554 |
euclidean_precision | 0.716 |
euclidean_recall | 0.812 |
euclidean_ap | 0.7898 |
max_accuracy | 0.748 |
max_accuracy_threshold | 275.4552 |
max_f1 | 0.7615 |
max_f1_threshold | 266.1473 |
max_precision | 0.716 |
max_recall | 0.94 |
max_ap | 0.7905 |
Training Details
Training Dataset
PiC/phrase_similarity
- Dataset: PiC/phrase_similarity at fc67ce7
- Size: 7,004 training samples
- Columns:
sentence1
,sentence2
, andlabel
- Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label type string string int details - min: 12 tokens
- mean: 26.35 tokens
- max: 57 tokens
- min: 12 tokens
- mean: 26.89 tokens
- max: 58 tokens
- 0: ~48.80%
- 1: ~51.20%
- Samples:
sentence1 sentence2 label newly formed camp is released from the membrane and diffuses across the intracellular space where it serves to activate pka.
recently made encampment is released from the membrane and diffuses across the intracellular space where it serves to activate pka.
0
According to one data, in 1910, on others – in 1915, the mansion became Natalya Dmitriyevna Shchuchkina's property.
According to a particular statistic, in 1910, on others – in 1915, the mansion became Natalya Dmitriyevna Shchuchkina's property.
1
Note that Fact 1 does not assume any particular structure on the set formula_65.
Note that Fact 1 does not assume any specific edifice on the set formula_65.
0
- Loss:
SoftmaxLoss
Evaluation Dataset
PiC/phrase_similarity
- Dataset: PiC/phrase_similarity at fc67ce7
- Size: 1,000 evaluation samples
- Columns:
sentence1
,sentence2
, andlabel
- Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label type string string int details - min: 9 tokens
- mean: 26.21 tokens
- max: 61 tokens
- min: 10 tokens
- mean: 26.8 tokens
- max: 61 tokens
- 0: ~50.00%
- 1: ~50.00%
- Samples:
sentence1 sentence2 label after theo's apparent death, she decides to leave first colony and ends up traveling with the apostles.
after theo's apparent death, she decides to leave original settlement and ends up traveling with the apostles.
0
The guard assigned to Vivian leaves her to prevent the robbery, allowing her to connect to the bank's network.
The guard assigned to Vivian leaves her to prevent the robbery, allowing her to connect to the bank's locations.
0
Two days later Louis XVI banished Necker by a "lettre de cachet" for his very public exchange of pamphlets.
Two days later Louis XVI banished Necker by a "lettre de cachet" for his very free forum of pamphlets.
0
- Loss:
SoftmaxLoss
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: stepsper_device_train_batch_size
: 16per_device_eval_batch_size
: 16learning_rate
: 2e-05num_train_epochs
: 5warmup_ratio
: 0.1load_best_model_at_end
: True
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: stepsprediction_loss_only
: Trueper_device_train_batch_size
: 16per_device_eval_batch_size
: 16per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 1eval_accumulation_steps
: Nonelearning_rate
: 2e-05weight_decay
: 0.0adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1.0num_train_epochs
: 5max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.1warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Falsefp16
: Falsefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Trueignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torchoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Falseresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Falsehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseeval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falseeval_on_start
: Falsebatch_sampler
: batch_samplermulti_dataset_batch_sampler
: proportional
Training Logs
Epoch | Step | Training Loss | loss | quora-duplicates-dev_max_ap |
---|---|---|---|---|
0 | 0 | - | - | 0.6721 |
0.2283 | 100 | - | 0.6805 | 0.6847 |
0.4566 | 200 | - | 0.5313 | 0.7905 |
0.6849 | 300 | - | 0.5383 | 0.7838 |
0.9132 | 400 | - | 0.6442 | 0.7585 |
1.1416 | 500 | 0.5761 | 0.5742 | 0.7843 |
1.3699 | 600 | - | 0.5606 | 0.7558 |
1.5982 | 700 | - | 0.5716 | 0.7772 |
1.8265 | 800 | - | 0.5573 | 0.7619 |
2.0548 | 900 | - | 0.6951 | 0.7760 |
2.2831 | 1000 | 0.3712 | 0.7678 | 0.7753 |
2.5114 | 1100 | - | 0.7712 | 0.7915 |
2.7397 | 1200 | - | 0.8120 | 0.7914 |
2.9680 | 1300 | - | 0.8045 | 0.7789 |
3.1963 | 1400 | - | 0.9936 | 0.7821 |
3.4247 | 1500 | 0.1942 | 1.0883 | 0.7679 |
3.6530 | 1600 | - | 0.9814 | 0.7566 |
3.8813 | 1700 | - | 1.0897 | 0.7830 |
4.1096 | 1800 | - | 1.0764 | 0.7729 |
4.3379 | 1900 | - | 1.1209 | 0.7802 |
4.5662 | 2000 | 0.1175 | 1.1522 | 0.7804 |
4.7945 | 2100 | - | 1.1545 | 0.7807 |
5.0 | 2190 | - | - | 0.7905 |
- The bold row denotes the saved checkpoint.
Framework Versions
- Python: 3.10.10
- Sentence Transformers: 3.0.1
- Transformers: 4.42.3
- PyTorch: 2.2.1+cu121
- Accelerate: 0.32.1
- Datasets: 2.20.0
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers and SoftmaxLoss
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}