kr-manish/bge-base-raw_pdf_finetuned_vf1

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kr-manish/bge-base-raw_pdf_finetuned_vf1")
# Run inference
sentences = [
    '~ " \'"-\'-en 25000 1 ,.,,µ,· ,, · .,-,.. •~h • 1 (1) ,\\ II J } 7; . \\ \\(9,i, .,u, 4\\:',
    'en 25000 I \' \'lJVL\' • -. • . .,.. ""~" \'\' \' I Q) l!J "667 7 ..._7 ... -,',
    '80, 85, or 95% identity to SEQ ID NO',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0
cosine_accuracy@3	0.0769
cosine_accuracy@5	0.0769
cosine_accuracy@10	0.2308
cosine_precision@1	0.0
cosine_precision@3	0.0256
cosine_precision@5	0.0154
cosine_precision@10	0.0231
cosine_recall@1	0.0
cosine_recall@3	0.0769
cosine_recall@5	0.0769
cosine_recall@10	0.2308
cosine_ndcg@10	0.1016
cosine_mrr@10	0.0623
cosine_map@100	0.0814

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0
cosine_accuracy@3	0.0769
cosine_accuracy@5	0.0769
cosine_accuracy@10	0.2308
cosine_precision@1	0.0
cosine_precision@3	0.0256
cosine_precision@5	0.0154
cosine_precision@10	0.0231
cosine_recall@1	0.0
cosine_recall@3	0.0769
cosine_recall@5	0.0769
cosine_recall@10	0.2308
cosine_ndcg@10	0.096
cosine_mrr@10	0.0566
cosine_map@100	0.0745

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0
cosine_accuracy@3	0.0769
cosine_accuracy@5	0.0769
cosine_accuracy@10	0.2308
cosine_precision@1	0.0
cosine_precision@3	0.0256
cosine_precision@5	0.0154
cosine_precision@10	0.0231
cosine_recall@1	0.0
cosine_recall@3	0.0769
cosine_recall@5	0.0769
cosine_recall@10	0.2308
cosine_ndcg@10	0.0982
cosine_mrr@10	0.059
cosine_map@100	0.0828

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0769
cosine_accuracy@3	0.2308
cosine_accuracy@5	0.2308
cosine_accuracy@10	0.3846
cosine_precision@1	0.0769
cosine_precision@3	0.0769
cosine_precision@5	0.0462
cosine_precision@10	0.0385
cosine_recall@1	0.0769
cosine_recall@3	0.2308
cosine_recall@5	0.2308
cosine_recall@10	0.3846
cosine_ndcg@10	0.2194
cosine_mrr@10	0.1701
cosine_map@100	0.1861

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0
cosine_accuracy@3	0.0769
cosine_accuracy@5	0.1538
cosine_accuracy@10	0.3077
cosine_precision@1	0.0
cosine_precision@3	0.0256
cosine_precision@5	0.0308
cosine_precision@10	0.0308
cosine_recall@1	0.0
cosine_recall@3	0.0769
cosine_recall@5	0.1538
cosine_recall@10	0.3077
cosine_ndcg@10	0.13
cosine_mrr@10	0.0763
cosine_map@100	0.1002

Training Details

Training Dataset

Unnamed Dataset

• Size: 111 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 2 tokens mean: 124.53 tokens max: 512 tokens	min: 3 tokens mean: 11.15 tokens max: 60 tokens

• Samples:

  positive	anchor
  ply C Tris pH8.0 Dextran Trehalose dNTPS Na2SO4 Triton X-100 DTT TABLE 3 GAS Lyophilization Mix -Reagent Composition vl.0 v2.0 Strep A (Target) Lyo Conditions 500 nM F30 500 nM F30b.5om 100 nM R41m 100 nM R41m.lb.5om 200 nM MB4 FAM 200 nM MB4_ Fam 3.0. ug 5.0 ug 30U 0.7 ug 1 ug 1 ug 50mM 50 mM Dextran 150 Dextran 500 5% in 2x Iyo 5% in 2x Iyo 100 mM in 2x Iyo 100 mM in 2x Iyo 0.3 mM 0.3 mM 15 mM 22.5 mM 0.10% 0.10% 2mM 2mM Strep A (IC) Lyo Conditions	NE
  CTGTTTG (SEQ ID NO, 5) To confirm that the targeted sequence was conserved among all GAS cepA sequences found in the public domain as well as unique to GAS, multiple sequence alignments and BLAST analyses were performed. Multiple alignment analysis of these sequences showed complete homology for the region of the gene targeted by the 3062 assay. Further, there are currently 24 complete GAS genomes (including whole genome shotgun sequence) available for sequence analysis in NCBI Genome. The cepA gene is present in all 24 genomes, and the 3062 target region within cepA is conserved among all 24 genomes. Upon BLAST analysis, it was confirmed that no other species contain significant homology to the 3062 target sequence. Assay Development As a reference, the reagent mixtures discussed below are	GCAATCTGAGGAGAGGCCATACTTGTTC
  AGATTGC (SEQ ID NO, 4)	CAAACAGGAACAAGTATGGCCTCTCCTC

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 32
• num_train_epochs: 15
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• fp16: True
• load_best_model_at_end: True
• optim: adamw_torch_fused

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 32
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 15
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0	0	-	0.0747	0.0694	0.0681	0.1224	0.0705
1.0	1	-	0.0750	0.0694	0.0681	0.1224	0.0705
2.0	2	-	0.1008	0.0724	0.0696	0.0719	0.0710
3.0	3	-	0.1861	0.0828	0.0745	0.1002	0.0814
4.0	4	-	0.1711	0.0968	0.0825	0.0861	0.1001
5.0	6	-	0.1505	0.1140	0.1094	0.1534	0.1502
6.0	7	-	0.1222	0.1143	0.1108	0.1528	0.1520
7.0	8	-	0.1589	0.1536	0.1512	0.1513	0.1516
8.0	9	-	0.1561	0.1550	0.1531	0.1495	0.1520
9.0	10	1.8482	0.1565	0.1558	0.1544	0.1483	0.1522
10.0	12	-	0.1562	0.1551	0.1557	0.1416	0.1531
11.0	13	-	0.1561	0.1558	0.1562	0.1401	0.1533
12.0	14	-	0.1559	0.1559	0.1562	0.1402	0.1533
13.0	15	-	0.1861	0.0828	0.0745	0.1002	0.0814

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.3.0+cu121
• Accelerate: 0.32.1
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}