khoj-ai/timely-arctic-large

Technical Report and Model Pipeline

To access our technical report and model pipeline scripts visit our github

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Snowflake/snowflake-arctic-embed-l
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Represent this sentence for searching relevant passages: Nov 6 2002 Easter seals (philately)',
    '06/11/2002 An Easter seal is a form of charity label issued to raise funds for charitable purposes. They are issued by the Easterseals charity in the United States, and by the Canadian Easter Seals charities. Easter seals are applied to the front of mail to show support for particular charitable causes. They are distributed along with appeals to donate to the charities they support. Easter seals are a form of Cinderella stamp. They do not have any postal value. Cinderella stamps\n',
    '2017 Winter The Waterfall Model was the first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model. ... The Waterfall model is the earliest SDLC approach that was used for software development. The waterfall Model illustrates the software development process in a linear sequential flow.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 55,736 training samples
• Columns: anchors and positive
• Approximate statistics based on the first 1000 samples:

  	anchors	positive
  type	string	string
  details	min: 14 tokens mean: 20.25 tokens max: 33 tokens	min: 15 tokens mean: 47.2 tokens max: 75 tokens

• Samples:

  anchors	positive
  Represent this sentence for searching relevant passages: are bugs attracted to citronella November 10?	Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10
  Represent this sentence for searching relevant passages: are bugs attracted to citronella 11/10/09?	Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10/09
  Represent this sentence for searching relevant passages: are bugs attracted to citronella Jan 15?	Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 01/15

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,000 evaluation samples
• Columns: anchors and positive
• Approximate statistics based on the first 1000 samples:

  	anchors	positive
  type	string	string
  details	min: 12 tokens mean: 21.64 tokens max: 40 tokens	min: 20 tokens mean: 66.86 tokens max: 512 tokens

• Samples:

  anchors	positive
  Represent this sentence for searching relevant passages: today:2068-02-10 what is the meaning of the idiom put two and two together last monday	put two and two together. to understand something by using the information you have: I didn't tell her George had left, but she noticed his car was gone and put two and two together. (Definition of put two and two together from the Cambridge Academic Content Dictionary © Cambridge University Press) 02/06/2068
  Represent this sentence for searching relevant passages: Complete the Throat of the World Quest in Skyrim	The Throat of the World is the fifth quest in the second act of the Skyrim’s main quest. During this mission, all the mystery about the game’s main antagonist, Alduin, will be revealed to you.
  Represent this sentence for searching relevant passages: are blanco kitchen faucets good 04/13/86?	Nevertheless, these are good to very good faucets built with good quality components throughout, backed by a strong warranty and superior customer service from a well-established company. Blanco sells only kitchen, prep and bar faucets, nothing for the bathroom. Apr 13 1986

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 1e-06
• weight_decay: 0.01
• num_train_epochs: 1
• warmup_ratio: 0.1
• warmup_steps: 400
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-06
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 400
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	loss
0.0006	1	1.9721	-
0.0057	10	1.9663	-
0.0115	20	1.947	-
0.0172	30	1.9039	-
0.0230	40	1.9672	-
0.0287	50	1.894	-
0.0344	60	1.8953	-
0.0402	70	1.9001	-
0.0459	80	1.8511	-
0.0517	90	1.7816	-
0.0574	100	1.7657	-
0.0631	110	1.6932	-
0.0689	120	1.6445	-
0.0746	130	1.6565	-
0.0804	140	1.5077	-
0.0861	150	1.4675	-
0.0918	160	1.4307	-
0.0976	170	1.2343	-
0.1033	180	1.1075	-
0.1091	190	1.1142	-
0.1148	200	1.0546	0.0897
0.1206	210	0.9872	-
0.1263	220	0.8933	-
0.1320	230	0.8066	-
0.1378	240	0.7317	-
0.1435	250	0.7404	-
0.1493	260	0.6348	-
0.1550	270	0.6399	-
0.1607	280	0.549	-
0.1665	290	0.4844	-
0.1722	300	0.5109	-
0.1780	310	0.4412	-
0.1837	320	0.4451	-
0.1894	330	0.373	-
0.1952	340	0.4318	-
0.2009	350	0.3996	-
0.2067	360	0.3534	-
0.2124	370	0.3795	-
0.2181	380	0.3195	-
0.2239	390	0.313	-
0.2296	400	0.3174	0.1864
0.2354	410	0.3255	-
0.2411	420	0.3172	-
0.2468	430	0.2601	-
0.2526	440	0.2862	-
0.2583	450	0.3042	-
0.2641	460	0.305	-
0.2698	470	0.2722	-
0.2755	480	0.2684	-
0.2813	490	0.2114	-
0.2870	500	0.2599	-
0.2928	510	0.2226	-
0.2985	520	0.213	-
0.3042	530	0.1968	-
0.3100	540	0.2005	-
0.3157	550	0.17	-
0.3215	560	0.2275	-
0.3272	570	0.1482	-
0.3330	580	0.1404	-
0.3387	590	0.1743	-
0.3444	600	0.1887	0.2803
0.3502	610	0.2018	-
0.3559	620	0.18	-
0.3617	630	0.146	-
0.3674	640	0.1308	-
0.3731	650	0.159	-
0.3789	660	0.1528	-
0.3846	670	0.1439	-
0.3904	680	0.1376	-
0.3961	690	0.1451	-
0.4018	700	0.1408	-
0.4076	710	0.1571	-
0.4133	720	0.1318	-
0.4191	730	0.1548	-
0.4248	740	0.1131	-
0.4305	750	0.1171	-
0.4363	760	0.1246	-
0.4420	770	0.1204	-
0.4478	780	0.1418	-
0.4535	790	0.0907	-
0.4592	800	0.1013	0.3217
0.4650	810	0.1067	-
0.4707	820	0.1064	-
0.4765	830	0.1089	-
0.4822	840	0.1044	-
0.4879	850	0.0916	-
0.4937	860	0.1344	-
0.4994	870	0.1377	-
0.5052	880	0.1078	-
0.5109	890	0.0837	-
0.5166	900	0.0893	-
0.5224	910	0.4395	-
0.5281	920	0.6783	-
0.5339	930	0.6341	-
0.5396	940	0.5763	-
0.5454	950	0.5283	-
0.5511	960	0.4955	-
0.5568	970	0.5138	-
0.5626	980	0.4983	-
0.5683	990	0.5239	-
0.5741	1000	0.5368	0.1056
0.5798	1010	0.5011	-
0.5855	1020	0.5244	-
0.5913	1030	0.39	-
0.5970	1040	0.4645	-
0.6028	1050	0.4164	-
0.6085	1060	0.4698	-
0.6142	1070	0.4074	-
0.6200	1080	0.4608	-
0.6257	1090	0.5081	-
0.6315	1100	0.4749	-
0.6372	1110	0.4384	-
0.6429	1120	0.3604	-
0.6487	1130	0.3853	-
0.6544	1140	0.3238	-
0.6602	1150	0.3656	-
0.6659	1160	0.2918	-
0.6716	1170	0.3919	-
0.6774	1180	0.3366	-
0.6831	1190	0.3731	-
0.6889	1200	0.4923	0.0583
0.6946	1210	0.3101	-
0.7003	1220	0.3177	-
0.7061	1230	0.3779	-
0.7118	1240	0.3342	-
0.7176	1250	0.2819	-
0.7233	1260	0.3247	-
0.7290	1270	0.4053	-
0.7348	1280	0.3277	-
0.7405	1290	0.3325	-
0.7463	1300	0.3827	-
0.7520	1310	0.2674	-
0.7577	1320	0.309	-
0.7635	1330	0.3193	-
0.7692	1340	0.3399	-
0.7750	1350	0.4044	-
0.7807	1360	0.3436	-
0.7865	1370	0.851	-
0.7922	1380	0.9553	-
0.7979	1390	0.8694	-
0.8037	1400	0.8736	0.0333
0.8094	1410	0.7984	-
0.8152	1420	0.8228	-
0.8209	1430	0.8026	-
0.8266	1440	0.8568	-
0.8324	1450	0.8529	-
0.8381	1460	0.757	-
0.8439	1470	0.779	-
0.8496	1480	0.8002	-
0.8553	1490	0.8532	-
0.8611	1500	0.7195	-
0.8668	1510	0.7598	-
0.8726	1520	0.8295	-
0.8783	1530	0.7588	-
0.8840	1540	0.7698	-
0.8898	1550	0.792	-
0.8955	1560	0.8175	-
0.9013	1570	0.7195	-
0.9070	1580	0.7383	-
0.9127	1590	0.4577	-
0.9185	1600	0.0621	0.0207
0.9242	1610	0.0644	-
0.9300	1620	0.0578	-
0.9357	1630	0.0368	-
0.9414	1640	0.056	-
0.9472	1650	0.059	-
0.9529	1660	0.0442	-
0.9587	1670	0.0527	-
0.9644	1680	0.0651	-
0.9701	1690	0.0515	-
0.9759	1700	0.0512	-
0.9816	1710	0.0543	-
0.9874	1720	0.0676	-
0.9931	1730	0.0593	-
0.9989	1740	0.0558	-

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.43.4
• PyTorch: 2.4.0+cu121
• Accelerate: 0.33.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}