---
language:
- en
license: apache-2.0
datasets:
- BEE-spoke-data/bees-internal
---

# BeeTokenizer

> note: this is **literally** a tokenizer trained on beekeeping text

After minutes of hard work, it is now available.


```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/BeeTokenizer")

test_string = "When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination."

output = tokenizer(test_string)
print(f"Test string: {test_string}")
print(f"Tokens:\n\t{output.input_ids}")
```


## Notes

- the default tokenizer (on branch `main`) has a vocab size of 32128



<details>
  <summary>How to Tokenize Text and Retrieve Offsets</summary>
  
  To tokenize a complex sentence and also retrieve the offsets mapping, you can use the following Python code snippet:

  ```python
  from transformers import AutoTokenizer

  # Initialize the tokenizer
  tokenizer = AutoTokenizer.from_pretrained("BEE-spoke-data/BeeTokenizer")

  # Sample complex sentence related to beekeeping
  test_string = "When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination."

  # Tokenize the input string and get the offsets mapping
  output = tokenizer.encode_plus(test_string, return_offsets_mapping=True)

  print(f"Test string: {test_string}")

  # Tokens
  tokens = tokenizer.convert_ids_to_tokens(output['input_ids'])
  print(f"Tokens: {tokens}")

  # Offsets
  offsets = output['offset_mapping']
  print(f"Offsets: {offsets}")
  ```

  This should result in the following (_Nov 2023 version_):
  
  <pre>&gt;&gt;&gt; print(f&quot;Test string: {test_string}&quot;)
  Test string: When dealing with Varroa destructor mites, it&apos;s crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
  &gt;&gt;&gt; 
  &gt;&gt;&gt; # Tokens
  &gt;&gt;&gt; tokens = tokenizer.convert_ids_to_tokens(output[&apos;input_ids&apos;])
  &gt;&gt;&gt; print(f&quot;Tokens: {tokens}&quot;)
  Tokens: [&apos;▁When&apos;, &apos;▁dealing&apos;, &apos;▁with&apos;, &apos;▁Varroa&apos;, &apos;▁destructor&apos;, &apos;▁mites,&apos;, &quot;▁it&apos;s&quot;, &apos;▁cru&apos;, &apos;cial&apos;, &apos;▁to&apos;, &apos;▁administer&apos;, &apos;▁the&apos;, &apos;▁right&apos;, &apos;▁acar&apos;, &apos;icides&apos;, &apos;▁during&apos;, &apos;▁the&apos;, &apos;▁late&apos;, &apos;▁autumn&apos;, &apos;▁months,&apos;, &apos;▁but&apos;, &apos;▁only&apos;, &apos;▁after&apos;, &apos;▁ensuring&apos;, &apos;▁that&apos;, &apos;▁the&apos;, &apos;▁worker&apos;, &apos;▁bee&apos;, &apos;▁population&apos;, &apos;▁is&apos;, &apos;▁free&apos;, &apos;▁from&apos;, &apos;▁pesticide&apos;, &apos;▁contamination&apos;, &apos;.&apos;]
  &gt;&gt;&gt; 
  &gt;&gt;&gt; # Offsets
  &gt;&gt;&gt; offsets = output[&apos;offset_mapping&apos;]
  &gt;&gt;&gt; print(f&quot;Offsets: {offsets}&quot;)
  Offsets: [(0, 4), (4, 12), (12, 17), (17, 24), (24, 35), (35, 42), (42, 47), (47, 51), (51, 55), (55, 58), (58, 69), (69, 73), (73, 79), (79, 84), (84, 90), (90, 97), (97, 101), (101, 106), (106, 113), (113, 121), (121, 125), (125, 130), (130, 136), (136, 145), (145, 150), (150, 154), (154, 161), (161, 165), (165, 176), (176, 179), (179, 184), (184, 189), (189, 199), (199, 213), (213, 214)]
  </pre>