File size: 2,787 Bytes
5691eac |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
---
license:
- other
language:
- en
pipeline_tag: text-generation
tags:
- pretrained
inference:
parameters:
temperature: 0.7
---
## Model sheet for AstraQuasar-4.5B-v.0.1
**AstraQuasar-4.5B-v.0.1** is our first pre-trained Large Language Model (LLM) for text generation. It is a model with **4.5B parameters**.
AstraQuasar-4.5B-v.0.1 is built upon the foundation of the Phi-2 architecture, with **significant enhancements including an increased number of layers and the innovative introduction of a novel technique known as the duplicate trick.**
AstraQuasar-4.5B-v.0.1 at the moment is an under trained model. Serving as a demonstration of the potential of the duplication trick and its implications for future advancements in language modeling. Despite its nascent status, our model has already demonstrated superior performance compared to both the base Phi-2 model and earlier iterations of AstraQuasar-4.5B that do not utilize the duplication trick.
One of the key milestones achieved by AstraQuasar-4.5B-v.0.1 is its successful application of backpropagation on the duplication trick, setting a precedent for future research and development in this area.
Our model's architecture is fully compatible with leading training frameworks such as Axolotl and LLaMA Factory, ensuring seamless integration into existing workflows leveraging the standard Hugging Face Transformers library.
## Example:
AstraQuasar-4.5B-v.0.1 can be easily instantiated using the Hugging Face Transformers library:
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AstraMindAI/AstraQuasar-4.5B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("AstraMindAI/AstraQuasar-4.5B")
# you can optionally disable the duplicate trick
# model.model.duplicate_trick = False
# You can also disable the duplicate gradient calculation
# model.model.duplicate_grad = False
# You can specify the layer ranges for the duplicate trick
# model.model.layer_ranges = [(0, 16),(8, 24),(17, 32),(25, 40),(33, 49),(40, 56)]
prompt = "This is an example script ."
inputs = tokenizer(prompt, return_tensors="pt")
# Generate
generate_ids = model.generate(inputs.input_ids, max_length=30)
tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
Pre-training and fine-tuning can be performed using **accelerate**.
## Notice
It's important to note that AstraQuasar-4.5B-v.0.1 is a pre-trained base model and does not incorporate any moderation mechanisms.
## NEWS
Stay tuned for exciting developments! A new architecture, **AstraPulsar**, is on the horizon, promising further advancements in language modeling.
|