Planck-OpenLAiNN-10M 🤗

Hey there fellow researchers, developers, and AI enthusiasts! Today I'm releasing a new family of Models, Planck LAiNN, These are probably some of the smallest LLMs that are on HF. They aren't super useful but it was a fun expierment!~

These are the GGUF quants of the models. For the original models, you can find them here.

Models Overview

• Panck-OpenLAiNN-10M: A Truely Tiny model with just 10 Million parameters, this is probably boarderline useless, but it IS functional.
• Panck-OpenLAiNN-25M: The second smallest model, 25 million parameters, it's not that much better.
• Panck-OpenLAiNN-50M: Surprisingly smart, it's 50 Million parameters and could potentially maybe, Possibly even be useful ;)
• Panck-OpenLAiNN-75M: The current ""heavy"" weight of the Plank-OpenLAiNN Models.

Pretraining Details

Plank-OpenLAiNN was trained on 32B tokens of the Fineweb dataset, it's the same one that was used for the Pico-LAiNN family of models. The model was pretrained with a context length of 1024 tokens.

Other information:

• Compatibility: Built to be compatible with existing projects that use LLAMA 2's tokenizer and architecture.
• Ease of Use: No need to reinvent the wheel. These models are ready to be plugged into your applications.
• Open Source: Fully open source, so you can tweak, tune, and twist them to your heart's content.

Getting Started

To start using these models, you can simply load them via the Hugging Face transformers library:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_NAME = "UUFO-Aigis/Panck-OpenLAiNN-10M"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)

def generate_text(prompt, model, tokenizer, max_length=512, temperature=1, top_k=50, top_p=0.95):
    inputs = tokenizer.encode(prompt, return_tensors="pt")

    outputs = model.generate(
        inputs,
        max_length=max_length,
        temperature=temperature,
        top_k=top_k,
        top_p=top_p,
        do_sample=True
    )


    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated_text

def main():
    # Define your prompt
    prompt = "According to all known laws of aviation, there is no way a bee should be able to fly."

    generated_text = generate_text(prompt, model, tokenizer)

    print(generated_text)

if __name__ == "__main__":
    main()

Benchy

Tasks	Value		Stderr
arc_challenge	0.1766	±	0.0111
arc_easy	0.3144	±	0.0095
boolq	0.5847	±	0.0086
hellaswag	0.2622	±	0.0044
lambada_openai	0.0047	±	0.0009
piqa	0.5718	±	0.0115
winogrande	0.4957	±	0.0141

Future Plans

• More Models: I'm currenetly training the bigger siblings of Pico-OpenLAiNN, including a 1B parameter version and beyond. 2-4 Billion parameter versions are planned. These will be Released as OpenLAiNN.
• New architecture: This is still up in the air and I'm still developing it, things are going well and I'll post updates.
• Paper: A detailed paper or training data will be posted at some point.

Credit Where Credit's Due

If you find these models useful and decide to use these models, a link to this repository would be highly appreciated. I am a one man show running this and I'm doing this for free, Thanks 🤗

Contact

If you have questions, Please reach out to me at [email protected]