LogiLlama

LogiLlama is a fine-tuned language model developed by Goppa AI. Built upon a 1B-parameter base from LLaMA, LogiLlama has been enhanced with injected knowledge and logical reasoning abilities. Our mission is to make smaller models smarter—delivering improved reasoning and problem-solving capabilities while maintaining a low memory footprint and energy efficiency for on-device applications.

Model Summary

While recent trends in language models have leaned towards scaling up parameters, LogiLlama demonstrates that “less can be more.” By fine-tuning a 1B parameter base model with advanced logical reasoning techniques, LogiLlama offers:

Enhanced Reasoning: Improved logical thinking and knowledge integration for more accurate and context-aware responses.
Efficiency: Designed for on-device processing with a low memory and energy footprint.
Transparency: Our training process and configuration files are fully open-source, reflecting our commitment to transparent and reproducible research.

LogiLlama is the first step in our journey at Goppa AI to develop efficient, intelligent, and resource-friendly models that challenge the notion that bigger is always better.

Model Description

Model Type: Small Language Model (SLM) fine-tuned from a 1B parameter LLaMA base
Architecture:
- Hidden Size: 2048
- Hidden Layers: 16
- Attention Heads: 32
- Intermediate Size: 8192
- Special Configuration: Incorporates a customized ROPE scaling (rope_type: "llama3")
Tokenization:
- Custom tokenizer with an extensive set of special tokens (defined in special_tokens_map.json and tokenizer_config.json)
Language: English
License: Llama 3 Community License Agreement

How to Use

Below is a sample code snippet demonstrating how to use LogiLlama with the Hugging Face Transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load tokenizer and model from our repository
tokenizer = AutoTokenizer.from_pretrained("goppa-ai/Goppa-LogiLlama", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("goppa-ai/Goppa-LogiLlama", trust_remote_code=True)

model.to('cuda')
text = "When faced with a complex problem, what should one do?"
input_ids = tokenizer(text, return_tensors="pt").to('cuda').input_ids
outputs = model.generate(
    input_ids,
    max_length=2500,
    temperature=0.6,
    top_p=0.9,
    repetition_penalty=1.2,
    pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.batch_decode(outputs[:, input_ids.shape[1]:-1])[0].strip())

Configuration Files

The model repository includes several key configuration files that ensure LogiLlama functions seamlessly within the Hugging Face ecosystem:

config.json: Contains the model architecture settings, including hidden size, number of layers, attention heads, and other hyperparameters.
generation_config.json: Defines generation parameters such as temperature, top-p sampling, and end-of-sequence tokens.
special_tokens_map.json: Maps special tokens (e.g., beginning-of-text, end-of-text, padding) used during tokenization.
tokenizer_config.json: Provides metadata and settings for the tokenizer, ensuring consistency with the model’s vocabulary and special tokens.

Training Details

LogiLlama was fine-tuned by injecting logical reasoning and domain-specific knowledge into a 1B parameter LLaMA base. By carefully curating training data and employing specialized techniques, we enhanced the model’s capability to handle reasoning tasks without significantly increasing its size. This project marks our commitment to advancing small, efficient models that do not compromise on performance.

Citation

If you use LogiLlama in your research, please cite:

@misc{goppa2025logillama,
      title={LogiLlama: Injecting Logical Reasoning into Small Language Models}, 
      author={Goppa AI},
      year={2025},
      note={https://github.com/GoppaAI/LogiLlama}
}

License

LogiLlama is released under the MIT License.

Inference & Deployment

Model Size: 1B parameters
Tensor Type: float32 (F32)
Deployment: Optimized for on-device inference and resource-constrained environments. Currently available for local deployment; stay tuned for updates on hosted inference solutions.

At Goppa AI, we are committed to pushing the boundaries of efficiency and intelligence in language models. LogiLlama is our first step towards creating small models that are not only resource-friendly but also smart enough to handle complex reasoning tasks.

Happy innovating!

goppa-ai
/

Goppa-LogiLlama