README.md

metadata

license: apache-2.0
datasets:
  - togethercomputer/RedPajama-Data-1T-Sample
language:
  - en

Landmark Attention LLaMA 33B

This model has been trained using the PEFT LoRA technique with the Landmark Attention method over 200 steps. Model will likely be trained further and updated later on.

Usage

Requires trust_remote_code to be set to True. In oobabooga, you can simply add the --trust_remote_code flag.

You will also need to disable the Add the bos_token to the beginning of prompts option in the settings.

PEFT Checkpoint

You can probably merge the checkpoint with any other LLaMA-based model (provided they're 33B, of course). This repo contains the merged weights, but you can grab the adapter here.

Training Code

You can find the training code here.