Hugging Face's logo

mesolitica
/
malaysian-llama-3-8b-65k

Text Generation
Transformers
Safetensors
llama
text-generation-inference
Inference Endpoints
Model card Files Community

Malaysian Llama-3 8B 65536 context length

65536 context length and 15300000 RoPE Theta.

WanDB, https://wandb.ai/huseinzol05/EasyContext-65536?nw=nwuserhuseinzol05

Source code, https://github.com/mesolitica/malaya/tree/master/session/llama3#extend-1m-context-length

Special thanks to https://github.com/jzhang38/EasyContext for wrapping https://github.com/zhuzilin/ring-flash-attention for distributed training!

Downloads last month
11
Safetensors
Model size
8.03B params
Tensor type
BF16
·
Inference Providers NEW
Text Generation
This model is not currently available via any of the supported Inference Providers.

Model tree for mesolitica/malaysian-llama-3-8b-65k

Quantizations
1 model

Collection including mesolitica/malaysian-llama-3-8b-65k