These transcoders were trained on the outputs of the first 15 MLPs in deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. We used 10 billion tokens from FineWeb edu deduped at a context length of 2048. The number of latents is 65,536 and a linear skip connection is included.

Fraction of variance unexplained ranges from 0.01 to 0.37.

Downloads last month: 0

Inference API

Unable to determine this model's library. Check the docs .

Model tree for EleutherAI/skip-transcoder-DeepSeek-R1-Distill-Qwen-1.5B-65k

Base model

deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

Finetuned

(8)

this model

Dataset used to train EleutherAI/skip-transcoder-DeepSeek-R1-Distill-Qwen-1.5B-65k

Collection including EleutherAI/skip-transcoder-DeepSeek-R1-Distill-Qwen-1.5B-65k

Sparse Autoencoders

Collection

SAEs are tools for understanding the internal representations of neural networks. These can be loaded using https://github.com/EleutherAI/sae • 6 items • Updated about 19 hours ago