Training an Sparse Autoencoder for Mechanistic Interpretability on PHI-3-mini-instruct with 1Billion Tokens
Dataset: mlfoundations/dclm-baseline-1.0
Hookpoint: blocks.16.hook_resid_post
Layer: 16
Trainingsteps 250_000
Batchsize: 4096
Context_size: 2048
ExpansionFaktor: 32
WandB Training Report: https://api.wandb.ai/links/kdt/h9edatb5
@misc{schacht2024sae4phi3,
title = {SAE for Phi-3 Mini Instruct Layer 16
author = {Sigurd Schacht},
year = {2024},
howpublished = {\url{}},
}}