bibtext: @inproceedings{
  lepori2024uncovering,
  title={Uncovering Intermediate Variables in Transformers using Circuit Probing},
  author={Michael A. Lepori and Thomas Serre and Ellie Pavlick},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=gUNeyiLNxr}
  }
abstract: Neural network models have achieved high performance on a wide variety
  of complex tasks, but the algorithms that they implement are notoriously
  difficult to interpret. It is often necessary to hypothesize intermediate variables involved in a network’s computation in order to understand these
  algorithms. For example, does a language model depend on particular
  syntactic properties when generating a sentence? Yet, existing analysis
  tools make it difficult to test hypotheses of this type. We propose a new
  analysis technique – circuit probing – that automatically uncovers low-level
  circuits that compute hypothesized intermediate variables. This enables
  causal analysis through targeted ablation at the level of model parameters.
  We apply this method to models trained on simple arithmetic tasks, demonstrating its effectiveness at (1) deciphering the algorithms that models have
  learned, (2) revealing modular structure within a model, and (3) tracking
  the development of circuits over training. Across these three experiments
  we demonstrate that circuit probing combines and extends the capabilities of existing methods, providing one unified approach for a variety of
  analyses. Finally, we demonstrate circuit probing on a real-world use case:
  uncovering circuits that are responsible for subject-verb agreement and
  reflexive anaphora in GPT2-Small and Medium.
title: Uncovering Intermediate Variables in Transformers using Circuit Probing
authors: Michael A. Lepori Thomas Serre Ellie Pavlick