Shears Model Card: shears-mpt-7b-50-base

The sparsified MPT-7B with 50% sparsity as a base model in Shears.

Model Sources

• Repository: https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears
• Paper: Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

Citation

@article{munoz2024shears,
  title = {Shears: Unstructured Sparsity with Neural Low-rank Adapter Search},
  author={J. Pablo Munoz and Jinjie Yuan and Nilesh Jain},
  journal={The 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-2024)},
  year={2024}
}

Acknowledgement

Thanks to the work Wanda (paper, code), which provides a simple but effective pruning approach.

License

Apache-2.0