File size: 4,797 Bytes
10d14ea b156091 10d14ea 6a1e424 10d14ea b156091 10d14ea | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
license: apache-2.0
---
# ReplaceMe: Training-Free Transformer Pruning via Layer Removal & Linear Transformations
[](https://arxiv.org/abs/2505.02819)
[](https://opensource.org/licenses/Apache-2.0)
## Model Description
ReplaceMe is a novel method for transformer model compression that enables **training-free** block/layer pruning while maintaining model performance through linear transformations. The approach:
- Identifies and removes block of layers
- Applies mathematically-derived transformations to preserve information flow
- Requires no fine-tuning or retraining
- Works with standard transformer architectures (The LTs are merged with the original model weights)
## Key Features
- 🚀 **Zero-Training Pruning**: Remove layers without any fine-tuning
- 🧠 **Performance Preservation**: <8% accuracy drop in most cases
- ⚡ **Instant Speedup**: less blocks -> faster inference + less memory
- 🔌 **Plug-and-Play**: Works with existing HuggingFace models
## 🔥 Performance Comparison of Pruning Methods (Llama 2 7B, 25% Compression)
| Method | Train-Free? | C3 | CMNLI | CHID (test) | WSC | HellaSwag | PIQA | Race-M | Race-H | MMLU | CMMLU | **AVG** | **RP** |
|------------------|-------------|------|-------|-------------|------|-----------|------|--------|--------|------|-------|---------|---------|
| **Llama 2 7B** (baseline) | | 43.8 | 33.0 | 41.6 | 37.5 | 71.3 | 78.1 | 33.1 | 35.5 | 46.8 | 31.8 | 45.3 | 100.0% |
| **LLM-Streamline*** | ❌ | 🏆 43.3 | 33.0 | 24.1 | 36.5 | 🏆 61.1 | 🏆 71.5 | 34.8 | 37.0 | 45.5 | 29.4 | 41.6 | 92.0% |
| **LLMPruner*** | ❌ | 29.7 | 33.4 | 28.4 | 40.4 | 54.6 | 72.0 | 22.9 | 22.0 | 25.3 | 25.0 | 35.4 | 78.2% |
| **SliceGPT*** | ❌ | 31.5 | 31.6 | 18.5 | 43.3 | 47.5 | 68.3 | 27.0 | 29.4 | 28.8 | 24.8 | 35.1 | 77.5% |
| **LaCo*** | ❌ | 39.7 | 🏆 34.4 | 🏆 36.1 | 40.4 | 55.7 | 69.8 | 23.6 | 22.6 | 26.5 | 25.2 | 37.4 | 82.7% |
| **UIDL*** | ❌ | 40.2 | 🏆 34.4 | 21.5 | 40.4 | 59.7 | 69.0 | 35.2 | 34.7 | 44.6 | 28.9 | 40.9 | 90.3% |
| | | | | | | | | | | | | | |
| **ReplaceMe (this model)** | ✅ | 42.5 | 33.0 | 25.2 | 38.5 | 59.4 | 71.1 | 35.4 | 🏆 36.7 | 🏆 46.4 | 🏆 30.4 | 🏆 **41.9** | 🏆 **92.5%** |
**Key:**
- 🏆 Best performance in column
- ✅ Training-free (our methods)
- ❌ Requires training
- *Numbers taken from Streamline paper
**Metrics Explained:**
- **RP**: Relative Performance (% of baseline)
- **Bold**: Best training-free results
- All numbers are accuracy scores
> 🔥 **Our training-free methods achieve 92.5% of baseline performance while other approaches require expensive retraining!**
## Installation
```bash
pip install replaceme
# or
git clone https://github.com/mts-ai/ReplaceMe
cd ReplaceMe
pip install -e .
```
## Basic Usage
```
# LSTSQ method (recommended)
run_replaceme --config ./reproduce/Replace_Me_pipeline_lstsq.yaml
# Cosine similarity method
run_replaceme --config ./reproduce/Replace_Me_pipeline_cosine.yaml
```
There are many parameters you can play with, visit our repo and dscover 🔥🔥
## Load Model
As we said we are merging the LTs with the original transformer architecture so you just do it as usual
```python
## EXAMPLE
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "MTSAIR/Llama2-5B-ReplaceMe"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "What is ReplaceME pruning method?!"
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
output = model.generate(
**model_inputs,
max_new_tokens=512
)
response = tokenizer.batch_decode(output, skip_special_tokens=True)[0]
```
# Citation
If you use ReplaceMe in your research, please cite our paper:
```bibtex
@article{shopkhoev2025replaceme0,
title = {ReplaceMe: Network Simplification via Layer Pruning and Linear Transformations},
author = {Dmitriy Shopkhoev and Ammar Ali and Magauiya Zhussip and Valentin Malykh and Stamatios Lefkimmiatis and Nikos Komodakis and Sergey Zagoruyko},
year = {2025},
journal = {arXiv preprint arXiv: 2505.02819}
}
``` |