jadechoghari HF Staff commited on
Commit
94a2166
·
verified ·
1 Parent(s): 44a3910

Migrate policy to PolicyProcessorPipeline system

Browse files

**Automated Policy Migration to PolicyProcessorPipeline**

This PR migrates your model to the new LeRobot policy format using the modern PolicyProcessorPipeline architecture.

## What Changed

### **New Architecture - PolicyProcessorPipeline**
Your model now uses external PolicyProcessorPipeline components for data processing instead of built-in normalization layers. This provides:
- **Modularity**: Separate preprocessing and postprocessing pipelines
- **Flexibility**: Easy to swap, configure, and debug processing steps
- **Compatibility**: Works with the latest LeRobot ecosystem

### **Normalization Extraction**
We've extracted normalization statistics from your model's state_dict and removed the built-in normalization layers:
- **Extracted patterns**: `normalize_inputs.*`, `unnormalize_outputs.*`, `normalize.*`, `unnormalize.*`, `input_normalizer.*`, `output_normalizer.*`
- **Statistics preserved**: Mean, std, min, max values for all features
- **Clean model**: State dict now contains only core model weights

### **Files Added**
- **preprocessor_config.json**: Configuration for input preprocessing pipeline
- **postprocessor_config.json**: Configuration for output postprocessing pipeline
- **model.safetensors**: Clean model weights without normalization layers
- **config.json**: Updated model configuration
- **train_config.json**: Training configuration
- **README.md**: Updated model card with migration information

### **Benefits**
- **Backward Compatible**: Your model behavior remains identical
- **Future Ready**: Compatible with latest LeRobot features and updates
- **Debuggable**: Easy to inspect and modify processing steps
- **Portable**: Processors can be shared and reused across models

### **Usage**
```python
# Load your migrated model
from lerobot.policies import get_policy_class
from lerobot.processor import PolicyProcessorPipeline

# The preprocessor and postprocessor are now external
preprocessor = PolicyProcessorPipeline.from_pretrained("your-model-repo", config_filename="preprocessor_config.json")
postprocessor = PolicyProcessorPipeline.from_pretrained("your-model-repo", config_filename="postprocessor_config.json")
policy = get_policy_class("your-policy-type").from_pretrained("your-model-repo")

# Process data through the pipeline
processed_batch = preprocessor(raw_batch)
action = policy(processed_batch)
final_action = postprocessor(action)
```

*Generated automatically by the LeRobot policy migration script*

README.md ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ datasets: unknown
3
+ library_name: lerobot
4
+ license: apache-2.0
5
+ model_name: xvla
6
+ pipeline_tag: robotics
7
+ tags:
8
+ - xvla
9
+ - lerobot
10
+ - robotics
11
+ ---
12
+
13
+ # Model Card for xvla
14
+
15
+ <!-- Provide a quick summary of what the model is/does. -->
16
+
17
+
18
+ _Model type not recognized — please update this template._
19
+
20
+
21
+ This policy has been trained and pushed to the Hub using [LeRobot](https://github.com/huggingface/lerobot).
22
+ See the full documentation at [LeRobot Docs](https://huggingface.co/docs/lerobot/index).
23
+
24
+ ---
25
+
26
+ ## How to Get Started with the Model
27
+
28
+ For a complete walkthrough, see the [training guide](https://huggingface.co/docs/lerobot/il_robots#train-a-policy).
29
+ Below is the short version on how to train and run inference/eval:
30
+
31
+ ### Train from scratch
32
+
33
+ ```bash
34
+ lerobot-train \
35
+ --dataset.repo_id=${HF_USER}/<dataset> \
36
+ --policy.type=act \
37
+ --output_dir=outputs/train/<desired_policy_repo_id> \
38
+ --job_name=lerobot_training \
39
+ --policy.device=cuda \
40
+ --policy.repo_id=${HF_USER}/<desired_policy_repo_id>
41
+ --wandb.enable=true
42
+ ```
43
+
44
+ _Writes checkpoints to `outputs/train/<desired_policy_repo_id>/checkpoints/`._
45
+
46
+ ### Evaluate the policy/run inference
47
+
48
+ ```bash
49
+ lerobot-record \
50
+ --robot.type=so100_follower \
51
+ --dataset.repo_id=<hf_user>/eval_<dataset> \
52
+ --policy.path=<hf_user>/<desired_policy_repo_id> \
53
+ --episodes=10
54
+ ```
55
+
56
+ Prefix the dataset repo with **eval\_** and supply `--policy.path` pointing to a local or hub checkpoint.
57
+
58
+ ---
59
+
60
+ ## Model Details
61
+
62
+ - **License:** apache-2.0
config.json ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "type": "xvla",
3
+ "n_obs_steps": 1,
4
+ "input_features": {
5
+ "observation.images.image": {
6
+ "type": "VISUAL",
7
+ "shape": [
8
+ 3,
9
+ 256,
10
+ 256
11
+ ]
12
+ },
13
+ "observation.images.image2": {
14
+ "type": "VISUAL",
15
+ "shape": [
16
+ 3,
17
+ 256,
18
+ 256
19
+ ]
20
+ },
21
+ "observation.state": {
22
+ "type": "STATE",
23
+ "shape": [
24
+ 8
25
+ ]
26
+ },
27
+ "observation.images.empty_camera_0": {
28
+ "type": "VISUAL",
29
+ "shape": [
30
+ 3,
31
+ 224,
32
+ 224
33
+ ]
34
+ }
35
+ },
36
+ "output_features": {
37
+ "action": {
38
+ "type": "ACTION",
39
+ "shape": [
40
+ 20
41
+ ]
42
+ }
43
+ },
44
+ "device": "cuda",
45
+ "use_amp": false,
46
+ "push_to_hub": true,
47
+ "repo_id": null,
48
+ "private": null,
49
+ "tags": null,
50
+ "license": null,
51
+ "pretrained_path": null,
52
+ "chunk_size": 30,
53
+ "n_action_steps": 30,
54
+ "normalization_mapping": {
55
+ "STATE": "IDENTITY",
56
+ "ACTION": "IDENTITY",
57
+ "VISUAL": "MEAN_STD"
58
+ },
59
+ "florence_config": {
60
+ "model_type": "florence2",
61
+ "bos_token_id": 0,
62
+ "eos_token_id": 2,
63
+ "ignore_index": -100,
64
+ "pad_token_id": 1,
65
+ "projection_dim": 1024,
66
+ "text_config": {
67
+ "vocab_size": 51289,
68
+ "activation_dropout": 0.1,
69
+ "activation_function": "gelu",
70
+ "attention_dropout": 0.1,
71
+ "d_model": 1024,
72
+ "decoder_attention_heads": 16,
73
+ "decoder_layers": 12,
74
+ "encoder_attention_heads": 16,
75
+ "encoder_layers": 12,
76
+ "dropout": 0.1,
77
+ "max_position_embeddings": 4096,
78
+ "num_hidden_layers": 12,
79
+ "num_beams": 3
80
+ },
81
+ "vision_config": {
82
+ "model_type": "davit",
83
+ "drop_path_rate": 0.1,
84
+ "patch_size": [
85
+ 7,
86
+ 3,
87
+ 3,
88
+ 3
89
+ ],
90
+ "patch_stride": [
91
+ 4,
92
+ 2,
93
+ 2,
94
+ 2
95
+ ],
96
+ "patch_padding": [
97
+ 3,
98
+ 1,
99
+ 1,
100
+ 1
101
+ ],
102
+ "patch_prenorm": [
103
+ false,
104
+ true,
105
+ true,
106
+ true
107
+ ],
108
+ "enable_checkpoint": false,
109
+ "dim_embed": [
110
+ 256,
111
+ 512,
112
+ 1024,
113
+ 2048
114
+ ],
115
+ "num_heads": [
116
+ 8,
117
+ 16,
118
+ 32,
119
+ 64
120
+ ],
121
+ "num_groups": [
122
+ 8,
123
+ 16,
124
+ 32,
125
+ 64
126
+ ],
127
+ "depths": [
128
+ 1,
129
+ 1,
130
+ 9,
131
+ 1
132
+ ],
133
+ "window_size": 12,
134
+ "projection_dim": 1024,
135
+ "visual_temporal_embedding": {
136
+ "type": "COSINE",
137
+ "max_temporal_embeddings": 100
138
+ },
139
+ "image_pos_embed": {
140
+ "type": "learned_abs_2d",
141
+ "max_pos_embeddings": 50
142
+ },
143
+ "image_feature_source": [
144
+ "spatial_avg_pool",
145
+ "temporal_avg_pool"
146
+ ]
147
+ },
148
+ "vocab_size": 51289,
149
+ "torch_dtype": "float32",
150
+ "is_encoder_decoder": true
151
+ },
152
+ "tokenizer_name": "facebook/bart-large",
153
+ "tokenizer_max_length": 1024,
154
+ "tokenizer_padding_side": "right",
155
+ "pad_language_to": "max_length",
156
+ "hidden_size": 1024,
157
+ "depth": 24,
158
+ "num_heads": 16,
159
+ "mlp_ratio": 4.0,
160
+ "num_domains": 30,
161
+ "len_soft_prompts": 32,
162
+ "dim_time": 32,
163
+ "max_len_seq": 512,
164
+ "use_hetero_proj": false,
165
+ "action_mode": "ee6d",
166
+ "num_denoising_steps": 10,
167
+ "use_proprio": true,
168
+ "max_state_dim": 20,
169
+ "domain_feature_key": null,
170
+ "resize_imgs_with_padding": [
171
+ 224,
172
+ 224
173
+ ],
174
+ "num_image_views": 3,
175
+ "empty_cameras": 1,
176
+ "optimizer_lr": 0.0001,
177
+ "optimizer_betas": [
178
+ 0.9,
179
+ 0.95
180
+ ],
181
+ "optimizer_eps": 1e-08,
182
+ "optimizer_weight_decay": 0.0001,
183
+ "optimizer_grad_clip_norm": 10.0,
184
+ "scheduler_warmup_steps": 1000,
185
+ "scheduler_decay_steps": 30000,
186
+ "scheduler_decay_lr": 2.5e-06
187
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2b78f26411d0d4c6e688230d6c2df80b9d93a043f9bbaffbed4e99549cc4c5ab
3
+ size 3519073692
policy_postprocessor.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_postprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "unnormalizer_processor",
6
+ "config": {
7
+ "eps": 1e-08,
8
+ "features": {
9
+ "action": {
10
+ "type": "ACTION",
11
+ "shape": [
12
+ 20
13
+ ]
14
+ }
15
+ },
16
+ "norm_map": {
17
+ "STATE": "IDENTITY",
18
+ "ACTION": "IDENTITY",
19
+ "VISUAL": "MEAN_STD"
20
+ }
21
+ }
22
+ },
23
+ {
24
+ "registry_name": "device_processor",
25
+ "config": {
26
+ "device": "cpu",
27
+ "float_dtype": null
28
+ }
29
+ }
30
+ ]
31
+ }
policy_preprocessor.json ADDED
@@ -0,0 +1,82 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "policy_preprocessor",
3
+ "steps": [
4
+ {
5
+ "registry_name": "rename_observations_processor",
6
+ "config": {
7
+ "rename_map": {}
8
+ }
9
+ },
10
+ {
11
+ "registry_name": "to_batch_processor",
12
+ "config": {}
13
+ },
14
+ {
15
+ "registry_name": "tokenizer_processor",
16
+ "config": {
17
+ "max_length": 1024,
18
+ "task_key": "task",
19
+ "padding_side": "right",
20
+ "padding": "max_length",
21
+ "truncation": true,
22
+ "tokenizer_name": "facebook/bart-large"
23
+ }
24
+ },
25
+ {
26
+ "registry_name": "device_processor",
27
+ "config": {
28
+ "device": "cuda",
29
+ "float_dtype": null
30
+ }
31
+ },
32
+ {
33
+ "registry_name": "normalizer_processor",
34
+ "config": {
35
+ "eps": 1e-08,
36
+ "features": {
37
+ "observation.images.image": {
38
+ "type": "VISUAL",
39
+ "shape": [
40
+ 3,
41
+ 256,
42
+ 256
43
+ ]
44
+ },
45
+ "observation.images.image2": {
46
+ "type": "VISUAL",
47
+ "shape": [
48
+ 3,
49
+ 256,
50
+ 256
51
+ ]
52
+ },
53
+ "observation.state": {
54
+ "type": "STATE",
55
+ "shape": [
56
+ 8
57
+ ]
58
+ },
59
+ "observation.images.empty_camera_0": {
60
+ "type": "VISUAL",
61
+ "shape": [
62
+ 3,
63
+ 224,
64
+ 224
65
+ ]
66
+ },
67
+ "action": {
68
+ "type": "ACTION",
69
+ "shape": [
70
+ 20
71
+ ]
72
+ }
73
+ },
74
+ "norm_map": {
75
+ "STATE": "IDENTITY",
76
+ "ACTION": "IDENTITY",
77
+ "VISUAL": "MEAN_STD"
78
+ }
79
+ }
80
+ }
81
+ ]
82
+ }