Hugging Face's logo

hZzy
/
qwen2.5-0.5b-expo-DPO-ES-0.1

Safetensors
qwen2
alignment-handbook
ndcg
trl
expo
Generated from Trainer
Model card Files
xet
Community
hZzy commited on
Commit
88a4513
·
verified ·
1 Parent(s): 7801574

Model save

Browse files
Files changed (6) hide show
  1. README.md +85 -0
  2. all_results.json +9 -0
  3. generation_config.json +7 -0
  4. model.safetensors +1 -1
  5. train_results.json +9 -0
  6. trainer_state.json +430 -0
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: hZzy/qwen2.5-0.5b-sft-news-IFT
4
+ tags:
5
+ - trl
6
+ - expo
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: qwen2.5-0.5b-expo-DPO-ES-0.1
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/zhiyuzha-university-of-florida/huggingface/runs/w0nbtpl2)
17
+ # qwen2.5-0.5b-expo-DPO-ES-0.1
18
+
19
+ This model is a fine-tuned version of [hZzy/qwen2.5-0.5b-sft-news-IFT](https://huggingface.co/hZzy/qwen2.5-0.5b-sft-news-IFT) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 0.7091
22
+ - Logps: -107.9610
23
+ - Logits: -1.9041
24
+ - Objective: 0.7134
25
+ - Dpo Loss: 0.7134
26
+ - Regularize: 0.7134
27
+ - Ranking Simple: 0.5616
28
+ - Ranking Idealized: 0.6030
29
+ - Ranking Idealized Expo: 0.5223
30
+ - Wo Beta: 8.1968
31
+
32
+ ## Model description
33
+
34
+ More information needed
35
+
36
+ ## Intended uses & limitations
37
+
38
+ More information needed
39
+
40
+ ## Training and evaluation data
41
+
42
+ More information needed
43
+
44
+ ## Training procedure
45
+
46
+ ### Training hyperparameters
47
+
48
+ The following hyperparameters were used during training:
49
+ - learning_rate: 5e-06
50
+ - train_batch_size: 4
51
+ - eval_batch_size: 4
52
+ - seed: 42
53
+ - distributed_type: multi-GPU
54
+ - num_devices: 3
55
+ - gradient_accumulation_steps: 12
56
+ - total_train_batch_size: 144
57
+ - total_eval_batch_size: 12
58
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
59
+ - lr_scheduler_type: cosine
60
+ - lr_scheduler_warmup_ratio: 0.1
61
+ - num_epochs: 5
62
+
63
+ ### Training results
64
+
65
+ | Training Loss | Epoch | Step | Validation Loss | Logps | Logits | Objective | Dpo Loss | Regularize | Ranking Simple | Ranking Idealized | Ranking Idealized Expo | Wo Beta |
66
+ |:-------------:|:------:|:----:|:---------------:|:---------:|:-------:|:---------:|:--------:|:----------:|:--------------:|:-----------------:|:----------------------:|:-------:|
67
+ | 0.6785 | 0.1417 | 50 | 0.6814 | -90.8721 | -1.6022 | 0.6843 | 0.6843 | 0.6843 | 0.5259 | 0.6030 | 0.5223 | 7.8749 |
68
+ | 0.618 | 0.2834 | 100 | 0.6733 | -98.8900 | -1.7799 | 0.6766 | 0.6766 | 0.6766 | 0.5399 | 0.6030 | 0.5223 | 7.7840 |
69
+ | 0.5667 | 0.4251 | 150 | 0.6867 | -99.1217 | -1.8072 | 0.6829 | 0.6829 | 0.6829 | 0.5409 | 0.6030 | 0.5223 | 7.8537 |
70
+ | 0.5214 | 0.5668 | 200 | 0.6902 | -99.5153 | -1.8895 | 0.6905 | 0.6905 | 0.6905 | 0.5445 | 0.6030 | 0.5223 | 7.7013 |
71
+ | 0.4922 | 0.7085 | 250 | 0.6976 | -82.8384 | -1.9887 | 0.6914 | 0.6914 | 0.6914 | 0.5481 | 0.6030 | 0.5223 | 7.8784 |
72
+ | 0.4535 | 0.8503 | 300 | 0.6923 | -90.9491 | -2.1209 | 0.6894 | 0.6894 | 0.6894 | 0.5564 | 0.6030 | 0.5223 | 7.4232 |
73
+ | 0.4228 | 0.9920 | 350 | 0.7064 | -87.7231 | -1.9803 | 0.6968 | 0.6968 | 0.6968 | 0.5538 | 0.6030 | 0.5223 | 8.0253 |
74
+ | 0.2845 | 1.1337 | 400 | 0.7305 | -101.3180 | -2.0805 | 0.7269 | 0.7269 | 0.7269 | 0.5430 | 0.6030 | 0.5223 | 8.6164 |
75
+ | 0.2989 | 1.2754 | 450 | 0.7005 | -93.1955 | -1.8646 | 0.6974 | 0.6974 | 0.6974 | 0.5606 | 0.6030 | 0.5223 | 8.2386 |
76
+ | 0.3065 | 1.4171 | 500 | 0.7179 | -97.0137 | -1.9983 | 0.7147 | 0.7147 | 0.7147 | 0.5549 | 0.6030 | 0.5223 | 8.2760 |
77
+ | 0.2885 | 1.5588 | 550 | 0.7091 | -107.9610 | -1.9041 | 0.7134 | 0.7134 | 0.7134 | 0.5616 | 0.6030 | 0.5223 | 8.1968 |
78
+
79
+
80
+ ### Framework versions
81
+
82
+ - Transformers 4.42.0
83
+ - Pytorch 2.3.0+cu121
84
+ - Datasets 2.19.1
85
+ - Tokenizers 0.19.1
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.5588096362777515,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.4483427975394509,
5
+ "train_runtime": 15087.4424,
6
+ "train_samples": 50802,
7
+ "train_samples_per_second": 16.836,
8
+ "train_steps_per_second": 0.117
9
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151644,
3
+ "eos_token_id": 151645,
4
+ "max_new_tokens": 2048,
5
+ "pad_token_id": 151645,
6
+ "transformers_version": "4.42.0"
7
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:ae659fb770ce9ce3c03fa9cb0f6aaf85eeb8b08302628844835afd3546778dcc
3
  size 1975192208
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1120a5c9f7c063c0bc98de4175c28350750ec77097b46d0375ea9edf93d2cf95
3
  size 1975192208
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.5588096362777515,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.4483427975394509,
5
+ "train_runtime": 15087.4424,
6
+ "train_samples": 50802,
7
+ "train_samples_per_second": 16.836,
8
+ "train_steps_per_second": 0.117
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,430 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 7.4231791496276855,
3
+ "best_model_checkpoint": "./qwen2.5-0.5b/qwen2.5-0.5b-expo-DPO-ES-0.1/checkpoint-300",
4
+ "epoch": 1.5588096362777515,
5
+ "eval_steps": 50,
6
+ "global_step": 550,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "dpo_loss": 0.6931471824645996,
13
+ "epoch": 0.002834199338686821,
14
+ "grad_norm": 18.44253347826331,
15
+ "learning_rate": 2.840909090909091e-08,
16
+ "logits": -1.359458565711975,
17
+ "logps": -84.69721221923828,
18
+ "loss": 0.6931,
19
+ "objective": 0.6931471824645996,
20
+ "ranking_idealized": 0.6458333134651184,
21
+ "ranking_idealized_expo": 0.5833333134651184,
22
+ "ranking_simple": 0.5833333134651184,
23
+ "regularize": 0.6931471824645996,
24
+ "step": 1,
25
+ "wo_beta": 5.271125316619873
26
+ },
27
+ {
28
+ "dpo_loss": 0.6822353601455688,
29
+ "epoch": 0.14170996693434104,
30
+ "grad_norm": 18.641365531241362,
31
+ "learning_rate": 1.4204545454545458e-06,
32
+ "logits": -1.508646011352539,
33
+ "logps": -82.822021484375,
34
+ "loss": 0.6785,
35
+ "objective": 0.6822353601455688,
36
+ "ranking_idealized": 0.608418345451355,
37
+ "ranking_idealized_expo": 0.5229591727256775,
38
+ "ranking_simple": 0.5267857313156128,
39
+ "regularize": 0.6822353601455688,
40
+ "step": 50,
41
+ "wo_beta": 7.122643947601318
42
+ },
43
+ {
44
+ "epoch": 0.14170996693434104,
45
+ "eval_dpo_loss": 0.6842507719993591,
46
+ "eval_logits": -1.6022367477416992,
47
+ "eval_logps": -90.87205505371094,
48
+ "eval_loss": 0.6813501715660095,
49
+ "eval_objective": 0.6842507719993591,
50
+ "eval_ranking_idealized": 0.6030020713806152,
51
+ "eval_ranking_idealized_expo": 0.5222567319869995,
52
+ "eval_ranking_simple": 0.5258799195289612,
53
+ "eval_regularize": 0.6842507719993591,
54
+ "eval_runtime": 308.2905,
55
+ "eval_samples_per_second": 18.781,
56
+ "eval_steps_per_second": 1.567,
57
+ "eval_wo_beta": 7.874889850616455,
58
+ "step": 50
59
+ },
60
+ {
61
+ "dpo_loss": 0.61030513048172,
62
+ "epoch": 0.2834199338686821,
63
+ "grad_norm": 19.29211957833529,
64
+ "learning_rate": 2.8409090909090916e-06,
65
+ "logits": -1.6967747211456299,
66
+ "logps": -88.03068542480469,
67
+ "loss": 0.618,
68
+ "objective": 0.61030513048172,
69
+ "ranking_idealized": 0.6016666889190674,
70
+ "ranking_idealized_expo": 0.5141666531562805,
71
+ "ranking_simple": 0.5729166865348816,
72
+ "regularize": 0.61030513048172,
73
+ "step": 100,
74
+ "wo_beta": 6.357723236083984
75
+ },
76
+ {
77
+ "epoch": 0.2834199338686821,
78
+ "eval_dpo_loss": 0.6765881776809692,
79
+ "eval_logits": -1.7799152135849,
80
+ "eval_logps": -98.88995361328125,
81
+ "eval_loss": 0.6732921004295349,
82
+ "eval_objective": 0.6765881776809692,
83
+ "eval_ranking_idealized": 0.6030020713806152,
84
+ "eval_ranking_idealized_expo": 0.5222567319869995,
85
+ "eval_ranking_simple": 0.5398550629615784,
86
+ "eval_regularize": 0.6765881776809692,
87
+ "eval_runtime": 307.7771,
88
+ "eval_samples_per_second": 18.812,
89
+ "eval_steps_per_second": 1.569,
90
+ "eval_wo_beta": 7.7840118408203125,
91
+ "step": 100
92
+ },
93
+ {
94
+ "dpo_loss": 0.5696364045143127,
95
+ "epoch": 0.42512990080302315,
96
+ "grad_norm": 18.142654654279198,
97
+ "learning_rate": 4.2613636363636365e-06,
98
+ "logits": -1.7447518110275269,
99
+ "logps": -90.71894073486328,
100
+ "loss": 0.5667,
101
+ "objective": 0.5696364045143127,
102
+ "ranking_idealized": 0.6066666841506958,
103
+ "ranking_idealized_expo": 0.5287500023841858,
104
+ "ranking_simple": 0.6016666889190674,
105
+ "regularize": 0.5696364045143127,
106
+ "step": 150,
107
+ "wo_beta": 5.971243858337402
108
+ },
109
+ {
110
+ "epoch": 0.42512990080302315,
111
+ "eval_dpo_loss": 0.6829443573951721,
112
+ "eval_logits": -1.807220697402954,
113
+ "eval_logps": -99.12174987792969,
114
+ "eval_loss": 0.686660647392273,
115
+ "eval_objective": 0.6829443573951721,
116
+ "eval_ranking_idealized": 0.6030020713806152,
117
+ "eval_ranking_idealized_expo": 0.5222567319869995,
118
+ "eval_ranking_simple": 0.5408902764320374,
119
+ "eval_regularize": 0.6829443573951721,
120
+ "eval_runtime": 308.0058,
121
+ "eval_samples_per_second": 18.798,
122
+ "eval_steps_per_second": 1.568,
123
+ "eval_wo_beta": 7.853672981262207,
124
+ "step": 150
125
+ },
126
+ {
127
+ "dpo_loss": 0.5278546214103699,
128
+ "epoch": 0.5668398677373642,
129
+ "grad_norm": 15.355172339669656,
130
+ "learning_rate": 4.997168347957521e-06,
131
+ "logits": -1.9084746837615967,
132
+ "logps": -92.50672149658203,
133
+ "loss": 0.5214,
134
+ "objective": 0.5278546214103699,
135
+ "ranking_idealized": 0.5924999713897705,
136
+ "ranking_idealized_expo": 0.5166666507720947,
137
+ "ranking_simple": 0.6362500190734863,
138
+ "regularize": 0.5278546214103699,
139
+ "step": 200,
140
+ "wo_beta": 5.543394565582275
141
+ },
142
+ {
143
+ "epoch": 0.5668398677373642,
144
+ "eval_dpo_loss": 0.69049072265625,
145
+ "eval_logits": -1.8895256519317627,
146
+ "eval_logps": -99.51531982421875,
147
+ "eval_loss": 0.6901801824569702,
148
+ "eval_objective": 0.69049072265625,
149
+ "eval_ranking_idealized": 0.6030020713806152,
150
+ "eval_ranking_idealized_expo": 0.5222567319869995,
151
+ "eval_ranking_simple": 0.544513463973999,
152
+ "eval_regularize": 0.69049072265625,
153
+ "eval_runtime": 308.0169,
154
+ "eval_samples_per_second": 18.798,
155
+ "eval_steps_per_second": 1.568,
156
+ "eval_wo_beta": 7.701313495635986,
157
+ "step": 200
158
+ },
159
+ {
160
+ "dpo_loss": 0.4935472011566162,
161
+ "epoch": 0.7085498346717053,
162
+ "grad_norm": 14.324676993767012,
163
+ "learning_rate": 4.973122855144066e-06,
164
+ "logits": -1.9081355333328247,
165
+ "logps": -88.3556137084961,
166
+ "loss": 0.4922,
167
+ "objective": 0.4935472011566162,
168
+ "ranking_idealized": 0.5991666913032532,
169
+ "ranking_idealized_expo": 0.5170833468437195,
170
+ "ranking_simple": 0.6608333587646484,
171
+ "regularize": 0.4935472011566162,
172
+ "step": 250,
173
+ "wo_beta": 5.325418472290039
174
+ },
175
+ {
176
+ "epoch": 0.7085498346717053,
177
+ "eval_dpo_loss": 0.6914450526237488,
178
+ "eval_logits": -1.9887231588363647,
179
+ "eval_logps": -82.8383560180664,
180
+ "eval_loss": 0.6975522041320801,
181
+ "eval_objective": 0.6914450526237488,
182
+ "eval_ranking_idealized": 0.6030020713806152,
183
+ "eval_ranking_idealized_expo": 0.5222567319869995,
184
+ "eval_ranking_simple": 0.5481366515159607,
185
+ "eval_regularize": 0.6914450526237488,
186
+ "eval_runtime": 307.641,
187
+ "eval_samples_per_second": 18.821,
188
+ "eval_steps_per_second": 1.57,
189
+ "eval_wo_beta": 7.878448486328125,
190
+ "step": 250
191
+ },
192
+ {
193
+ "dpo_loss": 0.4521400034427643,
194
+ "epoch": 0.8502598016060463,
195
+ "grad_norm": 13.428528764338076,
196
+ "learning_rate": 4.924776641419513e-06,
197
+ "logits": -2.0954272747039795,
198
+ "logps": -81.39044952392578,
199
+ "loss": 0.4535,
200
+ "objective": 0.4521400034427643,
201
+ "ranking_idealized": 0.5799999833106995,
202
+ "ranking_idealized_expo": 0.4970833361148834,
203
+ "ranking_simple": 0.6837499737739563,
204
+ "regularize": 0.4521400034427643,
205
+ "step": 300,
206
+ "wo_beta": 5.026640892028809
207
+ },
208
+ {
209
+ "epoch": 0.8502598016060463,
210
+ "eval_dpo_loss": 0.6893911957740784,
211
+ "eval_logits": -2.1208713054656982,
212
+ "eval_logps": -90.9490966796875,
213
+ "eval_loss": 0.6922824382781982,
214
+ "eval_objective": 0.6893911957740784,
215
+ "eval_ranking_idealized": 0.6030020713806152,
216
+ "eval_ranking_idealized_expo": 0.5222567319869995,
217
+ "eval_ranking_simple": 0.556418240070343,
218
+ "eval_regularize": 0.6893911957740784,
219
+ "eval_runtime": 307.9716,
220
+ "eval_samples_per_second": 18.8,
221
+ "eval_steps_per_second": 1.568,
222
+ "eval_wo_beta": 7.4231791496276855,
223
+ "step": 300
224
+ },
225
+ {
226
+ "dpo_loss": 0.40847164392471313,
227
+ "epoch": 0.9919697685403873,
228
+ "grad_norm": 15.041732370189118,
229
+ "learning_rate": 4.8526047530778175e-06,
230
+ "logits": -1.9279303550720215,
231
+ "logps": -84.23763275146484,
232
+ "loss": 0.4228,
233
+ "objective": 0.40847164392471313,
234
+ "ranking_idealized": 0.60916668176651,
235
+ "ranking_idealized_expo": 0.5270833373069763,
236
+ "ranking_simple": 0.7191666960716248,
237
+ "regularize": 0.40847164392471313,
238
+ "step": 350,
239
+ "wo_beta": 3.9990389347076416
240
+ },
241
+ {
242
+ "epoch": 0.9919697685403873,
243
+ "eval_dpo_loss": 0.6968410611152649,
244
+ "eval_logits": -1.9802873134613037,
245
+ "eval_logps": -87.72307586669922,
246
+ "eval_loss": 0.7063526511192322,
247
+ "eval_objective": 0.6968410611152649,
248
+ "eval_ranking_idealized": 0.6030020713806152,
249
+ "eval_ranking_idealized_expo": 0.5222567319869995,
250
+ "eval_ranking_simple": 0.5538302063941956,
251
+ "eval_regularize": 0.6968410611152649,
252
+ "eval_runtime": 307.977,
253
+ "eval_samples_per_second": 18.8,
254
+ "eval_steps_per_second": 1.568,
255
+ "eval_wo_beta": 8.025344848632812,
256
+ "step": 350
257
+ },
258
+ {
259
+ "dpo_loss": 0.28143200278282166,
260
+ "epoch": 1.1336797354747283,
261
+ "grad_norm": 12.112116162894052,
262
+ "learning_rate": 4.757316345716554e-06,
263
+ "logits": -2.0127880573272705,
264
+ "logps": -91.47932434082031,
265
+ "loss": 0.2845,
266
+ "objective": 0.28143200278282166,
267
+ "ranking_idealized": 0.6087499856948853,
268
+ "ranking_idealized_expo": 0.5337499976158142,
269
+ "ranking_simple": 0.8141666650772095,
270
+ "regularize": 0.28143200278282166,
271
+ "step": 400,
272
+ "wo_beta": 2.649115562438965
273
+ },
274
+ {
275
+ "epoch": 1.1336797354747283,
276
+ "eval_dpo_loss": 0.7269378900527954,
277
+ "eval_logits": -2.080526828765869,
278
+ "eval_logps": -101.31802368164062,
279
+ "eval_loss": 0.7304782867431641,
280
+ "eval_objective": 0.7269378900527954,
281
+ "eval_ranking_idealized": 0.6030020713806152,
282
+ "eval_ranking_idealized_expo": 0.5222567319869995,
283
+ "eval_ranking_simple": 0.5429606437683105,
284
+ "eval_regularize": 0.7269378900527954,
285
+ "eval_runtime": 308.0503,
286
+ "eval_samples_per_second": 18.796,
287
+ "eval_steps_per_second": 1.568,
288
+ "eval_wo_beta": 8.616350173950195,
289
+ "step": 400
290
+ },
291
+ {
292
+ "dpo_loss": 0.3042532503604889,
293
+ "epoch": 1.2753897024090695,
294
+ "grad_norm": 12.171625264502326,
295
+ "learning_rate": 4.639847716126855e-06,
296
+ "logits": -1.9084649085998535,
297
+ "logps": -92.91566467285156,
298
+ "loss": 0.2989,
299
+ "objective": 0.3042532503604889,
300
+ "ranking_idealized": 0.5975000262260437,
301
+ "ranking_idealized_expo": 0.5199999809265137,
302
+ "ranking_simple": 0.8075000047683716,
303
+ "regularize": 0.3042532503604889,
304
+ "step": 450,
305
+ "wo_beta": 3.341869592666626
306
+ },
307
+ {
308
+ "epoch": 1.2753897024090695,
309
+ "eval_dpo_loss": 0.6973706483840942,
310
+ "eval_logits": -1.8646337985992432,
311
+ "eval_logps": -93.1955337524414,
312
+ "eval_loss": 0.7005103826522827,
313
+ "eval_objective": 0.6973706483840942,
314
+ "eval_ranking_idealized": 0.6030020713806152,
315
+ "eval_ranking_idealized_expo": 0.5222567319869995,
316
+ "eval_ranking_simple": 0.5605590343475342,
317
+ "eval_regularize": 0.6973706483840942,
318
+ "eval_runtime": 308.4071,
319
+ "eval_samples_per_second": 18.774,
320
+ "eval_steps_per_second": 1.566,
321
+ "eval_wo_beta": 8.238639831542969,
322
+ "step": 450
323
+ },
324
+ {
325
+ "dpo_loss": 0.29817140102386475,
326
+ "epoch": 1.4170996693434104,
327
+ "grad_norm": 15.895661459470155,
328
+ "learning_rate": 4.501353102310901e-06,
329
+ "logits": -1.9325114488601685,
330
+ "logps": -91.53684997558594,
331
+ "loss": 0.3065,
332
+ "objective": 0.29817140102386475,
333
+ "ranking_idealized": 0.57833331823349,
334
+ "ranking_idealized_expo": 0.4983333349227905,
335
+ "ranking_simple": 0.7975000143051147,
336
+ "regularize": 0.29817140102386475,
337
+ "step": 500,
338
+ "wo_beta": 3.1243510246276855
339
+ },
340
+ {
341
+ "epoch": 1.4170996693434104,
342
+ "eval_dpo_loss": 0.7146824598312378,
343
+ "eval_logits": -1.9982556104660034,
344
+ "eval_logps": -97.01371002197266,
345
+ "eval_loss": 0.717850923538208,
346
+ "eval_objective": 0.7146824598312378,
347
+ "eval_ranking_idealized": 0.6030020713806152,
348
+ "eval_ranking_idealized_expo": 0.5222567319869995,
349
+ "eval_ranking_simple": 0.5548654198646545,
350
+ "eval_regularize": 0.7146824598312378,
351
+ "eval_runtime": 308.8392,
352
+ "eval_samples_per_second": 18.748,
353
+ "eval_steps_per_second": 1.564,
354
+ "eval_wo_beta": 8.27602767944336,
355
+ "step": 500
356
+ },
357
+ {
358
+ "dpo_loss": 0.2845906615257263,
359
+ "epoch": 1.5588096362777515,
360
+ "grad_norm": 10.66218965165015,
361
+ "learning_rate": 4.34319334202531e-06,
362
+ "logits": -1.9555292129516602,
363
+ "logps": -97.59473419189453,
364
+ "loss": 0.2885,
365
+ "objective": 0.2845906615257263,
366
+ "ranking_idealized": 0.5945833325386047,
367
+ "ranking_idealized_expo": 0.5116666555404663,
368
+ "ranking_simple": 0.8041666746139526,
369
+ "regularize": 0.2845906615257263,
370
+ "step": 550,
371
+ "wo_beta": 2.6915340423583984
372
+ },
373
+ {
374
+ "epoch": 1.5588096362777515,
375
+ "eval_dpo_loss": 0.7133627533912659,
376
+ "eval_logits": -1.9041162729263306,
377
+ "eval_logps": -107.96095275878906,
378
+ "eval_loss": 0.7091230750083923,
379
+ "eval_objective": 0.7133627533912659,
380
+ "eval_ranking_idealized": 0.6030020713806152,
381
+ "eval_ranking_idealized_expo": 0.5222567319869995,
382
+ "eval_ranking_simple": 0.5615941882133484,
383
+ "eval_regularize": 0.7133627533912659,
384
+ "eval_runtime": 308.0507,
385
+ "eval_samples_per_second": 18.796,
386
+ "eval_steps_per_second": 1.568,
387
+ "eval_wo_beta": 8.196797370910645,
388
+ "step": 550
389
+ },
390
+ {
391
+ "epoch": 1.5588096362777515,
392
+ "step": 550,
393
+ "total_flos": 0.0,
394
+ "train_loss": 0.4483427975394509,
395
+ "train_runtime": 15087.4424,
396
+ "train_samples_per_second": 16.836,
397
+ "train_steps_per_second": 0.117
398
+ }
399
+ ],
400
+ "logging_steps": 50,
401
+ "max_steps": 1760,
402
+ "num_input_tokens_seen": 0,
403
+ "num_train_epochs": 5,
404
+ "save_steps": 50,
405
+ "stateful_callbacks": {
406
+ "EarlyStoppingCallback": {
407
+ "args": {
408
+ "early_stopping_patience": 5,
409
+ "early_stopping_threshold": 0.0
410
+ },
411
+ "attributes": {
412
+ "early_stopping_patience_counter": 0
413
+ }
414
+ },
415
+ "TrainerControl": {
416
+ "args": {
417
+ "should_epoch_stop": false,
418
+ "should_evaluate": false,
419
+ "should_log": false,
420
+ "should_save": true,
421
+ "should_training_stop": true
422
+ },
423
+ "attributes": {}
424
+ }
425
+ },
426
+ "total_flos": 0.0,
427
+ "train_batch_size": 4,
428
+ "trial_name": null,
429
+ "trial_params": null
430
+ }