TrevorJS/dpo

README.md

@@ -0,0 +1,82 @@
+---
+license: other
+base_model: microsoft/phi-1_5
+tags:
+- generated_from_trainer
+model-index:
+- name: results
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# results
+
+This model is a fine-tuned version of [microsoft/phi-1_5](https://huggingface.co/microsoft/phi-1_5) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.0001
+- Rewards/chosen: -7.5874
+- Rewards/rejected: -24.0497
+- Rewards/accuracies: 1.0
+- Rewards/margins: 16.4623
+- Logps/rejected: -274.3435
+- Logps/chosen: -143.2090
+- Logits/rejected: -1.8100
+- Logits/chosen: -1.4786
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 4
+- eval_batch_size: 1
+- seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 100
+- training_steps: 1500
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.0417        | 0.07  | 100  | 0.0418          | -0.3892        | -8.0118          | 0.9792             | 7.6226          | -113.9640      | -71.2264     | 1.8258          | 1.7898        |
+| 0.0221        | 0.15  | 200  | 0.0303          | -2.5657        | -10.9212         | 0.9896             | 8.3555          | -143.0585      | -92.9920     | 1.9704          | 2.1047        |
+| 0.0107        | 0.22  | 300  | 0.0131          | -1.7388        | -11.6047         | 0.9965             | 9.8659          | -149.8935      | -84.7232     | 1.0731          | 0.9750        |
+| 0.0204        | 0.29  | 400  | 0.0108          | -2.0131        | -11.9647         | 0.9965             | 9.9516          | -153.4932      | -87.4658     | 1.3610          | 1.6740        |
+| 0.0067        | 0.36  | 500  | 0.0080          | -5.9488        | -19.6561         | 0.9974             | 13.7073         | -230.4076      | -126.8228    | -0.4464         | -0.2114       |
+| 0.0           | 0.44  | 600  | 0.0047          | -5.6456        | -20.2381         | 0.9983             | 14.5924         | -236.2268      | -123.7909    | -0.4142         | -0.0244       |
+| 0.0003        | 0.51  | 700  | 0.0018          | -7.2250        | -21.3351         | 0.9991             | 14.1101         | -247.1974      | -139.5853    | -0.3510         | -0.0203       |
+| 0.0005        | 0.58  | 800  | 0.0008          | -7.2263        | -21.2475         | 0.9991             | 14.0211         | -246.3209      | -139.5981    | -0.8673         | -0.7010       |
+| 0.0           | 0.66  | 900  | 0.0009          | -10.2371       | -26.0402         | 0.9991             | 15.8031         | -294.2486      | -169.7062    | -1.9784         | -1.7799       |
+| 0.0           | 0.73  | 1000 | 0.0008          | -5.9544        | -22.0767         | 0.9991             | 16.1223         | -254.6137      | -126.8789    | -1.0623         | -0.6039       |
+| 0.0           | 0.8   | 1100 | 0.0007          | -7.3374        | -23.8700         | 0.9991             | 16.5327         | -272.5467      | -140.7083    | -1.5517         | -1.1710       |
+| 0.0           | 0.87  | 1200 | 0.0007          | -7.6398        | -24.1605         | 0.9991             | 16.5207         | -275.4509      | -143.7327    | -1.8124         | -1.4901       |
+| 0.0           | 0.95  | 1300 | 0.0001          | -7.5920        | -24.0476         | 1.0                | 16.4556         | -274.3220      | -143.2550    | -1.8115         | -1.4816       |
+| 0.0001        | 1.02  | 1400 | 0.0001          | -7.5872        | -24.0480         | 1.0                | 16.4608         | -274.3262      | -143.2065    | -1.8102         | -1.4791       |
+| 0.0           | 1.09  | 1500 | 0.0001          | -7.5874        | -24.0497         | 1.0                | 16.4623         | -274.3435      | -143.2090    | -1.8100         | -1.4786       |
+
+
+### Framework versions
+
+- Transformers 4.33.2
+- Pytorch 2.0.1+cu118
+- Datasets 2.14.5
+- Tokenizers 0.13.3

adapter_config.json

@@ -0,0 +1,37 @@
+{
+  "auto_mapping": null,
+  "base_model_name_or_path": null,
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "lora_alpha": 64,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "revision": null,
+  "target_modules": [
+    "layers.0.wte",
+    "layers.20.mixer.Wqkv",
+    "layers.21.mixer.Wqkv",
+    "layers.22.mixer.Wqkv",
+    "layers.23.mixer.Wqkv",
+    "layers.24.mixer.Wqkv",
+    "layers.20.mixer.out_proj",
+    "layers.21.mixer.out_proj",
+    "layers.22.mixer.out_proj",
+    "layers.23.mixer.out_proj",
+    "layers.24.mixer.out_proj",
+    "layers.11.mlp.fc1",
+    "layers.17.mlp.fc1",
+    "layers.24.mlp.fc1",
+    "layers.11.mlp.fc2",
+    "layers.17.mlp.fc2",
+    "layers.24.mlp.fc2",
+    "layers.25.linear"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b75bae5a5d0f42909f0d1c4cbe510186e7a7a86e30bb765ebad5c0c494f66ef
+size 29373021

added_tokens.json

@@ -0,0 +1,40 @@
+{
+  "\t\t": 50294,
+  "\t\t\t": 50293,
+  "\t\t\t\t": 50292,
+  "\t\t\t\t\t": 50291,
+  "\t\t\t\t\t\t": 50290,
+  "\t\t\t\t\t\t\t": 50289,
+  "\t\t\t\t\t\t\t\t": 50288,
+  "\t\t\t\t\t\t\t\t\t": 50287,
+  "  ": 50286,
+  "   ": 50285,
+  "    ": 50284,
+  "     ": 50283,
+  "      ": 50282,
+  "       ": 50281,
+  "        ": 50280,
+  "         ": 50279,
+  "          ": 50278,
+  "           ": 50277,
+  "            ": 50276,
+  "             ": 50275,
+  "              ": 50274,
+  "               ": 50273,
+  "                ": 50272,
+  "                 ": 50271,
+  "                  ": 50270,
+  "                   ": 50269,
+  "                    ": 50268,
+  "                     ": 50267,
+  "                      ": 50266,
+  "                       ": 50265,
+  "                        ": 50264,
+  "                         ": 50263,
+  "                          ": 50262,
+  "                           ": 50261,
+  "                            ": 50260,
+  "                             ": 50259,
+  "                              ": 50258,
+  "                               ": 50257
+}

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+  "add_prefix_space": false,
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 2048,
+  "tokenizer_class": "CodeGenTokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8447974061ff3aa683909047bc0204601a3eb99eb7a33c17eb97165db995a0b3
+size 4027