| 2025-06-18 17:35:18,047 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 17:35:18,048 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 17:35:18,048 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 17:35:18,049 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 17:35:18,049 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 17:35:18,050 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 17:35:18,050 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 17:35:18,051 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 17:35:18,051 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 17:35:18,051 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 17:35:18,051 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 17:35:18,052 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 17:35:18,052 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 17:35:18,053 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 17:35:18,057 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,456 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,481 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,489 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,490 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,496 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,498 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:18,500 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:35:19,732 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,735 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,739 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,752 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,756 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,766 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,773 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:19,805 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:35:39,530 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,617 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,649 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,683 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 17:35:39,684 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 17:35:39,685 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 17:35:39,685 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 17:35:39,690 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,691 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,708 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:39,817 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:35:42,532 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:36:22,540 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,572 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,629 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,644 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,672 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,675 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,724 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,728 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,775 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,784 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,826 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,829 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,877 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:36:22,881 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,928 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:22,982 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:36:28,892 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 17:36:30,376 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,167 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:36:31,309 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:38:42,252 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 17:38:42,254 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 17:38:42,254 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 17:38:42,254 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 17:38:42,255 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 17:38:42,255 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 17:38:42,255 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 17:38:42,255 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 17:38:42,256 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 17:38:42,256 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 17:38:42,256 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 17:38:42,256 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 17:38:42,257 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 17:38:42,258 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 17:38:42,265 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,446 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,450 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,485 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,486 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,488 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,489 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:42,489 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:38:44,172 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,174 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,188 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,196 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,203 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,209 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,214 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:38:44,223 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:39:03,054 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,061 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,064 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,085 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,152 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,244 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,292 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 17:39:03,292 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 17:39:03,293 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 17:39:03,294 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 17:39:03,299 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:03,471 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:39:45,711 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,763 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,811 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,815 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:45,862 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,871 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:45,912 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,917 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:45,962 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:45,968 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:46,011 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:46,012 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:46,062 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:39:46,064 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:46,107 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:46,159 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:39:52,106 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 17:39:53,772 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,846 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,846 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,847 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,847 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,848 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,848 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:39:53,952 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:50:51,734 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 17:50:51,868 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 17:50:51,868 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 17:50:51,869 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 17:50:51,869 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 17:50:51,869 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 17:50:51,870 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 17:50:51,870 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 17:50:51,870 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 17:50:51,871 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 17:50:51,871 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 17:50:51,871 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 17:50:51,872 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 17:50:51,872 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 17:50:51,877 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,886 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,931 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,941 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,943 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,945 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,949 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:51,956 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 17:50:54,067 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,076 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,077 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,083 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,091 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,111 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,114 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:50:54,124 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 17:51:12,889 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 17:51:12,890 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 17:51:12,890 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 17:51:12,891 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 17:51:12,895 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:12,896 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:12,903 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:12,912 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:12,919 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:12,952 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:13,007 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:51:13,040 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 17:52:01,326 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,363 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,415 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,425 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,463 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,466 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,517 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,517 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,568 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,568 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,617 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,620 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,670 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 17:52:01,676 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,722 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:01,775 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 17:52:08,439 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 17:52:10,235 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,639 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,639 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,640 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,639 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,639 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,639 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 17:52:10,734 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:06:46,964 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 18:06:47,240 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 18:06:47,241 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 18:06:47,242 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 18:06:47,242 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 18:06:47,242 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 18:06:47,243 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 18:06:47,243 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 18:06:47,244 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 18:06:47,244 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 18:06:47,244 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 18:06:47,245 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 18:06:47,245 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 18:06:47,245 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 18:06:47,246 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,246 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,246 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,246 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,247 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,246 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,247 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:47,249 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:06:49,050 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,072 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,073 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,073 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,081 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,083 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,091 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:06:49,104 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:07:08,994 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,039 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,047 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,054 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 18:07:09,055 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 18:07:09,055 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 18:07:09,055 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 18:07:09,059 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,124 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,155 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,206 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:09,466 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:07:52,676 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,691 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,719 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,770 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,788 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:52,798 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:52,820 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,848 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:52,871 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,895 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:52,919 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,928 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:52,970 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:07:52,977 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:53,028 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:53,080 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:07:59,376 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 18:08:00,532 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,043 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,043 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,043 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,044 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,044 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,045 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:08:01,291 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:11:10,694 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 18:11:10,696 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 18:11:10,696 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 18:11:10,696 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 18:11:10,697 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 18:11:10,697 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 18:11:10,697 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 18:11:10,697 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 18:11:10,698 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 18:11:10,698 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 18:11:10,698 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 18:11:10,699 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 18:11:10,699 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 18:11:10,699 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 18:11:10,704 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,820 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,870 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,882 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,892 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,906 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,906 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:10,907 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 18:11:12,840 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,864 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,867 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,889 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,899 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,914 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,927 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:12,932 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 18:11:31,957 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:31,977 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:31,986 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:32,024 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 18:11:32,025 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 18:11:32,025 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 18:11:32,026 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 18:11:32,030 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:32,061 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:32,115 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:32,119 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:11:32,233 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 18:12:15,506 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,512 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,518 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,524 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,568 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,615 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,639 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,644 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,645 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,662 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,665 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,695 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,715 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 18:12:15,730 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,807 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:15,850 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 18:12:22,182 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 18:12:23,464 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,891 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,891 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,891 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,891 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,891 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,892 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 18:12:23,992 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:48:22,578 - __main__ - INFO - 📊 Configuration: | |
| 2025-06-18 21:48:22,761 - __main__ - INFO - Model: Qwen/Qwen3-1.7B-Base | |
| 2025-06-18 21:48:22,762 - __main__ - INFO - Dataset: thangvip/tokenized-ds-qwen3-legal-mixed | |
| 2025-06-18 21:48:22,762 - __main__ - INFO - Training mode: Full parameter training | |
| 2025-06-18 21:48:22,763 - __main__ - INFO - Distributed strategy: DDP (DistributedDataParallel) | |
| 2025-06-18 21:48:22,763 - __main__ - INFO - Accelerator state: Distributed environment: DistributedType.MULTI_GPU Backend: nccl | |
| Num processes: 8 | |
| Process index: 0 | |
| Local process index: 0 | |
| Device: cuda:0 | |
| Mixed precision type: bf16 | |
| 2025-06-18 21:48:22,763 - __main__ - INFO - Number of processes: 8 | |
| 2025-06-18 21:48:22,763 - __main__ - INFO - Device: cuda:0 | |
| 2025-06-18 21:48:22,764 - __main__ - INFO - Mixed precision: bf16 | |
| 2025-06-18 21:48:22,764 - __main__ - INFO - 🚀 DDP Optimizations: | |
| 2025-06-18 21:48:22,764 - __main__ - INFO - DDP bucket size: 25MB | |
| 2025-06-18 21:48:22,765 - __main__ - INFO - DDP broadcast buffers: True | |
| 2025-06-18 21:48:22,765 - __main__ - INFO - DDP find unused parameters: False | |
| 2025-06-18 21:48:22,765 - __main__ - INFO - Strategy: Data Parallelism - each GPU has full model copy | |
| 2025-06-18 21:48:22,767 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,769 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,787 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,793 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,813 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,835 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,836 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:22,836 - __main__ - INFO - 📚 Loading tokenizer... | |
| 2025-06-18 21:48:25,072 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,086 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,106 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,111 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,114 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,120 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,127 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:25,149 - __main__ - INFO - 🔧 Loading model... | |
| 2025-06-18 21:48:44,229 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,250 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,250 - __main__ - INFO - 🔥 Full Parameter Training Enabled | |
| 2025-06-18 21:48:44,251 - __main__ - INFO - Total parameters: 1,720,574,976 | |
| 2025-06-18 21:48:44,251 - __main__ - INFO - Trainable parameters: 1,720,574,976 | |
| 2025-06-18 21:48:44,252 - __main__ - INFO - Trainable %: 100.00% | |
| 2025-06-18 21:48:44,256 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,258 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,263 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,354 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,363 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:48:44,379 - __main__ - INFO - 📊 Preparing dataset... | |
| 2025-06-18 21:49:28,437 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,447 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,481 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,483 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,488 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,502 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,528 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,574 - __main__ - INFO - Dataset size: 392686 training examples | |
| 2025-06-18 21:49:28,584 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,606 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,651 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,654 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,660 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,673 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,687 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:28,716 - __main__ - INFO - 🎯 Creating SFT Trainer... | |
| 2025-06-18 21:49:35,378 - accelerate.utils.other - WARNING - Detected kernel version 4.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| 2025-06-18 21:49:36,621 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,908 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,910 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,910 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,912 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,918 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:36,918 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-18 21:49:37,114 - __main__ - INFO - 🔄 Resuming from checkpoint: /data/outputs/qwen3-1.7b-legal-mixed/checkpoint-800 | |
| 2025-06-18 21:49:37,115 - __main__ - INFO - 🚂 Starting TRL training... | |
| 2025-06-19 00:50:37,274 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:37,393 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:37,394 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:39,073 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:39,178 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:39,179 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:40,865 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:40,972 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:40,973 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:42,661 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:42,766 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:42,767 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:44,458 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:44,560 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:44,561 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:46,251 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:46,352 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:46,353 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:48,045 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:48,148 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:48,149 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:49,841 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:49,943 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:50:49,943 - accelerate.accelerator - INFO - The used dataset had no length, returning gathered tensors. You should drop the remainder yourself. | |
| 2025-06-19 00:52:54,008 - __main__ - INFO - 💾 Saving final model... | |
| 2025-06-19 00:53:25,255 - __main__ - INFO - 🚀 Pushing model to Hugging Face Hub: thangvip/qwen3-1.7b-legal-mixed | |