Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

minicpm_2_6o全量微调验证集eval_loss不计算不打印,也不绘制eval_loss图 #6988

Open
1 task done
Maflyflyy opened this issue Feb 18, 2025 · 1 comment
Open
1 task done
Labels
bug Something isn't working pending This problem is yet to be addressed

Comments

@Maflyflyy
Copy link

Reminder

  • I have read the above rules and searched the existing issues.

System Info

Reminder

I have read the README and searched the existing issues.
System Info
llamafactory version: 0.9.2.dev0
Python version: 3.10.
PyTorch version: 2.2.0 (NPU)
Transformers version: 4.44.2
NPU type: 910b
Reproduction
model
model_name_or_path: minicpm-o
template: minicpm_o
trust_remote_code: true

method
stage: sft
do_train: true
finetuning_type: sft
xxxxxxx

dataset
dataset: train_dataset
template: xxxxx
cutoff_len: 2048
max_samples: 180000000
overwrite_cache: true
preprocessing_num_workers: 16

output
output_dir: saves/train_output
logging_steps: 10
save_steps: 100
plot_loss: true
overwrite_output_dir: true

train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 0.0001
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
fp16: true

eval
val_size: 0.1
per_device_eval_batch_size: 2
evaluation_strategy: steps
eval_steps: 100

Reproduction

运行后会进行验证操作,但没有输出eval_loss,以及最后的eval_loss变化图像。训练日志中,仅有{'eval_runtime': 226.5617, 'eval_samples_per_second': 4.423, 'eval_steps_per_second': 0.737, 'epoch': 1.0},里面不包含eval_loss。并且训练日志中有提示 WARNING - llamafactory.extras.ploting - No metric eval_loss to plot.
同样的config,qwen2正常输出eval_loss和绘图,但是minicpm不行

Others

No response

@Maflyflyy Maflyflyy added bug Something isn't working pending This problem is yet to be addressed labels Feb 18, 2025
@Maflyflyy
Copy link
Author

同样的yaml改成qwen2的,qwen2正常输出eval_loss和绘图,但是minicpm不行

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pending This problem is yet to be addressed
Projects
None yet
Development

No branches or pull requests

1 participant