About lambda_st

[aberhu]

About lambda_st

[wnma3mz]

@Ga-Lee My own guess is that it depends more on the loss of ground truth. st_loss exists as a constraint. Either should work, but it may depend on the specific downstream task.

[Ga-Lee]

Thank you very much for your project and it has helped me a lot. But I have some doubts about train_st.py.

Knowledge-Distillation-Zoo/train_st.py

Lines 184 to 188 in b7bce27

	cls_loss = criterionCls(output_s, target)
	st_loss = criterionST(F.log_softmax(output_s/args.T, dim=1),
	F.softmax(output_t/args.T, dim=1)) * (args.T*args.T) / img.size(0)
	st_loss = st_loss * args.lambda_st
	loss = cls_loss + st_loss

Here lambda_st is similar to a relaxation factor. But I see the following formula elsewhere.

 cls_loss = criterionCls(output_s, target) 
 st_loss  = criterionST(F.log_softmax(output_s/args.T, dim=1), 
 	                   F.softmax(output_t/args.T, dim=1)) * (args.T*args.T) / img.size(0) 

 loss = (1 - * args.lambda_st ) * cls_loss + st_loss * args.lambda_st 

Could you tell me the impact of the two?

https://github.com/PolarisShi/distillation/blob/8290b4b2138410cb4f1f39ecf2411f7dbb2a676a/student2.py#L218-L223

hello, I feel a little confused about it ,too. May you explain it to me? Thanks a lot

[Ga-Lee]

@Ga-Lee My own guess is that it depends more on the loss of ground truth. st_loss exists as a constraint. Either should work, but it may depend on the specific downstream task.

Got that, thank you~