hobbitlong/supcontrast

PyTorch implementation of "Supervised Contrastive Learning" (and SimCLR incidentally)

supcontrast's Issues

Can you please explain order of data?

I apologize for the stupid question. I read the original article, but could not connect it with the implementation. I try to run simple code:

    features = torch.randn(256, 2, 128)
    features = F.normalize(features, dim=1)
    labels = torch.randint(10, size=(256,))
    criterion(features, labels)

and get nan.
This occurs because some values in exp_logits.sum (1, keepdim = True) are zero. What are the restrictions on the features/labels?

Help Needed

Hey, Can Supervised Contrastive Loss be implemented for Semantic Segmentation where the output of the model is of the shape;
features = [bsz, feat_dim, h, w]
target = [bsz, h, w]
target_embedding = [bsz, embed_dim, h, w]
It would be very helpful if you can help me. Thanks:)

Training time

Hi @HobbitLong
Thanks for the wonderful repo.

I am currently using a machine with 2 gpus (1080) to reproduces the SimCLR and SupConstrast on cifar10.
Could you please tell me how much time it would take for both methods until their convergence? I had some terrible experience of waiting for weeks for some parallel approaches.

Thanks for the information.
Jizong

Could you share the imagenet downstream model?

Hi, thanks for sharing this codebase. Could you please share the final imagenet downstream model (fc layer weights)? Thanks in advance

fairness issues for comparision

I find the proposed training strategy is 1) train the backbone with the labels and the contrastive loss, 2) finetune the last linear layer. The baseline approach is train the backbone and last linear layer at the same time with cross entropy loss. Do you have a reference of 1) train the backbone with cross entropy loss, 2) re-train the last linear layer from scratch?

The reason is that, the baseline here has multiple differences with the proposed solution. The gain could come from more iterations, i.e. iterations in pre-training + iterations in fine-tuning.

questions about effect of number of positives

Hi, thanks for your nice paper and codes!
I have a question after reading the paper, in the section "Effect of Number of Positives", the number of positives always contain one positive which is the same sample with the anchor but different data augmentations, and the remainders are different samples from the same class, have you tried make all the positive from the same sample but different augmentations? Does it improve?

loss is nan

the code:
mask.sum(1) has the number 0
so the loss is nan
Is there any problem with my code?
respect for your answer,thank you

pretrained model

Hi, due to the lack of GRU resource, I can not run the model with batch_size=1024. Therefore, can you release a pretrained simCLR model?
Thanks very much!

using Python 3.7.6 and PyTorch 1.4.0 loss is always NaN

Haven't tested on Python 2

Is there a big difference between one-stage and two-stage training?

I am a beginner in contrastive learning. I have one question.

Why isn't supervised contrastive loss trained as a regular term with a classifier(with cross-entropy)? What are the benefits of this two-stage training compared with one-stage training?

Look forward to your reply.

About contrast mode

What is the difference between all and one of the contrast modes?

How to monitor the stage 1 training

Hi ,
Thanks for this implimentation . i like to know how to monitor stage 1 training . i am applying this concept for a custom data for image classification. i the visualized projection are not getting grouped together even after running for 50-60 epoch . can you share some idea on how to monitor. also my batch size id limited to 12 because of GPU limitation . do you think this is am issue ?

thanks

Is there ignoring a log operation in SCL?

Hi HobbitLong:
Thank you for releasing code. Here, i have a question. Is there ignoring a log operation in 89 line of loss.py.

# compute log_prob
        exp_logits = torch.exp(logits) * logits_mask
        log_prob = logits - torch.log(exp_logits.sum(1, keepdim=True))

running CE loss does not work

Hi I am running given example in README
thanks

Traceback (most recent call last):
File "main_ce.py", line 333, in
main()
File "main_ce.py", line 302, in main
loss, train_acc = train(train_loader, model, criterion, optimizer, epoch, opt)
File "main_ce.py", line 214, in train
acc1, acc5 = accuracy(output, labels, topk=(1, 5))
File "/remote/idiap.svm/user.active/rkarimi/dev/internship/SupContrast/util.py", line 48, in accuracy
correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Where is the negative pairs loss?

Thanks for your sharing. I found the code just calculate the positive loss. I am confused with where is the negative pair loss calculation？

Thanks for your reply.

what is the purpose for subtracting the max logit?

Hi, thanks for the nice code!

Can I know why do you subtract the max logit? I'm curious what difference does it make to the optimization?

how to predict

it seems there is no prediction in main_supcon.py, how do you determine prediction? get the max of features between size (2, classes)?

L2 normalization not performed in the code?

Hi,

Where do you perform L2 normalization in the code? Is this something that needs to be done separately? Currently running into nan losses in my current dataset, and i'm trying to figure out if that is the reason.

cifar100 reproducing problem

Hi
I found a reproducing problem in cifar100&resnet50 with your parameters.
Accuracy with batch size 1024 is less than it of 512.

Simclr: 70.5 | 68.84 (512, 1024)
SupCon: 76.51 | 75.27 (512, 1024)

Is there something more to reproduce..?

base_temperature

Thanks for sharing the code!

What is the intuition behind base_temperature? I found that its value is always 0.07. Is it just for scaling the learning rate by temperature / 0.07?
https://github.com/HobbitLong/SupContrast/blob/master/losses.py#L95

Performance problem

Thank you for the great work!

I have been training the network with your code. In the Pretraining stage, I use 8 GPU to train in the cifar10 task and use this command:
python main_supcon.py --batch_size 1024
--learning_rate 0.5
--temp 0.1
--cosine

Then, in the Linear evaluation stage, I also adopt the following code to train:

python main_linear.py --batch_size 512
--learning_rate 5
--ckpt /path/to/model.pth

But the performance is 95.58. But your report performance is 96.0. Could you check it is normal?

Thanks~

Kecheng

Has anyone tried adding self-supervised constrative loss to SCL?

That is to say, supervised contrastive loss and self-supervised constrative loss, the former can be regarded as global and the latter is local.

supervised contrastive loss function

input tensor to SupConLoss need to have 3 dimenstion.
batch size, 2(two features made from augmentation), z_dim(128)

if I dont want augmentation,
[batch size, 1, z_dim(128)] is ok for SupConLoss ??

loss value is almost nan for batch(32)
thanks

SimCLR performance

Does the performance of the SimCLR implementation in this repo match the published results?

what this code do

for idx, (images, labels) in enumerate(train_loader):
    data_time.update(time.time() - end)

    images = torch.cat([images[0], images[1]], dim=0)

Supervised Contrastive learning

Hi , I have trained the network using 3 files provided, now how to check the accuracies of the model. for cifar100 there are multiple model files, so which one to choose. Moreover how the graphs are displayed using tensorboard (please, provide the instructions).

scatter

Pretrained models

Hello!
Are there any pretrained on Imagenet SupContrast models?

No L2 Normalization for the output of the encoder network?

In the paper it is described that L2 normalization is performed on the output of the encoder as well as the output of the projection head. However, in the code I could only find the L2 normalization being performed on the output of the projection head. Could someone clarify this mismatch?

Numbers of positives Nyi

Hi thanks for the great work! It seems that the numbers of positives described in the paper is a critical parameters, but the train_loader code isn't specified of this parameter Nyi. Does that means it is ok to randomly sample images of different class?

Loading the pretrained model on CPU

The model loading command "model.load_state_dict(state_dict)"
in the main_linear.py is overindented. Thus without a GPU the model is not loaded, which really got me confused at first.

Questions about DataParallel

As the model uses the DataParallel with k GPUs, the BatchSize m will be split into k parts (m = n*k) and then gather the gradients for backward.
So is the loss computed in the batch m (with more negative samples) or the small parts n (with less negative samples).

How many GPUs did you use?

Hi, I am wondering how many GPUs did you use for the parameters you provided in Readme?

Also, I am wondering why you only trained the encoder on Multiple GPUs rather than the whole model? Or did I miss something?

        model.encoder = torch.nn.DataParallel(model.encoder)

SimCLR can not converge on CIFAR10.

I train the model using setting as provided in README.md:

python main_supcon.py --batch_size 1024 \
  --learning_rate 0.5 \
  --temp 0.5 \
  --cosine --syncBN \
  --method SimCLR

The loss in tensorboard log doesn't converge as follows:
Step Loss
1 52.37631607055664
2 51.31996154785156
3 50.25985336303711
4 49.499568939208984
5 49.09306335449219
6 48.87476348876953
7 48.573631286621094
8 47.988162994384766
9 47.17831802368164
10 46.756771087646484
11 46.54935073852539
12 46.39372253417969
13 46.233951568603516
14 46.09927749633789
15 45.99102783203125
16 45.91159439086914
17 45.81057357788086
18 45.5342903137207
19 45.24546432495117
20 44.95869827270508
21 44.70873260498047
22 44.49382781982422
23 44.37926483154297
24 44.300750732421875
25 44.23575210571289
26 44.17127227783203
27 44.10074996948242
28 44.01371383666992
29 43.969486236572266
30 43.929256439208984
31 43.905479431152344
32 43.84855651855469
33 43.79642105102539
34 43.77620315551758
35 43.73455810546875
36 43.67094039916992
37 43.634544372558594
38 43.63047409057617
39 43.593658447265625
40 43.54821014404297
41 43.52227020263672
42 43.50453186035156
43 43.482696533203125
44 43.438812255859375
45 43.373321533203125
46 43.3624267578125
47 43.338783264160156
48 43.31117630004883
49 43.26757049560547
50 43.25471115112305
51 43.23273468017578
52 43.24850082397461
53 43.206825256347656
54 43.17913818359375
55 43.155879974365234
56 43.145408630371094
57 43.11878204345703
58 43.12966537475586
59 43.11239242553711
60 43.062042236328125
61 43.04346466064453
62 43.037723541259766
63 43.011138916015625
64 42.994407653808594
65 42.99079513549805
66 42.968414306640625
67 42.97438049316406
68 42.9185905456543
69 42.92195129394531
70 42.92424392700195
71 42.89225769042969
72 42.92551040649414
73 42.88059616088867
74 42.89714813232422
75 42.8614501953125
76 42.8603401184082
77 42.82914352416992
78 42.83755111694336
79 42.825382232666016
80 42.82973098754883
81 42.79133987426758
82 42.790252685546875
83 42.770957946777344
84 42.80356979370117
85 42.773014068603516
86 42.77165603637695
87 42.74653244018555
88 42.75342559814453
89 42.741119384765625
90 42.75617980957031
91 42.72799301147461
92 42.748008728027344
93 42.71396255493164
94 42.69316101074219
95 42.670108795166016
96 42.70542907714844
97 42.694095611572266
98 42.67554473876953
99 42.65801239013672
100 42.67591094970703
101 42.64384460449219
102 42.63705062866211
103 42.627235412597656
104 42.64301681518555
105 42.624053955078125
106 42.62214660644531
107 42.60559844970703
108 42.61735534667969
109 42.62445831298828
110 42.60248947143555
111 42.57283020019531
112 42.58974838256836
113 42.59029006958008
114 42.58818054199219
115 42.552772521972656
116 42.594791412353516
117 42.57866287231445
118 42.55373764038086
119 42.55537033081055
120 42.55070877075195
121 42.525482177734375
122 42.559322357177734
123 42.544517517089844
124 42.52219772338867
125 42.535728454589844
126 42.50553894042969
127 42.51144790649414
128 42.52484130859375
129 42.506187438964844
130 42.502166748046875
131 42.49041748046875
132 42.476890563964844
133 42.50419616699219
134 42.46426773071289
135 42.497535705566406
136 42.466556549072266
137 42.45137023925781
138 42.44359588623047
139 42.46516418457031
140 42.45024871826172
141 42.447364807128906
142 42.456512451171875
143 42.42683029174805
144 42.44383239746094
145 42.42966842651367
146 42.42776870727539
147 42.44707107543945
148 42.43446350097656
149 42.43083953857422
150 42.42207336425781
151 42.42733383178711
152 42.42057418823242
153 42.408233642578125
154 42.40159225463867
155 42.43068313598633
156 42.40238571166992
157 42.40227127075195
158 42.410133361816406
159 42.360538482666016
160 42.39555740356445
161 42.406089782714844
162 42.374088287353516
163 42.37333679199219
164 42.37459182739258
165 42.36992263793945
166 42.35036849975586
167 42.370765686035156
168 42.3534049987793
169 42.34937286376953
170 42.34113693237305
171 42.3719367980957
172 42.356689453125
173 42.3499755859375
174 42.35215759277344
175 42.367679595947266
176 42.37221145629883
177 42.35476303100586
178 42.34419631958008
179 42.33774185180664
180 42.34867477416992
181 42.343505859375
182 42.332584381103516
183 42.331886291503906
184 42.30333709716797
185 42.30495834350586
186 42.31267547607422
187 42.331504821777344
188 42.328800201416016
189 42.31763458251953
190 42.293487548828125
191 42.3109016418457
192 42.3144416809082
193 42.29933166503906
194 42.3012809753418
195 42.286216735839844
196 42.30648422241211
197 42.2964973449707
198 42.31986618041992
199 42.29719161987305
200 42.29789733886719
201 42.28410339355469
202 42.28727722167969
203 42.29293441772461
204 42.26756286621094
205 42.26470184326172
206 42.28857421875
207 42.28956985473633
208 42.278404235839844
209 42.27909469604492
210 42.28044891357422
211 42.249176025390625
212 42.26814270019531
213 42.2680778503418
214 42.25834655761719
215 42.24523162841797
216 42.25312042236328
217 42.26319122314453
218 42.25532150268555
219 42.235530853271484
220 42.23767852783203
221 42.23487091064453
222 42.2308349609375
223 42.2532844543457
224 42.222412109375
225 42.24436950683594
226 42.23404312133789
227 42.257808685302734
228 42.25067138671875
229 42.25416564941406
230 42.225440979003906
231 42.220970153808594
232 42.21751403808594
233 42.22220230102539
234 42.19740295410156
235 42.212947845458984
236 42.22639846801758
237 42.225074768066406
238 42.187904357910156
239 42.21226501464844
240 42.2158317565918
241 42.21928787231445
242 42.211605072021484
243 42.19694137573242
244 42.22465515136719
245 42.210567474365234
246 42.211029052734375
247 42.20466995239258
248 42.19947814941406
249 42.18659591674805
250 42.207908630371094
251 42.18763732910156
252 42.16108703613281
253 42.17030334472656
254 42.16826629638672
255 42.1864128112793
256 42.178138732910156
257 42.207794189453125
258 42.17320251464844
259 42.18295669555664
260 42.18595504760742
261 42.18653869628906
262 42.168556213378906
263 42.16699981689453
264 42.14784240722656
265 42.165992736816406
266 42.15581130981445
267 42.18312454223633
268 42.15531539916992
269 42.155147552490234
270 42.13705062866211
271 42.14425277709961
272 42.15453338623047
273 42.15085220336914
274 42.15422821044922
275 42.166961669921875
276 42.13074493408203
277 42.15336608886719
278 42.150901794433594
279 42.14059066772461
280 42.13984298706055
281 42.13311004638672
282 42.14921951293945
283 42.14160919189453
284 42.12321853637695
285 42.133209228515625
286 42.14018249511719
287 42.146156311035156
288 42.148895263671875
289 42.12226486206055
290 42.119327545166016
291 42.12271499633789
292 42.09735107421875
293 42.13063430786133
294 42.1333122253418
295 42.128719329833984
296 42.13404083251953
297 42.11404037475586
298 42.13532257080078
299 42.13797378540039
300 42.12031555175781
301 42.13229751586914
302 42.1247673034668
303 42.10991287231445
304 42.096988677978516
305 42.09716033935547
306 42.09797668457031
307 42.112796783447266
308 42.09970474243164
309 42.10281753540039
310 42.117523193359375
311 42.091373443603516
312 42.08698654174805
313 42.09238052368164
314 42.100791931152344
315 42.08475112915039
316 42.08399200439453
317 42.09626007080078
318 42.07393264770508
319 42.10096740722656
320 42.07324981689453
321 42.06714630126953
322 42.06916809082031
323 42.104183197021484
324 42.07938003540039
325 42.078514099121094
326 42.08796691894531
327 42.08335494995117
328 42.069602966308594
329 42.06338119506836
330 42.06172180175781
331 42.084617614746094
332 42.05459213256836
333 42.06312942504883
334 42.07081985473633
335 42.05907440185547
336 42.04154968261719
337 42.0722770690918
338 42.04973220825195
339 42.055843353271484
340 42.05887222290039
341 42.05158233642578
342 42.043949127197266
343 42.02438735961914
344 42.013797760009766
345 42.03730773925781
346 42.04892349243164
347 42.04452133178711
348 42.01693344116211
349 42.03768539428711
350 42.035709381103516
351 42.03788757324219
352 42.03336715698242
353 42.02366256713867
354 42.03593826293945
355 42.03465270996094
356 42.01338195800781
357 42.0163459777832
358 42.018856048583984
359 42.02410888671875
360 42.01831817626953
361 42.03579330444336
362 42.02202224731445
363 42.02480697631836
364 42.00835418701172
365 41.999298095703125
366 42.02631759643555
367 42.01470947265625
368 41.99877166748047
369 42.00832748413086
370 42.008548736572266
371 41.99556350708008
372 41.995269775390625
373 42.00551223754883
374 42.00926971435547
375 42.016422271728516
376 41.9869384765625
377 42.00880813598633
378 41.98211669921875
379 42.00701141357422
380 41.989627838134766
381 42.001365661621094
382 41.98550033569336
383 41.99925994873047
384 41.985443115234375
385 41.98691940307617
386 41.964473724365234
387 41.971683502197266
388 41.97077178955078
389 41.96350860595703
390 41.97527313232422
391 41.96897888183594
392 41.96701431274414
393 41.97261047363281
394 41.97776412963867
395 41.993186950683594
396 41.994205474853516
397 41.97821044921875
398 41.95921325683594
399 41.94550323486328
400 41.97230529785156
401 41.95723342895508
402 41.95551300048828
403 41.97821044921875
404 41.96847915649414
405 41.977046966552734
406 41.9465217590332
407 41.95376968383789
408 41.94062423706055
409 41.95761489868164
410 41.93984603881836
411 41.941184997558594
412 41.959529876708984
413 41.95286178588867
414 41.943336486816406
415 41.949058532714844
416 41.925315856933594
417 41.9235725402832
418 41.923301696777344
419 41.94486999511719
420 41.91834259033203
421 41.93389129638672
422 41.94923782348633
423 41.91939163208008
424 41.94850540161133
425 41.94353103637695
426 41.95414733886719
427 41.91848373413086
428 41.93660354614258
429 41.93687438964844
430 41.91402053833008
431 41.91341781616211
432 41.91807174682617
433 41.91950988769531
434 41.9164924621582
435 41.92567443847656
436 41.92313766479492
437 41.92038345336914
438 41.90835952758789
439 41.904605865478516
440 41.900421142578125
441 41.901634216308594
442 41.9135856628418
443 41.892555236816406
444 41.89800262451172
445 41.912933349609375
446 41.9108772277832
447 41.88585662841797
448 41.8737678527832
449 41.90205383300781
450 41.8900032043457
451 41.886817932128906
452 41.902130126953125
453 41.888797760009766
454 41.88259506225586
455 41.874332427978516
456 41.87350082397461
457 41.8810920715332
458 41.887630462646484
459 41.8868522644043
460 41.88219451904297
461 41.8549690246582
462 41.871681213378906
463 41.876739501953125
464 41.86948776245117
465 41.86659622192383
466 41.86448669433594
467 41.84907913208008
468 41.864837646484375
469 41.863677978515625
470 41.8576545715332
471 41.86296081542969
472 41.85995864868164
473 41.87732696533203
474 41.84720993041992
475 41.85145568847656
476 41.85929870605469
477 41.85707092285156
478 41.83163833618164
479 41.82987594604492
480 41.83938980102539
481 41.827880859375
482 41.85377502441406
483 41.820987701416016
484 41.839542388916016
485 41.84109115600586
486 41.82099914550781
487 41.839717864990234
488 41.82582473754883
489 41.83481979370117
490 41.841758728027344
491 41.81592559814453
492 41.82505798339844
493 41.83061599731445
494 41.81941223144531
495 41.838600158691406
496 41.802886962890625
497 41.812198638916016
498 41.815799713134766
499 41.817596435546875
500 41.82366943359375
501 41.83607864379883
502 41.80805206298828
503 41.81154251098633
504 41.80747985839844
505 41.80255126953125
506 41.780853271484375
507 41.786861419677734
508 41.80241775512695
509 41.792362213134766
510 41.79828643798828
511 41.80982208251953
512 41.77899169921875
513 41.79093551635742
514 41.7978401184082
515 41.785980224609375
516 41.773624420166016
517 41.786102294921875
518 41.79802322387695
519 41.782596588134766
520 41.77231979370117
521 41.78274154663086
522 41.78874969482422
523 41.781307220458984
524 41.79595184326172
525 41.781002044677734
526 41.75046920776367
527 41.768497467041016
528 41.771583557128906
529 41.77878952026367
530 41.773895263671875
531 41.75687026977539
532 41.762786865234375
533 41.768733978271484
534 41.74891662597656
535 41.74837875366211
536 41.74883270263672
537 41.75190353393555
538 41.74944305419922
539 41.74364471435547
540 41.77530288696289
541 41.74995803833008
542 41.727783203125
543 41.741119384765625
544 41.73719787597656
545 41.730220794677734
546 41.73965072631836
547 41.75122833251953
548 41.74309158325195
549 41.75099182128906
550 41.72608184814453
551 41.72199249267578
552 41.719600677490234
553 41.731605529785156
554 41.709617614746094
555 41.72447967529297
556 41.73017120361328
557 41.70937728881836
558 41.71445846557617
559 41.7201042175293
560 41.708614349365234
561 41.72523880004883
562 41.71660232543945
563 41.705814361572266
564 41.71420669555664
565 41.725250244140625
566 41.700626373291016
567 41.69807052612305
568 41.69969940185547
569 41.70137023925781
570 41.68778610229492
571 41.68761444091797
572 41.69614791870117
573 41.67936706542969
574 41.67458724975586
575 41.67314529418945
576 41.67515182495117
577 41.67046356201172
578 41.67753601074219
579 41.667606353759766
580 41.67475128173828
581 41.6955451965332
582 41.66904067993164
583 41.670658111572266
584 41.64606857299805
585 41.644012451171875
586 41.668724060058594
587 41.65727233886719
588 41.682044982910156
589 41.66094970703125
590 41.67304611206055
591 41.65132141113281
592 41.6454963684082
593 41.66543197631836
594 41.65175247192383
595 41.67536163330078
596 41.64577102661133
597 41.662147521972656
598 41.644168853759766
599 41.6528205871582
600 41.64643478393555
601 41.63383865356445
602 41.63182830810547
603 41.63913345336914
604 41.6375732421875
605 41.6337890625
606 41.628639221191406
607 41.6407470703125
608 41.636878967285156
609 41.62710952758789
610 41.636619567871094
611 41.610801696777344
612 41.6071662902832
613 41.60697555541992
614 41.626426696777344
615 41.618568420410156
616 41.615962982177734
617 41.59489059448242
618 41.60431671142578
619 41.58615493774414
620 41.608211517333984
621 41.607261657714844
622 41.58834457397461
623 41.619384765625
624 41.589935302734375
625 41.59272003173828
626 41.59053421020508
627 41.58329391479492
628 41.59050750732422
629 41.57986068725586
630 41.58558654785156
631 41.59832000732422
632 41.584468841552734
633 41.58831787109375
634 41.5704345703125
635 41.58049774169922
636 41.573272705078125
637 41.567806243896484
638 41.5692138671875
639 41.56473922729492
640 41.5532341003418
641 41.56444549560547
642 41.562835693359375
643 41.55622100830078
644 41.55236053466797
645 41.559200286865234
646 41.560638427734375
647 41.54945755004883
648 41.55522537231445
649 41.549259185791016
650 41.5435791015625
651 41.54538345336914
652 41.55497741699219
653 41.535770416259766
654 41.5565071105957
655 41.5403938293457
656 41.5374641418457
657 41.52592849731445
658 41.530433654785156
659 41.529991149902344
660 41.53873825073242
661 41.52483367919922
662 41.52631759643555
663 41.52540969848633
664 41.50712203979492
665 41.52194595336914
666 41.508949279785156
667 41.50340270996094
668 41.51433563232422
669 41.506813049316406
670 41.509037017822266
671 41.511940002441406
672 41.48933410644531
673 41.49421691894531
674 41.51228713989258
675 41.48188018798828
676 41.5179328918457
677 41.51668930053711
678 41.4907341003418
679 41.4776611328125
680 41.478702545166016
681 41.4755744934082
682 41.470985412597656
683 41.473052978515625
684 41.46126937866211
685 41.467952728271484
686 41.468509674072266
687 41.458740234375
688 41.46387481689453
689 41.46723175048828
690 41.479740142822266
691 41.451942443847656
692 41.459686279296875
693 41.4549446105957
694 41.44732666015625
695 41.45650863647461
696 41.436283111572266
697 41.4654541015625
698 41.45811080932617
699 41.440711975097656
700 41.448612213134766
701 41.444801330566406
702 41.424259185791016
703 41.42734146118164
704 41.42820739746094
705 41.41264343261719
706 41.428672790527344
707 41.433197021484375
708 41.41559600830078
709 41.42765808105469
710 41.41390609741211
711 41.41726303100586
712 41.41946029663086
713 41.40107345581055
714 41.41954040527344
715 41.40443420410156
716 41.39267349243164
717 41.418861389160156
718 41.411834716796875
719 41.40359115600586
720 41.396331787109375
721 41.390079498291016
722 41.37653732299805
723 41.37510299682617
724 41.387081146240234
725 41.37966537475586
726 41.371559143066406
727 41.3805046081543
728 41.37225341796875
729 41.36929702758789
730 41.36435317993164
731 41.36384963989258
732 41.36091613769531
733 41.362030029296875
734 41.35578918457031
735 41.3519172668457
736 41.35300827026367
737 41.358482360839844
738 41.347862243652344
739 41.37198257446289
740 41.35877990722656
741 41.35511016845703
742 41.33341598510742
743 41.3454704284668
744 41.338096618652344
745 41.32744216918945
746 41.3354606628418
747 41.318241119384766
748 41.33156204223633
749 41.326908111572266
750 41.34585189819336
751 41.32599639892578
752 41.32189178466797
753 41.31526184082031
754 41.3254280090332
755 41.31843185424805
756 41.316680908203125
757 41.31470489501953
758 41.31026840209961
759 41.29541015625
760 41.289337158203125
761 41.2948112487793
762 41.30418395996094
763 41.30305862426758
764 41.288063049316406
765 41.30720520019531
766 41.30964279174805
767 41.28276443481445
768 41.28688049316406
769 41.28529357910156
770 41.27595901489258
771 41.28208923339844
772 41.28265380859375
773 41.27253341674805
774 41.268795013427734
775 41.26056671142578
776 41.26545715332031
777 41.26414108276367
778 41.26213836669922
779 41.254638671875
780 41.264461517333984
781 41.255760192871094
782 41.24617385864258
783 41.253177642822266
784 41.24758529663086
785 41.247886657714844
786 41.24738311767578
787 41.247623443603516
788 41.23225021362305
789 41.23267364501953
790 41.23270034790039
791 41.21902847290039
792 41.21857452392578
793 41.22606658935547
794 41.2296257019043
795 41.21458053588867
796 41.22758483886719
797 41.22265625
798 41.213966369628906
799 41.205604553222656
800 41.20670700073242
801 41.207645416259766
802 41.19804763793945
803 41.19807052612305
804 41.19544982910156
805 41.18919372558594
806 41.19837951660156
807 41.18947982788086
808 41.180660247802734
809 41.189613342285156
810 41.17950439453125
811 41.1776123046875
812 41.18416213989258
813 41.163352966308594
814 41.16533660888672
815 41.16762161254883
816 41.168434143066406
817 41.17054748535156
818 41.1525993347168
819 41.172096252441406
820 41.17144775390625
821 41.151554107666016
822 41.15265655517578
823 41.156288146972656
824 41.14745330810547
825 41.13243865966797
826 41.13278579711914
827 41.12716293334961
828 41.138736724853516
829 41.133758544921875
830 41.11493682861328
831 41.12683868408203
832 41.12635040283203
833 41.133602142333984
834 41.13371658325195
835 41.10513687133789
836 41.10260009765625
837 41.101417541503906
838 41.10969543457031
839 41.11932373046875
840 41.1033821105957
841 41.1043586730957
842 41.095211029052734
843 41.096317291259766
844 41.0890007019043
845 41.09583282470703
846 41.09392547607422
847 41.08848190307617
848 41.0904655456543
849 41.084877014160156
850 41.08682632446289
851 41.0809211730957
852 41.093109130859375
853 41.077423095703125
854 41.0758056640625
855 41.0784797668457
856 41.07194137573242
857 41.07193374633789
858 41.06140899658203
859 41.0634651184082
860 41.05282211303711
861 41.050453186035156
862 41.05570983886719
863 41.05071258544922
864 41.059444427490234
865 41.05086898803711
866 41.054115295410156
867 41.040008544921875
868 41.03927993774414
869 41.044532775878906
870 41.03165817260742
871 41.04341506958008
872 41.03428649902344
873 41.0253791809082
874 41.0325927734375
875 41.015987396240234
876 41.016868591308594
877 41.01715087890625
878 41.01891326904297
879 41.022621154785156
880 41.015464782714844
881 41.012969970703125
882 41.00737762451172
883 41.008392333984375
884 41.01278305053711
885 41.00990295410156
886 41.0096321105957
887 41.007503509521484
888 41.00027084350586
889 40.9947395324707
890 40.99526596069336
891 41.001243591308594
892 41.007118225097656
893 40.98543167114258
894 40.989437103271484
895 40.991146087646484
896 40.98223114013672
897 40.99018478393555
898 40.980106353759766
899 40.98013687133789
900 40.97758483886719
901 40.97883605957031
902 40.97135925292969
903 40.96484375
904 40.97567367553711
905 40.96530532836914
906 40.9644775390625
907 40.9733772277832
908 40.96117401123047
909 40.95415496826172
910 40.96121597290039
911 40.955406188964844
912 40.95452117919922
913 40.95441436767578
914 40.953819274902344
915 40.94948959350586
916 40.94862365722656
917 40.9548454284668
918 40.9460563659668
919 40.95115280151367
920 40.942928314208984
921 40.93732833862305
922 40.94527053833008
923 40.94487762451172
924 40.93137741088867
925 40.931636810302734
926 40.94733428955078
927 40.93010711669922
928 40.9337158203125
929 40.936161041259766
930 40.93152618408203
931 40.932403564453125
932 40.94334411621094
933 40.92143630981445
934 40.9327278137207
935 40.922019958496094
936 40.934322357177734
937 40.92058181762695
938 40.92997741699219
939 40.91799545288086
940 40.92168045043945
941 40.915496826171875
942 40.911827087402344
943 40.91210174560547
944 40.91031265258789
945 40.90728759765625
946 40.909610748291016
947 40.90523910522461
948 40.917598724365234
949 40.90797805786133
950 40.92169189453125
951 40.91349792480469
952 40.90373229980469
953 40.91371536254883
954 40.913360595703125
955 40.91090393066406
956 40.908233642578125
957 40.90796661376953
958 40.90784454345703
959 40.90428924560547
960 40.90233612060547
961 40.89424133300781
962 40.903560638427734
963 40.89834213256836
964 40.890567779541016
965 40.89088821411133
966 40.890174865722656
967 40.896453857421875
968 40.897300720214844
969 40.889373779296875
970 40.89697265625
971 40.89498519897461
972 40.90333557128906
973 40.9049186706543
974 40.88907241821289
975 40.897438049316406
976 40.899349212646484
977 40.88850021362305
978 40.89493179321289
979 40.878849029541016
980 40.886966705322266
981 40.89059829711914
982 40.88986587524414
983 40.90074157714844
984 40.88678741455078
985 40.89237976074219
986 40.888328552246094
987 40.88460922241211
988 40.891021728515625
989 40.893646240234375
990 40.881473541259766
991 40.88935089111328
992 40.88579559326172
993 40.88222885131836
994 40.892173767089844
995 40.882301330566406
996 40.8866081237793
997 40.88020324707031
998 40.88868713378906
999 40.882789611816406
1000 40.88542556762695

about SupConLoss

appreciate of your great work! I wanna figrue out contrast_count = features.shape[1],the value of fetures.shape[1] is n_views,and I
can't understand what n_views represent.Is it always eaual to 2?

SyncBN?

Hi @HobbitLong

I see you use SyncBN from apex to train with DataParallel, however, SyncBN seems to be designed with DistributedDataParallel. Could you please confirm if SyncBN works in this case?

Best,
Jizong

Missing l2-normalization in the embedding function of ResNet(dim=2048)

Hi, thank you for your easy and clear implementation of "Supervised Contrastive Learning".

While reading the original paper and comparing code between the original repository of Google Research(https://github.com/google-research/google-research/blob/master/supcon/models.py) and yours, I became to have a question.

As noted in the original paper, before fed into the projection head, the output of the encoder should be l2-normalized, however, I couldn't find any l2-normalization process in your SupConResNet implementation in resnet_big.py.

Is there any meaningful performance difference between yours and the original paper due to this?

How to understand the figure 4

Hi, I found that the symbols in the figure 4 in the paper are similar to Candlestick chart in the stock market, but I don't know how to understand it, could you please describe it?

Why are the first few losses all the same?

Train: [1][10/196]	BT 0.429 (1.060)	DT 0.000 (0.093)	loss 8.909 (8.925)
Train: [1][20/196]	BT 0.429 (0.744)	DT 0.000 (0.047)	loss 8.909 (8.917)
Train: [1][30/196]	BT 0.426 (0.639)	DT 0.000 (0.031)	loss 8.909 (8.914)
Train: [1][40/196]	BT 0.430 (0.587)	DT 0.000 (0.024)	loss 8.909 (8.913)
Train: [1][50/196]	BT 0.429 (0.555)	DT 0.000 (0.019)	loss 8.909 (8.912)
Train: [1][60/196]	BT 0.431 (0.534)	DT 0.000 (0.016)	loss 8.909 (8.912)
Train: [1][70/196]	BT 0.433 (0.520)	DT 0.000 (0.014)	loss 8.909 (8.911)
Train: [1][80/196]	BT 0.428 (0.508)	DT 0.000 (0.012)	loss 8.909 (8.911)
Train: [1][90/196]	BT 0.431 (0.500)	DT 0.000 (0.011)	loss 8.909 (8.911)
Train: [1][100/196]	BT 0.431 (0.493)	DT 0.000 (0.010)	loss 8.909 (8.911)
Train: [1][110/196]	BT 0.429 (0.487)	DT 0.000 (0.009)	loss 8.909 (8.911)
Train: [1][120/196]	BT 0.430 (0.482)	DT 0.000 (0.008)	loss 8.909 (8.910)
Train: [1][130/196]	BT 0.430 (0.478)	DT 0.000 (0.007)	loss 8.909 (8.910)
Train: [1][140/196]	BT 0.432 (0.475)	DT 0.000 (0.007)	loss 8.909 (8.910)
Train: [1][150/196]	BT 0.429 (0.472)	DT 0.000 (0.006)	loss 8.909 (8.910)
Train: [1][160/196]	BT 0.435 (0.470)	DT 0.000 (0.006)	loss 8.909 (8.910)
Train: [1][170/196]	BT 0.435 (0.467)	DT 0.000 (0.006)	loss 8.909 (8.910)
Train: [1][180/196]	BT 0.428 (0.465)	DT 0.000 (0.005)	loss 8.909 (8.910)
Train: [1][190/196]	BT 0.429 (0.464)	DT 0.000 (0.005)	loss 8.909 (8.910)
epoch 1, total time 92.35
Train: [2][10/196]	BT 0.428 (0.565)	DT 0.000 (0.148)	loss 8.909 (8.909)
Train: [2][20/196]	BT 0.433 (0.498)	DT 0.000 (0.074)	loss 8.909 (8.909)
Train: [2][30/196]	BT 0.430 (0.476)	DT 0.000 (0.050)	loss 8.909 (8.909)
Train: [2][40/196]	BT 0.429 (0.464)	DT 0.000 (0.037)	loss 8.909 (8.909)
Train: [2][50/196]	BT 0.432 (0.458)	DT 0.000 (0.030)	loss 8.909 (8.909)
Train: [2][60/196]	BT 0.437 (0.453)	DT 0.000 (0.025)	loss 8.909 (8.909)

Thanks for sharing your code. I run your code of "(2) Supervised Contrastive Learning
Pretraining stage" and it shows the above training losses. I wonder why there are the same first few losses.

Thanks in advance.

Supervised contrastive loss saturation from the beginning on custom data

Hi,
Thank you for the great work.

I am trying to run supervised contrastive loss on my custom dataset of images. It seems that loss is stuck at 6.920 from the beginning and 30 epochs are already done. The batch size is 64 due to GPU memory constraints. Can you tell me why it might be the case?
The data has around 17000 samples and 6 classes (somewhat imbalanced) and I am just on the training stage 1 for now but loss has begun and stuck with 6.920. Any help woule be appreciated.

SupContrast with Moco trick

Hi, thank you for sharing the nice work. I notice the result of SupContrast with Moco trick on ImageNet 79.1. Do you have the plan to push the code here? It would be helpful to see that. Thanks!

loss saturate after several iteration

Hi, thanks for your sharing.
I try to test your loss funtion, but is constant after some times.

the problem is that learning is going to wrong direction.
dot product of model's output is all, it means all vector is grouped together regardless of label?

what should I check??
thanks.

-> loss
1.9459104537963867
-> label
tensor([3, 5, 0, 1])
-> dot production of augmented data's output
tensor([[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000]],

1.9459099769592285
tensor([4, 4, 3, 3])
tensor([[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000],
[1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000, 1.0000]],
grad_fn=)

Acc on Cifar 100 with bsz 256

First of all, thank you for the great work! your repo is really helpful for me

I have been training the network with your code, but the performance is much higher than your report. Could you check it is normal?

At pretraining stage, my experimental setting is same as readme EXCEPT for bsz (1024 --> 256 due to the lack of GPU)
and linear evaluation stage, it is extremely equal as your guide.

but my accuracy is almost 87% (your report is 76.5)

below is my tensorboard graphs

Could you give your opinion?

Thank you !

Visualization results with t-sne

Hello.
Thank you for providing me with great code.
I would like to see the results of the visualization when I actually run it myself.
It would be great if you could provide me with the code to visualize using t-sne.
Thank you for your help.

About the experimesnt of 'Effect of Number of Positives'

How did you do this experiment? That is, how to control the number of positive samples? My understanding is that when the number of positive samples is 5, that is, when the batchsize is set to 5000, because 5000/1000=5, the result is 78.8. But the batch size set in the paper is 8192, and the performance obtained is 78.8.

Loss saturates

I tried training SimCLR on CIFAR10 with the exact config that you've provided in the docs, except a 128 batch size, and without SyncBN since I'm running it on Colab.

The loss saturates at 28.xxx about 25 epochs. I've tried adjusting the learning rate but no difference. Is this because of the smaller batch size? Can you please help me with this?

ImageNet

Does this implementation currently work for ImageNet, or is there some more work that needs to be done (i.e. MoCo trick)? If it does, is 1000 epochs enough, and approximately how long did this take to train?

When do I stop training?

Thank you for the amazing code.

It seems that the loss saturates quite fast.
If that's the case, when should I stop training?

Do I stop it as ordinary DNN networks as its loss saturates?

Thanks in advance!

DistributedDataParallel

Is there a way to make SupContrast work for DistributedDataParallel? By default each worker can only see its own sub-batch so the inter-sub-batch relationship of the samples will be utilized.

Final output of the loss

Hi, I am wondering what is the point for the following line? And can I just simply call loss.mean()?

SupContrast/losses.py

Line 96 in d566677

loss = loss.view(anchor_count, batch_size).mean()