yfzhang114/adanpc

This is an official PyTorch implementation of the ICML 2023 paper AdaNPC and SIGKDD paper DRM.

adanpc's Issues

leak test data infromation

It seems to take the groud truth features from all test data.

In your AdaNPC, you take all the test data with ground truth labels (e.g., the name with out datasets) as the prototypes for inference.

AdaNPC/unsupervised_adaptation.py

Lines 327 to 328 in c280d6b

	if 'AdaNPC' in args.adapt_algorithm and str(args.test_envs[0]) not in name:
	adapted_algorithm.classifier.extend_test(z.to(device), y.to(device))

Officially, the condition and 'out' not in name should be added to avoid such leakage of test-data information.

AdaNPC/domainbed/scripts/unsupervised_adaptation.py

Lines 324 to 325 in c280d6b

	if 'KNN' in args.adapt_algorithm and str(args.test_envs[0]) not in name and 'out' not in name:
	adapted_algorithm.classifier.extend_test(z.to(device), y.to(device))

reproduing DRM results

To Reproduce results for DRM,
sh scripts/launch.sh pretrain resnet50 10 3 local DRM
sh scripts/launch.sh sup resnet50 10 3 local DRM
sh scripts/launch.sh unsup resnet50 10 3 local DRM
First of all,what specific results are these three reproducing?(Table?Figure? in the paper)
Secondly,i wonder are these commands complete?cuz I got some problems while running the second command(fail to use pertrained model)

Eventually,i'm kinda confused cuz this codebase is a mixture of AdaNPC and DRM while i'm only interested in DRM.If I only want to compare DRM as a test-time-adaptation algorithm with others, how should I do it? (If reproducing already includes this part, please ignore this question).

quick start

python -m domainbed.scripts.download --data_dir=/my/datasets/path --dataset pacs;
return D:\Anaconda3\envs\final\python.exe: No module named domainbed.scripts.download;
all the requirements are satisfied

About run error

when i want to exeute the domainbed.scripts.train.py knn algorithm，i get errors as follows：
self.classifier = MomentumQueue(self.featurizer.n_outputs, self.hparams['queue_size'], self.hparams['temperature'], self.hparams['k'], num_classes)
KeyError: 'queue_size'

so how to fix it? thanks

Why care about env{test}_out splits

When testing with test-time adaptation algorithm in unsupervised_adaptation.py, what is the meaning or usage of storing accuracies and get the best accuracy for env{test}_out ? As I think env{test}_in should be the split for testing, while env{test}_out is for test-domain model selection (according to the original DomainBed).

AdaNPC/unsupervised_adaptation.py

Lines 408 to 415 in 007ec44

	for name, loader, weights in evals:
	acc, ent = accuracy_ent(adapted_algorithm, loader, weights, device, adapt=True)
	results[name+'_acc'] = acc
	results[name+'_ent'] = ent
	if 'out' in name and str(args.test_envs[0]) in name:
	results_on_test.append(acc)
	ent_on_test.append(acc)
	adapted_algorithm.reset()

Btw I think ent should be passed instead of acc on line 414

Mis-Enabling Gradient In AdaNPC

Hi,

Thank you for your outstanding contribution. I'd like to bring to your attention a potential concern in implementation. It's possible that it might inadvertently enable gradient in adaptation (see the link below), leading to increased GPU memory consumption. Would you mind to have a check on this?
https://github.com/yfzhang114/AdaNPC/blob/282e4130fa4e88878e00197561eba508ffb81448/domainbed/adapt_algorithms.py#L510C5-L510C5

One minor code mistake

domainbed. scripts.unsupervised_adapation Line 420：
ent_on_test.append(acc)
should be
ent_on_test.append(ent)

Lack the implementation of KNN and EM

line 149 in the class KNN(Algorithm) is,

AdaNPC/domainbed/algorithms.py

Lines 147 to 149 in c280d6b

	z = self.featurizer(all_x)
	# loss = F.cross_entropy(self.classifier(z), all_y)
	loss = F.nll_loss(torch.log(self.classifier(z)), all_y)

Eq.(4) in the paper is，

Thanks for your project.

I have a question about Where is the detailed $L_{KNN}$, concretely, the codes of Eq.(4) (see above) and EM algorithm? Detailed questions are shown below:

Although line 149 in the class KNN(Algorithm) (see above) , replaces F.cross_entropy(a, b) with F.nll_loss(torch.log(a), b), I think it still does not achieve Eq.(4) function.

Because F.nll_loss(torch.log(a), b) and F.cross_entropy(a, b) have the same results, referring to How is Pytorch’s Cross Entropy function related to softmax, log softmax, and NLL.

Besides, there seems to be no implementation of EM in this project.

Train a model on source domains

python -m domainbed.scripts.train --data_dir /my/datasets/path --output_dir /my/pretrain/path --algorithm ERM --dataset PACS --hparams '{"backbone": "resnet50"}'
return File "F:\ART_1\AdaNPC-master\domainbed\scripts\train.py", line 228, in
for x,y in next(train_minibatches_iterator)]
StopIteration