txsun1997/black-box-tuning

ICML'2022: Black-Box Tuning for Language-Model-as-a-Service & EMNLP'2022: BBTv2: Towards a Gradient-Free Future with Large Language Models

black-box-tuning's Issues

Can the user-side can obtain the final prompt in literal?

I found this optimization is about vector after user obtain the results from server. I wander if there any possible to get the final prompt text to help other people to design prompts

结果复现问题

您好，我在sst2，yelpp和agnews上的(seed=42)复现结果为87.61(90.33)，91.00(92.86)，82.53(85.28)。
括号中是论文报告的结果，我所使用的机器是V100，配置如下：
python -u bbt.py
--task_name XXX
--n_prompt_tokens 50
--intrinsic_dim 500
--k_shot 16
--device "cuda:3"
--seed 42
--loss_type "ce"
--cat_or_add "add"
--budget 8000
--print_every 50
--eval_every 100
请问是我的参数设置有问题吗？或者其他什么原因导致的结果上的差异吗？

add web demo/models/datasets to ICML organization on Hugging Face

Hi, congrats on acceptance at ICML 2022 for Black-Box-Tuning. We are having a event on Hugging Face for ICML 2022, where you can submit spaces(web demos), models, and datasets for papers for a chance to win prizes, after joining the organization using this link https://huggingface.co/organizations/ICML2022/share/BpynfJtfsOTktlmXYoKNqqCnyufKLFXuay, let me know if you need any help with the above steps, thanks

Support for Flan-T5 models

Hello, I see that BBTv2 supports a couple of t5 models

Would it be easily extendable to support Flan-T5 models as well ?

关于Dbpedia

/remote-home/txsun/zfhe/dbpedia_csv.tar.gz 这个文件可以提供下吗？使用datasets自动下载dbpedia出问题了：
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100.0)
谢谢哈～

Why not calculate prompt on the server side?

In the paper, it is assumed that the user can access only an inference API provided by the server, and thus we need a solution to calculate the prompt on the user side. This limit is from the requirement of keeping PLM parameters secret.

Why not calculate the prompt on the server side. Returning the prompt, or, the gradient on the prompt, does not leak PLM parameters, too.

We do not prefer the above solution, because of additional burden at the server side?

Truncation Length

Hi, very nice work. Also, we are currently trying to reproduce your work as our baseline.

Can I ask you what strategies do you try with in our Yelp P. dataset? Especially for very long sentences. In my current experiments, it tells me that I have to perform truncation...

Thanks!

DeepBBT : the outputs of modified Roberta don't have hidden states

Hello, there is a bug in RobertaL Model ouput when I run deepbbt because it do not have hidden state even the config.output_hidden_states = True (Line439).

Then I found the Line 975 of deep_modeling_roberta.py only output a dict with logit rather than the original MskedLMOutput, which leads to the above bug.

When I try to directly replace the logit of MaskedLMOutput to fix it, it suggest the inner model also do not output hidden state. I would appreciate if you could help to fix that !

How big is the improvement in NLG about BBT v2

In the paper, it seems all NLU tasks, how about NLG tasks?

modeling_roberta.py line 632

TypeError: add_code_sample_docstrings() got an unexpected keyword argument 'tokenizer_class'
How to solve it?

Comparison with gradient-based methods on large models

It would be very useful a comparison with performances of gradient-based methods (lora,p-tuning,prompt tuning, etc.) on the same datasets and using the same models (i.e., t5-xxl) commonly used in literature. For instance you can compare with results quoted here https://aclanthology.org/2021.emnlp-main.243.pdf .
It is not clear from your manuscript whether or not the proposed approach is still competitive with (very) large models (larger than roberta-large), where it is well known that gradient-based models are performing very well.

Thank you, and congratulation for the very very interesting method!

Can't replicate results of BBTv2 paper

Hi, I tried your BBTv2 code but failed to get comparable results as reported in your paper.

In my case, using the command

python deepbbt.py   --model_name "roberta-large"  --task_name "snli"   --n_prompt_tokens 50   --intrinsic_dim 500   --k_shot 16   --device "cuda:0"   --seed 42   --loss_type "ce"   --cat_or_add "add"   --random_proj "normal"   --sigma1 1   --sigma2 0.2   --popsize 20   --bound 0   --budget 8000   --print_every 50   --eval_every 100

gives the following results

Done. Elapsed time: 39.49383888641993 (mins)
Evaluate on test data...
Evaluate data in 75.54 seconds!                                                                                                                                                                                     
[tester] 
SNLIMetric: acc=0.5509975570032574, hinge=2.8394456026220167, ce=11.656479801339513
Test acc: 0.551

which is higher than other gradient-free baselines but much smaller that the number reported in your paper (60.62).

I'm wondering why. Do I need to tune the random seed?

very interested in moss

I am in HuaWei，and I have the same age with you。I am very interested in moss. Can I get your contact information or you contact me via @1697540432qq.com or WeChat@19949292778. Very appreciated if you contact me！

Instead of pip install 'sklearn', please pip install scikit-learn