chaos96/fourierft

• 👋 Hi, I’m @Chaos96
• 👀 I’m interested in ...
• 🌱 I’m currently learning ...
• 💞️ I’m looking to collaborate on ...
• 📫 How to reach me ...

fourierft's Issues

Request for Training Scripts of Baseline Models (LoRA, AdaLoRA)

Hello!

Thank you for your excellent work on this project! I have noticed that the repository includes a bash script for training FourierFT. Could you please share the training bash scripts for the baseline models, such as LoRA and AdaLoRA?

For some issues related to natural language generation tasks

Thank you for your code. Could you please provide the code for evaluating using GPT-4 on the MT-Bench and Vicuna datasets?

Thanks for your answer, but there is another question when I update your code: I can not run on multi-GPUs

Can you please give a comparison between fourierft and lora in terms of time complexity?

For example, how long does it take to finetune llama2-7B&13B with fourierft? Using the similar experimental setting, how long does it take to finetune llama2-7B&13B with lora?

Thanks.

Inquiry About Code Release Date and Schedule

Hello @Chaos96 ,

Thank you all for your work.

Given the high interest in your project, I was wondering if there is a specific timeline for when the code might be released?

Best regards

Possible Typo in Fourier Layer Code

Possible Typo in Fourier Layer Code

Hey, I noticed that the scale is set to a constant 300.

Is this a typo? Should the scale be self.scale[adapter] instead of 300?

/peft/src/peft/tuners/fourier/layer.py#L306

weight = torch.fft.ifft2(torch.sparse.FloatTensor(indices, spectrum, [self.in_features, self.in_features]).to_dense()).real * 300

Hi,Thanks for your answer, but there is another question when I update your code: I can not run on multi-GPUs

Table 1 Mistype?

can your share your detailed environment

can you post your conda environment of thi sproject?

hi, thanks for your contribution.

you have changed the SFT.py, but I have a question,how to find the dataset way /apdcephfs_qy3/share_1594716/bingzhe/dataset/alpaca-cleaned

Inquiry About the Hyper-Parameters for Replication

Inconsistent Hyper-Parameters Between Paper and Code

Hi there,

I found some inconsistencies in the hyper-parameters between the paper and the code. Here are the details:

1. Issue with n for Roberta-base on GLUE:

   • In Table 1, the value of n for Roberta-base is 200.
   • In Table 9, the value of n for both Roberta-base and Roberta-large is 1000.
   • In the script, n for Roberta-base is set to 1000.
   • Screenshots:
     • 
     • 

2. Issue with LLaMA's Learning Rate and Scaling:

   • In Table 11, the learning rate and scaling are set to 3E-3 and 16, respectively.
   • In the scripts:
     • The learning rate for llama1-7B is 0.1
     • The learning rate for llama1-13B, llama2-7B, and llama2-13B is 0.05.
     • The scaling is set to 300.
   • Screenshot:
     • 

Could you please clarify the correct hyper-parameters to replicate the results from the paper?

Thank you!

Question about F

Thank you very much for providing the open-source code.
I have a question about the initialization of F.
F, as a sparse matrix with a shape of (in_dim, out_dim), its non-zero indices are determined by E, where the values of E range from [0, in_dim*out_dim). This seems like it could cause index out-of-bounds errors.

Could you please tell me what details I might be missing?