yihongdong/codep

codep's Issues

Push Down Automata instruction

Dear Yihong,

I hope this message finds you well. I recently had the opportunity to delve into your work, and I must say I am genuinely impressed by the depth and insights it offers.

However, I had a few questions regarding a specific aspect that I couldn't find detailed in the paper. Would it be possible for you to provide some guidance or specific instructions on how to construct a PDA for a programming language? I believe this would greatly enhance my understanding and application of your research.

I truly appreciate your time and consideration in addressing my query. Looking forward to your insightful response.

Warm regards,
Zhirong

Question about runtime for generation

Dear Yihong,

I hope this message finds you well. I recently had the opportunity to delve into your work, and I am very impressed by the insights it brings.

However, I have a few questions regarding the time taken for generation. I was trying to run Llama-7b to generate Human-eval dataset and see how it works with Llama models. I tried to run the PDAConstraintSamples folder by using the PDAConstraints for generation. When I try running this, it is taking more than I expected. It takes more than 30 minutes to generate the answer for a single prompt. I believe that this has happened because of my lack of understanding of your code, and mistake during the reproduction. Would it be possible for you to provide me the time taken for generation? Moreover, it will be very helpful if you can provide me the generation time taken for other models such as codeGen and CodeT5 with multi-humaneval dataset.

Just to give you more insight about my problem, below is the prompt from multi-humaneval dataset and also the token that I printed while running the code:

            token_name = self.tokenizer.convert_tokens_to_string(self.tokenizer.convert_ids_to_tokens([token_id]))
            print("Tokens after modification:", token_name)

task id is: HumanEval/0
The task id is: HumanEval/0:: prompt is: from typing import List

def has_close_elements(numbers: List[float], threshold: float) -> bool:
        """ Check if in given list of numbers, are any two numbers closer to each other than
        given threshold.
        >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
        False
        >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
        True
        """

Tokens after modification: def
Tokens after modification: NAME
Tokens after modification: (
Tokens after modification: def
Tokens after modification: NAME
Tokens after modification: def
Tokens after modification: def
Tokens after modification: NAME
Tokens after modification: def
Tokens after modification: def
Tokens after modification: NAME
Tokens after modification: (
Tokens after modification: NAME
Tokens after modification: **
Tokens after modification: )
Tokens after modification: *

If there is anything that you need from me, please let me know! I truly appreciate your time and consideration in addressing my question!

Sincerely,
Hangoo Kang.