gasolsun36/tog

[ICLR 2024] Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph

tog's Issues

Request for input-output text of LLMs

Hi, there

I appreciate the work you've put into this project.

I kindly request that you upload the input-output text data from LLMs during the "Thinking" process. This will provide a deeper understanding of the model's thinking process and its responses. It can also assist in identifying any biases or limitations in the model's output.

Thank you in advance for your consideration.

How can I ensure that Freebase is installed correctly?

What's the correct result after running the Test example?
My result is {'head': {'link': [], 'vars': ['name3']}, 'results': {'distinct': False, 'ordered': True, 'bindings': []}}.
I think there is some mistake, but I follow the readme in the "Freebase" folder.

Query Regarding Discrepancy Between Paper and Code

About the cost

How much does it cost to call the GPT3.5API if the results of the experiment need to be replicated?

CWQ Ground Truth

Where did you obtain the ground truth answers for the CWQ dataset?
I just reran the provided SPARQL queries on my installation of Freebase and I am getting 81.11% answers correct.
Shouldn't I be getting 100%?

Some bugs while reimplement

Hi, I found some bug while reimplement ToG:

• ./ToG/utils.py line 155: there is a typo of missing underline between “pre” and “head”
• ./ToG/README.md lines 12 and 23: duplicated argument name, I think the duplicated one should be temperature_reasoning.
• ./ToG/prompt_list.py lines 14 and 20: typo, it should be Entities instead of Entites. Does it affect LLM’s performance?
• ./ToG/utils.py lines 105-110: why the code called llama server when ”llama" not in engine.lower()? I think it should call GPT 3.5-turbo in this situation.

Have you tried running it on Windows?

Have you tried running it on Windows?

Some confusion about the data set

Is this method validated on relatively complex knowledge graph inference datasets such as win18rr or nell datasets? Recent papers have generally not tested on these datasets, is this because the previous dataset was too complex or because of other considerations?

Question about CWQ entity linking detail steps.

hi, thanks for your talent work.
I see that topic entities in CWQ have already been linked to Freebase and Wikidata. In the paper, it says"TOG first prompts LLMs to automatically extract the topic entities in question and gets the top-N topic entities. " But the preprocessing details are missing in the code.

Request for CMD to Reproduce Results on WebQSP with Chat-GPT-3.5

Hello,

I am trying to reproduce the results on WebQSP using Chat-GPT-3.5, but I am facing some challenges with the provided command. I find that the exploration process is sensitive to the temperature. I was wondering if you could provide the correct command to reproduce the results successfully.

I currently use python main_freebase.py --dataset webqsp --max_length 256 --temperature_exploration 1 --temperature_reasoning 0 --width 3 --depth 2 --remove_unnecessary_rel True --LLM_type gpt-3.5-turbo --opeani_api_keys sk-xxxx --num_retain_entity 5 --prune_tools llm

Thank you!

Question about the datasets

Hi! Thank you for your work!

I ran the test example without installing the Freebase dataset and encountered an error at "results = sparql.query().convert()", which says: "{URLError} <urlopen error [Errno 111] Connection refused". When I run main_freebase.py, it also reports "urllib.error.URLError: <urlopen error [Errno 110] Connection timed out". Can this issue be resolved by installing the Freebase dataset, or is it caused by an internet connection error? Thank you!

Additionally, is it possible to avoid installing the dataset? My server doesn't have 400GB of RAM, making it very difficult for me to run the program. Projects like RoG (Reasoning on Graph) can run on WebQSP and CWQ with only a few hundred MBs of memory. Is it possible to run ToG with a single dataset (like WebQSP) with less memory consumption? Thank you!

Prompt for Generating topic entities

Hi! Thank you very much for your work.
In your paper you write "ToG first prompts LLMs to automatically extract the topic entities in question and gets the top-N topic entities", but could not find the prompt, which you used to extract those topic entities?
In the repository they seem to be already hard-coded in the dataset JSONs due to a previous pre-processing step?

Could you describe how you extracted the topic entities?

Estimated Memory Footprint of Virtuoso Database Post-Import of 125GB Filtered Freebase Data

Hello,

I have a question regarding the FilterFreebase data set which is approximately 125GB in size after filtering. Can you please advise on how much memory the virtuoso.db file might approximately occupy after the data has been imported into the Virtuoso database?

Thank you for your assistance.

Linking topic entities to KG

Hi,

Once you generate the initial topic entities using the LLM, how do you find the corresponding entities in the KG? The LLM could output entities that do not exist in the KG. Also, what is the prompt used to find the topic entities?

Thanks,
Mohammad

How can I ensure that Freebase is installed correctly?

What's the correct result after running the Test example?
My result is {'head': {'link': [], 'vars': ['name3']}, 'results': {'distinct': False, 'ordered': True, 'bindings': []}}.
I think there is some mistake, but I follow the readme in the "Freebase" folder.

Issue with id2entity_name_or_type Function: SPARQL Query Handling

Hi,
Thanks for sharing the code.

The id2entity_name_or_type function seems not function as expected due to variable overwriting.

ToG/ToG/freebase_func.py

Line 35 in 5badd58

def id2entity_name_or_type(entity_id):

Is that right if I rewrite into this ? Thanks.

def id2entity_name_or_type(entity_id): sparql_txt = sparql_id % (entity_id, entity_id) sparql = SPARQLWrapper(SPARQLPATH) sparql.setQuery(sparql_txt)