bahuia/sscl-text2sql

sscl-text2sql's Issues

Some Questions About Evaluation Methods?

Hello, author. Thank you for your work. We have some questions about the final evaluation method. Could you please explain how to calculate the results in Table 1 of the paper after obtaining the four lists from the code below?

SSCL-Text2SQL/train_sfnet.py

Line 44 in 99a28dd

avg_acc_list, whole_acc_list, bwt_list, fwt_list = trainer.train_with_teacher()

Specifically, do we need to calculate the average of the ten values in avg_acc_list and whole_acc_list separately to obtain ACC_a and ACC_w in Table 1? Furthermore, how do we calculate BWT and FWT in Table 1 from bwt_list and fwt_list?

Request for Assistance: WikiSQL Testing Issue

Hi author!
First of all, I'd like to express my gratitude for your hard work on this project. Your dedication is greatly appreciated.
Recently, I encountered an issue while testing your code on WikiSQL. The results were not as satisfying as I had hoped. I attempted to diagnose and rectify the problem myself but regrettably, I was unable to find a solution. To my surprise, I discovered that another user has also faced the same issue and it remains unresolved.
I apologize for any inconvenience, but I was wondering if you could spare some time to look into this matter. Your expertise and insights would be invaluable in resolving this challenge.
Thank you once again for your contribution and assistance. I look forward to your response and any potential solutions you may have.

您好, 很高兴看到这个项目

由于chatgpt的成功, 受其影响我自己最近也在尝试开发关系数据问答有关的软件。之前也有过类似的想法，但因各种原因，没有尝试取落地。看到您的项目与此相关，不知能否借鉴或交流。非常期望能获得一些书籍或、论文、软件框架、数据集方面的指点。谢谢

Some Questions Regarding the WikiSQL Results

Hi author! @Bahuia
Thank you very much for your fix. However, we have some questions regarding the results on WikiSQL and would appreciate your insights.

1. In the revised WikiSQL results, we noticed that all values for the BWT metric are positive, which we also observed in our own replication. This seems somewhat unexpected and inconsistent with the results reported in the paper, unless the model demonstrates a very strong generalization ability.
   

   SSCL-Text2SQL/wikisql.log

   Line 12543 in e55e111

   Backward Transfer: [-inf, 0.033, 0.017, 0.023, 0.031, 0.034, 0.023, 0.022, 0.012, 0.018]

2. Simultaneously, we noticed that on WikiSQL, SFNET's performance even surpasses the ORACLE method, which appears to be a quite intriguing discovery. However, the reasons behind this phenomenon are not entirely clear to us. Could you please provide further insights on this matter?
   

   SSCL-Text2SQL/wikisql.log

   Line 12544 in e55e111

   Forward Transfer: [-inf, 0.445, 0.565, 0.584, 0.606, 0.62, 0.632, 0.639, 0.649, 0.655]

Originally posted by @tom68-ll in #3 (comment)

Some questions aboout evaluation metrics

Dear Author,
We have noticed that in the SFNET code, you have used semSQL's intermediate representation as the evaluation benchmark instead of the final SQL, which is not the ultimate goal of the semantic parsing task. Therefore, we attempted to follow the method provided in IRNET to convert semSQL into formal SQL, we obtained the following results on the SQL Exact Match metric:
ACC_a = 54.67; ACC_w = 55.78; BWT = -0.46; FWT=41.09
Although it is normal for the results to vary, we observed significant changes in the model's performance on the BWT and FWT metrics (originally BWT = -1.0; FWT = 45.9). Notably, the model's performance on catastrophic forgetting almost disappeared. Could you explain why this is the case?