Comments (5)
Hi,
Regarding the footnote, I agree that we should have been more explicit in indicating which methods were performed on the USPTO-full and USPTO-MIT datasets. The current form of the table may not be sufficiently clear for readers.
I hope that this issue created on GitHub may help to compensate for the confusion. Thank you for your valuable feedback.
Additionally, I appreciate the reference to the paper by Zhong et al., which highlights the performance difference between the two datasets.
Best.
from retrotrae.
Just to add one thing: In ref 33 (AutoSynRoute), they evaluated neuralysm as 47.8% for the MIT dataset. Dai et. al. evaluated neuralysm at 35.8% on the full dataset. Given that both authors implemented neuralysm in the same way, it seems that the full dataset is way harder to perform well on.
from retrotrae.
Hello,
I appreciate your interest and inquiry regarding the comparison of our approach to other existing methods.
While I understand your concern, I would not fully concur with the assertion that "the evaluation on a dataset 2x the size is surely more difficult." The relationship between dataset size and accuracy is multifaceted, influenced by various factors, including the quality of the additional data and the training procedure employed.
To illustrate, one could argue that evaluating performance on the USPTO-MIT dataset should be considerably more challenging than on the well-curated USPTO-50K dataset, given that the former is approximately ten times larger.
However, this argument is not corroborated by the results reported in very same ref 33, which demonstrated that Segler-Corey achieved 47.8% accuracy on the MIT dataset versus 38.7% on the 50K dataset, and AutoSynRoute attained 54.1% accuracy on the MIT dataset compared to 43.1% on the 50K dataset.
The discrepancy between the neuralysm results you mentioned could potentially be explained if the USPTO-full dataset is noisier than the USPTO-MIT dataset.
As you pointed out, the lack of consensus on dataset selection can indeed result in inconsistencies when comparing various studies. We have acknowledged this issue by including a footnote stating that some of the results in the comparison table are derived from either the USPTO-full or USPTO-MIT datasets.
from retrotrae.
Hello,
Thank you very much for your prompt reply.
Indeed, you are correct that I was wrong in inferring that a larger dataset would be harder to perform on. I should have referred, as you pointed out, to the noisiness/sparsity of the dataset.
In Dai et. al., they say that:
Despite the noisiness of the full USPTO set relative to the clean USPTO-50k, our method still outperforms the two best baselines in top-k accuracies.
Possibly, this is the indication that this dataset is harder to perform on (as it was curated by Dai et. al.).
Regarding the footnote, are you referring to this?
The results are based on either filtered MIT-full [46,47] or MIT-fully atom mapped [15] reaction datasets.
If so, there are no superscripts/subscripts which indicate which of the algorithms are performed on USPTO-full and USPTO-MIT. That is why I got confused and possibly other readers will get, too.
The performance difference between the two datasets becomes more evident in this paper: "Root-aligned SMILES: a tight representation for chemical reaction prediction" by Zhong et. al. Here we can see a performance decrease of (12-15%), which is rather large. Of course, your approach might not exhibit the same; however, I still believe that the comparison in Table 3 is misleading for the reader.
from retrotrae.
Hi,
Thanks for the clarification. Again, well done for this very interesting framework.
from retrotrae.
Related Issues (9)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from retrotrae.