universalconceptualcognitiveannotation/universalconceptualcognitiveannotation.github.io

universalconceptualcognitiveannotation.github.io's Issues

Add citations

• Ackermann L., Neuberger J., Jablonski S. (2021) Data-Driven Annotation of Textual Process Descriptions Based on Formal Meaning Representations. In: La Rosa M., Sadiq S., Teniente E. (eds) Advanced Information Systems Engineering. CAiSE 2021. Lecture Notes in Computer Science, vol 12751. Springer, Cham. https://doi.org/10.1007/978-3-030-79382-1_5
• Paths to Relation Extraction through Semantic Structures. Jonathan Yellin and Omri Abend. Findings of ACL 2021.
• Great Service! Fine-grained Parsing of Implicit Arguments. Ruixiang Cui and Daniel Hershcovich. IWPT 2021.

Action items for benchmarking UCCA

• Build a webpage similar to https://nlpprogress.com/english/semantic_parsing.html#ucca-parsing where we: (1) detailed description of the official evaluation protocol (for different corpora?) including eval scripts + versions, normalization, dataset versions etc.; (2) a leader-board with parser outputs, sorted by UCCA official score and another column where they are evaluated on the MRP metric; (3) a bottom part of the page with links to other (unofficial or legacy) exp setups and corresponding leader boards.
• Post the description in (1) as a file in the ucca code repo. huji-nlp/ucca#92
• Improve UCCA score to more sensible handle unary expansions / multiple categories over the same edge. This will become the new official score. Ask participants of the semeval shared task and conll shared tasks whether they'd like to re-evaluate their systems and post their scores. huji-nlp/ucca#91
• Run the new script on the MRP 2019 and 2020 submitted UCCA parses, after converting them from JSON to XML.