letuananh/bela

👸 BELA - A pathway for creating and analysing multi-lingual transcripts using BELA convention and ELAN software

Hi there 👋

Le Tuan Anh is a semanticist who is deeply interested in well-being, languages, and free software. His main research interests are in computational meaning and theory of language. He is the author of the coolisf software, which provides computational deep semantic analysis by combining structural semantics from construction grammars and lexical semantics from ontologies in a single representation. The dissertation (2019), source code, and data are available on the Open Science Framework.

In his free time, Tuan Anh develops jamdict - a free and open-source Python 3 library for manipulating Japanese linguistic resources: Jim Breen’s JMdict, KanjiDic2, JMnedict, and kanji-radical mappings. There is an online demo available at: https://jamdict.herokuapp.com/

From Mar 2019 to Apr 2022, Tuan Anh was a research fellow at Nanyang Technological University (NTU), Singapore and contributed to the development of BELA - a pathway for creating, checking, visualizing, and analysing multilingual corpuses of natural languages.

bela's Issues

Remove ellipsis when tokenizing special tokens such as `:m:` and `:si:`

BELA v2.0.0a21.post6 tokenize() function does not process ellipsis when they are embedded in singing or mimicking segments.

E.g., applying tokenize() function on :si:walking+in+the+jungle+walking+in+the+jung... yields: ['walking', 'in', 'the', 'jungle', 'walking', 'in', 'the', 'jung...']

Proposing to remove ellipsis if ellipsis parameter is set True. Expected result would look like this: ['walking', 'in', 'the', 'jungle', 'walking', 'in', 'the', 'jung'].

Retain linguistic information after tokenize()

Suggesting a feature to retain linguistic information of bound morphemes (e.g., =s, =ing, =ed, =le) or ellipsis.
This will help differentiate between truncated words/sounds and bound morphemes in lexicon after tokenize().

For example:
le... [stuttering] let's see jiejie=s [plural morpheme] are playing with the s... [stuttering] silly bottle=le [Tamil =le morpheme added] game -> jiejie s were playing with the s silly bottle le game

In the above case, the output from bela.tokenize() does not allow us to differentiate between whether a lexical entry is a speech error or a bound morpheme.

Flag baby `:v:` language tags as errors if they appear on adult language tiers

Currently, if baby language tags (:v:vocalizations,:v:laughter, :v:airstream, :v:crying) appear on adult language tiers, it is not flagged as an error.

Proposed implementation in Python:

def _find_baby_lang_adult(belan):
    for p in belan.persons:
        if 'Baby' not in p.name:
            if 'Sibling' not in p.name:
                for u in p.utterances:
                    for c in u.chunks:
                        if c.language.startswith(':v:'):
                            errors.append(f"{belan.path} : {c} tagged as {c.language} found on {p.name} (Language) tier (should be reserved for Baby (Language) only)")

flag special tokens and ### as errors (e.g., `:si:###` and `:m:###`)

As of April 12th 2022, :si:### and :m:### are allowed in BELA but it should not be allowed