digitallinguistics/tags2dlx

A JavaScript (Node.js) library that converts a tagged (monolinear) text to DLx JSON format

DLx Organization Profile

This repository contains the code for the Digital Linguistics (DLx) profile page.

tags2dlx's Issues

validate returned text against DLx Text schema

This is currently blocked because the main build of the DLx spec isn't working, due to a broken link to the GeoJSON spec.

Waiting on digitallinguistics/data-format#330

option: compact

Add a Boolean compact option, which allows the user to set the JSON output to compact, with no unnecessary white space.

Internally, this will set the space argument of JSON.stringify to 0 rather than 2.

should return a DLx Text object

The tags2dlx library should return a DLx Text Object, as a POJO

option: tag name

The user should be able to pass an option to the library (tagSetName) which provides the name of the tag set. This will be used to set the property name of the Tabs object in the DLx text:

"tags": {
  "penn": "N"
}

If this option is not provided, the tags will be added individually with their value set to true:

"tags": {
  "N": true
}

option: postprocessors

The user should be able to pass a postprocessors hash as an option to tags2dlx which allows them to specify postprocessors at the text, utterance, and word level.

The library itself should use this functionality for its own postprocessing.

set up project board

Set up a project board for managing tasks in this repo, with the following (automated) columns:

• wish list
• to do
• in progress
• done

option: NDJSON

Add a Boolean NDJSON option which allows the user to specify that the output should be in NDJSON format, where each line in the resulting NDJSON file represents an utterance.

setup GitHub pages

• setup DLx subdomain
• change repository settings
• update URL of repo
• add link to DLx developer documentation

command line script should use CommonJS

documentation should use directory rather than file as example

Currently the documentation shows this as an example command in several places:

tags2dlx text.txt

Change this to a directory instead.

should tokenize text into utterances

The tags2dlx library should tokenize a text into utterances based on punctuation.

set up Zenodo

• enable Zenodo for repository
• make a release
• add Zenodo badge and citation to readme

publish library to npm

• set up automated deployment based on GitHub releases
• remove GitHub releases / version badge(s) from readme
• add npm version badge to readme
• make the first GitHub release

words should be DLx Word Token objects

Each word object in the returned text should adhere to the DLx Word Token schema.

• transcription
• translation (empty)
• tags (with the tag attached to that token)

update readme

Go through the readme checklist

• introduction
• example of a converted sentence
• links to DLx JSON format
• basic usage
• version - latest stable release of Node
• issues
• author
• license
• badges
  • Travis CI status
  • GitHub issue count
  • license
  • GitHub stars
• running tests
• contributing

do not run file conversions in parallel

Running all the file conversions in parallel throws an error once too many files are open. Run them in sequence using for await of instead.

option: tag separator

The tags2dlx function should accept a tagSeparator option, which allows the user to specify one or more separators that delimit the tag at the end of the word from the word itself.

set up Jasmine

• tests should run on Node stable release
• add to Travis CI

option: utterance separators

The tags2dlx library should accept an utteranceSeparator option which allows the user to specify which characters should be used to separate utterances.

Possibly allow this to be a regular expression?

Defaults to basic English punctuation.
https://en.wikipedia.org/wiki/Punctuation_of_English

publish to GitHub package registry

use GitHub Actions rather than Travis CI

Replace Travis CI status badge with GitHub status badge

option: preprocessors

The user should be able to pass a preprocessors hash as an option to tags2dlx that allows them to specify preprocessors for the text, utterance, and word.

The library itself should use this functionality for its own preprocessing.

add citation file

Add a CITATION.cff file to this repository.

option: metadata

The user should be able to pass a metadata option to this library that includes other metadata about the text, such as the title.

add transcription property to utterance

Extract the original transcript / transcription (which? both?) for each utterance, and include it in the transcription / transcript property of the utterance.

You'll probably need to keep an in-memory version of the text, which you progressively trim each utterance from the beginning of, using a regular expression to find the next instance of the current utterance separator.

should tokenize utterances into word tokens

The tags2dlx library should tokenize utterances into word tokens (based on white space by default).

support different languages

Technically, this library only currently supports English corpora, because the output sets the transcription property as a simple string. According to the DLx format, when the transcription field is just a string, it should be interpreted as English.

Add a language option to the tags2dlx function which allows the user to specify the language of the corpus. English is assumed by default.

option: word separator

The tags2dlx library should accept a wordSeparator option which allows the user to specify one or more characters to use as word separators.

set up Travis CI

• run Jasmine tests in Node.js

add command line interface

The user should be able to run this library from the command line. The command line interface should accept 1 required argument: the path to the file or folder to convert.

The command line script should either:

• convert the provided single file to JSON and save the new file alongside the original
• recurse the provided directory and convert each file as above

treat double new lines as a new utterance

DaFoDiL output not formatted correctly

Attributes whose values are Transcriptions should always be objects rather than strings.

master > main

Rename master branch to main.

set up code scanning

set up code scanning

add test for correct year in copyright, license, and citation

add test for correct year in copyright, license, and citation

library should be platform-independent

Check that tests pass on Windows, Mac, and Linux.

option: punctuation

The user should be able to provide a set of punctuation characters to ignore.

update punctuation option in readme

The default value for the punctuation option needs to be updated in the readme.