ogallagher/arcaeca2-lang-stats

Scripts for analyzing languages

arcaeca2-lang-stats's Issues

Alternative for network file fetch from webserver

An environment without node JS interpreter installed should be able to run the webpage in a browser with file:// protocol by replacing network file fetch with a file upload widget.

Make code browser compatible

1. Create parent webpage.
2. Replace node file system calls with networked file import.
3. Replace module syntax where needed.

Condition for 3-phoneme hole

Isn't this condition backwards?

arcaeca2-lang-stats/FindHoles.js

Lines 108 to 112 in 9d99885

	// if "AB" occurs, and "BC" occurs, but "ABC" doesn't
	if ( (tTop[sTop] < 2) && (tBottom[sBottom] < 2) && (tPattern[sKey] > 2) ) {
	// I'm not really sure what a good metric would be for deciding if a combination "occurs" or not
	tOut.push(sKey)
	}

I think we instead want:

if (top > X && bottom > X and pattern < X) {

expandCategories always returns single phoneme sequence

arcaeca2-lang-stats/FindHoles.js

Lines 257 to 269 in 0351957

	let output = [[]]
	// TODO this method currently overwrites the output list for every subsequent code from the pattern
	for (let i = 0, ch = inputString.charAt(0); i < inputString.length; i++, ch = inputString.charAt(i)) {
	// input char is a category
	if (categories[ch] !== undefined) {
	output = multiply(output, categories[ch])
	}
	// input char is anything else
	else {
	// for every existing item in out, add this char
	output.forEach((op) => op.push(ch))
	}
	}

corresponding failing test

arcaeca2-lang-stats/test/FindHoles.js

Lines 150 to 157 in 0351957

	it('handles non trivial patterns', function() {
	let sequences = expandCategories('AB', categories)
	assert.strictEqual(
	sequences.length,
	categories['A'].length * categories['B'].length
	)
	})
	})

  9 passing (17ms)
  1 failing

  1) FindHoles
       #expandCategories
         handles non trivial patterns:

      AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

1 !== 6
      
      actual expected
      1      6

Statistical definition of 3-phoneme anomaly

Capital letters A B C each represent a phoneme class.
Pattern ABC represents all strings made of an instance of each phoneme class/category, in that order.
For example, if a1 is an instance of A and b1 is an instance of B, then a1b1 is an instance of AB.

Currently, we define a 3 phoneme hole anomaly ABC where the number of occurrences (count) of the full pattern is 0, but the counts of each sub pattern AB and BC are greater than 0. A hole/valley is unusually low count, and a hill/peak is an unusually high count.

arcaeca2-lang-stats/FindHoles.js

Line 118 in a82201a

if ( (tTop[sTop] >= 1) && (tBottom[sBottom] >= 1) && (tPattern[sKey] < 1) ) {

We can instead define valleys and peaks as statistical outliers, taking the average and then look for statistical outliers?

Start by expressing the ratio of frequency between ABC and AB + BC as a ratio: ABC / (AB + BC).

Say the average ABC/AB+BC occurrence ratio is 0.25, with a standard deviation of 0.1.
That means you should look for any individual char sequence within the ABC pattern whose ratio is less than 0.15 (valley) or greater than 0.35 (peak).

ABC/AB+BC for any one instance of the ABC pattern would be, for example:

(count of a1b1bc1 occurrences) / ((count of a1b1) + (count of b1c1))

Display frontend results in webpage

Usage instructions for backend and frontend

As of now, this issue is about half resolved.

FindHoles start/top and end/bottom sequences are incorrect

A single phoneme can be more than one character. For example: ch and sh could be English phonemes of the consonant category.

Here, the code is assuming that the 3-phoneme string is only 3 characters:

arcaeca2-lang-stats/FindHoles.js

Lines 113 to 115 in a82201a

	let sTop = sKey.substring(0,2) // the corresponding "AB"
	let sBottom = sKey.substring(1) // the corresponding "BC"
	console.log(`whole start end:\t${sKey}\t${sTop}\t${sBottom}\t|\t${tPattern[sKey]}\t${tTop[sTop]}\t${tBottom[sBottom]}`)

See proof of corresponding logs:

whole start end:        a1a1a1  a1      1a1a1   |       0       undefined       undefined
whole start end:        a1a1a2  a1      1a1a2   |       0       undefined       undefined
whole start end:        a1a1a3  a1      1a1a3   |       0       undefined       undefined
...

The third row in the above logs should instead be a1a1a3 a1a1 a1a3.