Context opcode does not match when in a capitalized word/passage

[tibbsa]

Context opcode does not match when in a capitalized word/passage

[bertfrees]

@tibbsa Thanks for the investigation and the test file, and sorry that you got no response to your message to the mailing list.

I think this might be related to #818, and the associated test file.

Both issues are caused by an inconsistency between how context rules are compiled/stored and how they are matched with input. When a particular input segment, say "OUND", is processed:

• the first two letters "OU" are downcased
• then, based on these two letters", the noback context _$l["ound"] @46-145a rule is selected as a candidate matching rule
• then the _$l["ound"] test is actually performed on "OUND", returning in no match found

[bertfrees]

My suggestion is to fix the second step:

• then, based on these two letters", the noback context _$l["ound"] @46-145a rule is selected as a candidate matching rule

Both context _$l["ound"] @46-145a and context _$l["OUND"] @46-145a should be found as candidates. Moreover, they should be in the same ordered list together with other candidates. First looking for a match based on the letters "ou", and then looking for a match based on "OU" would complicate things, as you would have to compare the two to determine which rule should win, which is currently the job of the table compiler.

Making _$l["ound"] match "OUND", in other words making context rules case insensitive, would be another solution, but would be too drastic. It's not unlikely that some authors rely on the fact that context rules are case sensitive.

[bertfrees]

I propose the following change:

diff --git a/liblouis/compileTranslationTable.c b/liblouis/compileTranslationTable.c
index 0338b3d3..8c98b49d 100644
--- a/liblouis/compileTranslationTable.c
+++ b/liblouis/compileTranslationTable.c
@@ -4461,6 +4461,43 @@ finalizeTable(TranslationTableHeader *table) {
 			characterOffset = character->next;
 		}
 	}
+	// Rearrange rules in `forRules' so that when iterating over candidate rules in
+	// for_selectRule(), both case-sensitive and case-insensitive rules are contained
+	// within the same ordered list
+	for (unsigned long int i = 0; i < HASHNUM; i++) {
+		TranslationTableOffset *rp = &table->forRules[i];
+		while (*rp) {
+			TranslationTableRule *r = (TranslationTableRule *)&table->ruleArea[*rp];
+			// For now only move the rules that we know are case-sensitive, namely
+			// `context' rules. (Note that there may be other case-sensitive rules that
+			// we're currently not aware of.) We don't move case insensitive rules because
+			// the user can/should define them using all lowercases.
+			if (r->opcode == CTO_Context) {
+				unsigned long int hash = _lou_stringHash(&r->charsdots[0], 1, table);
+				if (hash != i) {
+					// compute new position
+					TranslationTableOffset *rrp = &table->forRules[hash];
+					while (*rrp) {
+						TranslationTableRule *rr =
+								(TranslationTableRule *)&table->ruleArea[*rrp];
+						if (r->charslen > rr->charslen)
+							break;
+						else if (r->charslen == rr->charslen && rr->opcode == CTO_Always)
+							break;
+						rrp = &rr->charsnext;
+					}
+					// remove rule from current list and insert it at the correct position
+					// in the new list
+					TranslationTableOffset tmp = r->charsnext;
+					r->charsnext = *rrp;
+					*rrp = *rp;
+					*rp = tmp;
+					continue;
+				}
+			}
+			rp = &r->charsnext;
+		}
+	}
 	table->finalized = 1;
 	return 1;
 }

Note that it can be optimized a bit, but this gives you the idea.