giellalt/lang-sms

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Skolt Sami language

The Skolt Sami morphology and tools

Download nightly / CI/CD installation packages for testing (contains the core zhfst file(s)):

NB!! Note that the nightly / CI/CD installation packages are not tested for language quality, and might contain regressions and errors.

This directory contains source files for the Skolt Sami language morphology and dictionary. The data and implementation are licenced under __LICENSE__ licence also detailed in the LICENSE. The authors named in the AUTHORS file are available to grant other licencing choices.

Install proofing tools and keyboards for the Skolt Sami language by using the Divvun Installer

Download and test speller files

The speller files downloadable at the top of this page (the *.bhfst files) can be used with divvunspell, to test their performance. These files are the exact same ones as installed on users' computers and mobile phones. Desktop and mobile speller files differ from each other in the error model and should be tested separately — thus also two different downloads.

Documentation

Documentation can be found here:

• Language specific documentation
• General documentation

Core dependencies

In order to compile and use the Skolt Sami language morphology and dictionaries, you need:

• an FST compiler: HFST, Foma or Xerox Xfst
• VislCG3 Constraint Grammar tools

To install VislCG3 and HFST, just copy/paste this into your Terminal on macOS:

curl https://apertium.projectjj.com/osx/install-nightly.sh | sudo bash

or terminal on Ubuntu, Debian or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get install cg3 hfst

or terminal on RedHat, Fedora, CentOS or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/rpm/install-nightly.sh -O - | sudo bash
sudo dnf install cg3 hfst

Alternatively, the Apertium wiki has good instructions on how to install the dependencies for Mac OS X and how to install the dependencies on linux

Further details and dependencies are described on the GiellaLT Getting Started pages.

Downloading

Using Git:

git clone https://github.com/giellalt/lang-sms

Using Subversion:

svn checkout https://github.com/giellalt/lang-sms.git/trunk lang-sms

Building and installation

INSTALL describes the GNU build system in detail, but for most users it is the usual:

./autogen.sh # This will automatically clone or check out other GiellaLT dependencies
./configure
make
(as root) make install

Citing

If you use the Skolt Sami FST in an academic publication, please cite it as follows:

Rueter, Jack & Hämäläinen, Mika. (2020). FST Morphology for the Endangered Skolt Sami Language. In Proceedings of the 1st Joint SLTU and CCURL Workshop, May 2020, Marseille, France. European Language Resources association, pp. 250--257.

@InProceedings{rueter-hmlinen:2020:SLTUCCURL,
author = {Rueter, Jack and Hämäläinen, Mika}, 
title = {FST Morphology for the Endangered Skolt Sami Language},
booktitle ={Proceedings of the 1st Joint SLTU and CCURL Workshop},
year = {2020}
}

If you use language data from more than one GiellaLT language, consider citing our LREC 2022 article on whole infra:

Linda Wiechetek, Katri Hiovain-Asikainen, Inga Lill Sigga Mikkelsen, Sjur Moshagen, Flammie Pirinen, Trond Trosterud, and Børre Gaup. 2022. Unmasking the Myth of Effortless Big Data - Making an Open Source Multi-lingual Infrastructure and Building Language Resources from Scratch. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1167–1177, Marseille, France. European Language Resources Association.

If you use bibtex, following is as it is on ACL anthology:

@inproceedings{wiechetek-etal-2022-unmasking,
    title = "Unmasking the Myth of Effortless Big Data - Making an Open Source
    Multi-lingual Infrastructure and Building Language Resources from Scratch",
    author = "Wiechetek, Linda  and
      Hiovain-Asikainen, Katri  and
      Mikkelsen, Inga Lill Sigga  and
      Moshagen, Sjur  and
      Pirinen, Flammie  and
      Trosterud, Trond  and
      Gaup, B{\o}rre",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation
    Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.125",
    pages = "1167--1177"
}

lang-sms's Issues

Speller is incomplete

The speller (at least for sms) seems to lack some lemmas after the reorganizing.

$ husmsNorm
mieʹrreed
mieʹrreed	mieʹrreed+V+Imprt+Pl2	0.000000
mieʹrreed	mieʹrreed+V+Inf	0.000000

$ hfst-ospell sms.zhfst 
mieʹrreed
"mieʹrreed" is NOT in the lexicon:

Error: The file lexicon.tmp.lexc did not compile cleanly

I get an error when trying to compile.
If I run make V=1, I get the following:

...
N_Prop_Toponyms_sms2x...400 Num_sms2x...91 Pcle_sms2x...31 Prefix_sms2x...36 Pron_sms2x...47 V_sms2x...5133 ABBR_sms2x...105 ACRO_sms2x...11 Punctuation...60 PunctEnd...1 Symbols...Compiling... Warning: Sublexicon is mentioned but not defined. (OY-sur) 
*** ERROR: could not parse lexc file: treating warnings as errors [--Werror] ***
/usr/local/bin/hfst-lexc: The file lexicon.tmp.lexc did not compile cleanly.
(if there are no error messages above, try -v or -d to get more info)
make[2]: *** [lexicon.tmp.hfst] Error 1
make[2]: *** Deleting file `lexicon.tmp.hfst'
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Wrong analysis for "10"

I get the following analysis for "10":

echo 10|hfst-tokenise -cg tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst
"<10>"
     Use/Circ"1" Use/Circ"0" Num Sg Acc <W:0.0>
     Use/Circ"1" Use/Circ"0" Num Sg Gen <W:0.0>
     Use/Circ"1" Use/Circ"0" Num Sg Ill Attr <W:0.0>
     Use/Circ"1" Use/Circ"0" Num Sg Loc Attr <W:0.0>
     Use/Circ"1" Use/Circ"0" Num Sg Nom <W:0.0>
     Use/Circ"10" Num Sg Acc <W:0.0>
     Use/Circ"10" Num Sg Nom <W:0.0>
:\n

I compiled today:

ls -l tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst
-rw-r--r--  1 car010  staff  112445830 May 21 11:58 tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst

Add missing affixes/acronyms.lexc

SMS fails in Tino's nightlies, and the reason is a missing contlex for acronyms, cf https://apertium.projectjj.com/apt/logs/giella-sms/sid-amd64.log (@TinoDidriksen for info).

It turns out that SMS does not have an affix file for acronyms at all, as opposed to all other SMx repos. One could probably just copy e.g. from SMN or SME, and adjust the continuations according to the needs of SMS.

./configure --with-hfst --without-xfst

When running
../configure --disable-hfst-desktop-spellers --enable-spellers --enable-fst-hyphenator --enable-hfst-mobile-speller
sms virtually requires: --with-hfst --without-xfst --enable-reversed-intersect
without specifically selecting hfst and deselecting xfst, and without reversed intersect,
the time required for a successful compilation of approx. 13 minutes may rise to nearly one hour.

Cannot compile hyphenator

I have tried to compile the hyphenator multiple times, both on a Mac and on two different Linux machines, but the process gets stuck when compiling hyphenator-gt-desc-no_fallback.hfst and is eventually killed.

Configuration: ./configure --enable-fst-hyphenator

Making all in .
make[3]: Entering directory `/home/trondtynnol/giellalt/lang-sms/tools/hyphenators'
  HFST2FST lexicon-gt-desc.hfst
  HXFST    lexicon-gt-desc-clean.hfst
  HREWGHT  lexicon-gt-desc-tag_weighted.hfst
  HPROJECT lexicon-gt-desc-tag_weighted_no_analysis.hfst
  HINTRSCT hyphenator-raw-gt-desc.tmp.hfst
  CP       hyphenator-raw-gt-desc.hfst
  HXFST    hyphenator-gt-desc-input.hfst
  HXFST    hyphenator-gt-desc-output.hfst
  HXFST    hyphenator-gt-desc-no_fallback.hfst
/bin/sh: line 5: 37051 Done                    /usr/bin/printf "read regex 			@\"hyphenator-gt-desc-input.hfst\" 		.o. @\"hyphenator-gt-desc-output.hfst\" 	; \n	 save stack hyphenator-gt-desc-no_fallback.hfst\n	 quit\n"
     37052 Killed                  | /usr/bin/hfst-xfst -p -q
make[3]: *** [hyphenator-gt-desc-no_fallback.hfst] Error 137
make[3]: Leaving directory `/home/trondtynnol/giellalt/lang-sms/tools/hyphenators'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/trondtynnol/giellalt/lang-sms/tools/hyphenators'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/trondtynnol/giellalt/lang-sms/tools'
make: *** [all-recursive] Error 1

When compiling with make V=1, this is the output of the last compilation before it hangs:

/usr/bin/printf "read regex \
@\"hyphenator-gt-desc-input.hfst\" \
.o. @\"hyphenator-gt-desc-output.hfst\" \
; \n\
save stack hyphenator-gt-desc-no_fallback.hfst\n\
quit\n" | /usr/local/bin/hfst-xfst -p -v
Using default output format OpenFst with tropical weight class
Using OpenFst's tropical weights as output
Reading from standard input...
warning: both composition arguments contain flag diacritics that are not harmonized

System-wide speller suggests what it does not accept in TextEdit MacOS

On 13.5.1 Ventura the spell checker does not accept word forms like ‹säämas›, but its top suggestion is the very same ‹säämas› form. This is related to giellalt/lang-lut#4 and, most likely, giellalt/lang-lut#1

hfst-lookup src/fst/analyser-gt-norm.hfstol 
> säämas
säämas	säämas+Adv	0,000000
säämas	sääʹmm+N+Lat	0,000000

echo 'säämas' | hfst-tokenise -g tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst 
"<säämas>"
	"säämas" Adv <W:0.0>
	"sääʹmm" N Sem/Hum Lat <W:0.0>

hfst-ospell -S -n 5 tools/spellcheckers/sms.zhfst 
säämas
"säämas" is in the lexicon...

BUT, it is not accepted by the speller.