kellymarchisio/ee-giza

Improved word alignment by incorporating monolingual embedding spaces into MGIZA/GIZA++. (Embedding-Enhanced Giza++, Marchisio et al., 2021. Preprint)

Embedding-Enhanced GIZA++: Improving Word Alignment in Very Low-Resource Settings

This is an implementation of the alignment system presented in:

• Kelly Marchisio, Conghao Xiong, and Philipp Koehn. 2021. Embedding-Enhanced GIZA++: Improving Word Alignment in Very Low-Resource Settings Using Embeddings. Preprint.

If you use this software for academic research, please cite the paper above.

Requirements

• python3
• cupy
• scipy
• pytorch
• boost

---

Setup

To download pretrained word embeddings, run sh get_data.sh from the embs/ folder. To download the necessary software, run sh get_packages.sh from the third_party/ folder. Manually download the En-De data. Instructions: https://github.com/lilt/alignment-scripts Run sh setup.sh

To build ee-giza++:

cd eegizapp 
mkdir build
cd build
cmake ..
make

Usage

Map embeddings for the language pair you want to align. For example, to map German and English pretrained embedding spaces with VecMap, run:

sh map-embs.sh de en

Then run EE-Giza++ with 1000 sentences:

sh run-eegiza.sh de en 1000 

Results for eegiza++ will be found in exps/de-en/1000/results/deen-results.txt

Results for vanilla giza++ will be found in exps/de-en/1000/results-vanilla/deen-results.txt

Troubleshooting

You'll need Boost to compile eegiza++. Here's what my env variables look like:

export LIBRARY_PATH="/PATH/TO/BOOST/boost_1_65_1/install/lib:$LIBRARY_PATH"
export LD_LIBRARY_PATH="/PATH/TO/BOOST/boost_1_65_1/install/lib:$LD_LIBRARY_PATH"
export INCLUDE_PATH="/PATH/TO/BOOST/boost_1_65_1/install/include:$INCLUDE_PATH"
export CMAKE_LIBRARY_PATH="/PATH/TO/BOOST/boost_1_65_1/install/lib:$CMAKE_LIBRARY_PATH"
export CMAKE_INCLUDE_PATH="/PATH/TO/BOOST/boost_1_65_1/install/include:$CMAKE_INCLUDE_PATH"
export BOOST_ROOT="/PATH/TO/BOOST/boost_1_65_1/install"
export BOOST_INCLUDE="/PATH/TO/BOOST/boost_1_65_1/install/include"
export BOOST_LIBDIR="/PATH/TO/BOOST/boost_1_65_1/install/lib"

ee-giza's Issues

hi，

hi! I am trying to run eegiza. I placed the English German data in the data/train directory of ee-giza and then submitted it to run-eegiza.sh. I checked exps/***/result/deen.talp, but the alignment effect I ran is very poor (even worse than traditional giza). May I know where the problem is with me?
In addition, when I submit run eegiza. sh using sh, it will prompt "combine. sh: 17:4: not found". I choose bash to submit, and my environment is Ubuntu 20.04