Research on embeddings in different languages for COS 597A.
- Fetch 6000+ random sentences across various languages from Tatoeba data
- Embed with ada-002, store in Pinecone
- Investigate relationship between semantic meaning, language, and location in embedding space
- Use PCA to visualize embeddings in 3D (https://github.com/openai/openai-cookbook/blob/main/examples/Visualizing_embeddings_in_3D.ipynb, as seen at 'Text similarity models’ in https://openai.com/blog/introducing-text-and-code-embeddings)
- Do related languages place more closely, even if the sentences have the same meanings?
- English (original)
- French
- Spanish
- German
- Chinese
- Japanese
- Russian
- Portuguese
- Create a virtual environment:
python3 -m venv venv
- Activate the virtual environment:
- On macOS/Linux:
source venv/bin/activate
- On Windows:
.\venv\Scripts\activate
- Install the requirements:
pip3 install -r requirements.txt
-
Download the latest language pair data from Tatoeba into
data/raw
. Or, decompressdata/compressed/compressed.zip
anddata/compressed/compressed2.zip
and copy their files intodata/raw
. -
Run
create_pairs.py
to create the example sentences in the given languages. With the data provided there should be 6696. -
Make sure you have a
.env
file which looks like this:
OPENAI_API_KEY=""
PINECONE_API_KEY=""
PINECONE_ENV = ""
Replace the empty strings with your environment variables.
-
Run
embed_and_store
to store embeddings in Pinecone. -
Run
langmap.py
for results.