shimo-lab/universal-geometry-with-ica

Discovering Universal Geometry in Embeddings with ICA

Universal-Geometry-with-ICA

Discovering Universal Geometry in Embeddings with ICA
Hiroaki Yamagiwa*, Momose Oyama*, Hidetoshi Shimodaira
EMNLP 2023

English word embeddings

Heatmap of ICA-transformed word embeddings

Cross-lingual embeddings

Heatmaps of ICA-transformed word embeddings

Spiky shape of embedding distributions

Scatter plots of ICA-transformed word embeddings

English	Spanish

Russian	Arabic	Hindi	Chinese	Japanese

Code and Data

• The code for English embeddings is currently being prepared.
• For cross-lingual embeddings, dynamic embeddings, and image model embeddings, please refer to the universal directory.

Citation

If you find our code or data useful in your research, please cite our paper:

@inproceedings{DBLP:conf/emnlp/YamagiwaOS23,
  author       = {Hiroaki Yamagiwa and
                  Momose Oyama and
                  Hidetoshi Shimodaira},
  editor       = {Houda Bouamor and
                  Juan Pino and
                  Kalika Bali},
  title        = {Discovering Universal Geometry in Embeddings with {ICA}},
  booktitle    = {Proceedings of the 2023 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023},
  pages        = {4647--4675},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://aclanthology.org/2023.emnlp-main.283},
  timestamp    = {Wed, 13 Dec 2023 17:20:20 +0100},
  biburl       = {https://dblp.org/rec/conf/emnlp/YamagiwaOS23.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

universal-geometry-with-ica's Issues

Why ica.mixing_?

https://github.com/shimo-lab/Universal-Geometry-with-ICA/blob/92a1c4fd628f2c9457df710b461370fa1ecdcc65/universal/src/crosslingual_save_pca_and_ica_embeddings.py#L254C1-L257C30

ica = FastICA(**ica_params)
ica.fit(pca_embed)
R = ica.mixing_
ica_embed = pca_embed @ R

Hi, authors. Thank you for your work. I would like to ask why here using "ica.mixing_", rather than "ica.components_"? I think pca_embed @ ica.components_ is to transform the original data space into a more independent one and it is equal to "ica.fit_transform". I don't know if I am right when I am reading this sklearn guidance.

Can you release your code?

Hi, thank you for your work! Could you release your code?