yao-dou/lens

LENS

The repo contains the datasets and the metric checkpoints from our paper: LENS: A Learnable Evaluation Metric for Text Simplification

data folder contains the SimpEval datasets.

lens folder contains the LENS metric checkpoint. Further details to install and use LENS are in the folder.

lens's Issues

Why the score so low?

Add Support for non-CUDA devices

Unfortunately, LENS does not currently support devices without CUDA, or not correctly. In the lens.score function there is the possibility to deactivate CUDA via devices=None, but without CUDA I am not able to initialize LENS at all:

from lens import download_model, LENS

lens_path = download_model("davidheineman/lens")
lens = LENS(lens_path, rescale=True)

will result in an error:

lens/models/__init__.py:68, in load_from_checkpoint(checkpoint_path)
     [64](.../site-packages/lens/models/__init__.py:64)     model_class = str2model[hparams["class_identifier"]]
     [65](.../site-packages/lens/models/__init__.py:65)     # model = model_class.load_from_checkpoint(
     [66](.../site-packages/lens/models/__init__.py:66)     #     checkpoint_path, load_pretrained_weights=False, strict=False
     [67](.../site-packages/lens/models/__init__.py:67)     # )
...
    [253](.../site-packages/torch/serialization.py:253)                        'to map your storages to the CPU.')
    [254](.../site-packages/torch/serialization.py:254) device_count = torch.cuda.device_count()
    [255](.../site-packages/torch/serialization.py:255) if device >= device_count:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Workaround:

I'm able to use LENS on my Mac by making the following change in the load_from_checkpoint function:

From:

LENS/lens/lens/models/__init__.py

Lines 68 to 70 in 8fa5b14

	model = model_class.load_from_checkpoint(
	checkpoint_path, **hparams, strict=False
	)

To:

        model = model_class.load_from_checkpoint(
            checkpoint_path, **hparams, strict=False, map_location=torch.device('cpu')
        )

(Don't forget to import torch)

Download the checkpoint

Where should I place the downloaded checkpoint? and should I unzip the file or LENS code will do that? because I keep getting errors regarding the path and missing yaml file.

Considerations for ensuring model reproducibility

I tried running the LENS model through the COMET description, and it seems there's a slight difference in performance compared to the checkpoint model. Starting from epoch 0 and going up to 5 (actually 6), it appears that the checkpoint steps are different between us (me: 5760/you: 6102).

I suspect the reason for this might be differences in the data split ratios. Could you please let me know how the train and valid datasets were seperated?(I split the SIMPEVAL_past dataset into an 8:2 ratio.) The parameter settings were followed from the checkpoint model.

How can I train the model?

Hi there,

I have been exploring the regression_metric_multi_ref.py file in the repository and I'm interested in understanding the process of training a model using this script.

I noticed that there seems to be no explicit training_step defined in the regression_metric_multi_ref.py file. However, I might be overlooking something, and I would greatly appreciate it if you could shed some light on how the model is trained using this script.

Do you have any plans to release the code for training the Comet model, aside from the checkpoint model?

Best regards,

Are the SARI implementations on Lens platform and github repo the same?

Hello, I've been using the Lens platform from http://nlprx.cc.gatech.edu:9015/ and noticed that the SARI score is calculated as an average of sentence-level scores.

However, when I used the sari.py script from experiments/meta_evaluation/metrics, it yielded a different result.

Could you please explain the SARI calculation method used on your platform?

Exception: hparams.yaml file is missing!

Hello, when I want to do a test run I always get the error message "Exception: hparams.yaml file is missing!". It doesn't matter if I only use the ckpt file or the whole LENSE folder in the file path. This is the error message which I am getting:

`"G:\My Drive\M5\Masterarbeit\implementation_metrics\LENS\venv\Scripts\python.exe" "G:\My Drive\M5\Masterarbeit\implementation_metrics\LENS\run-dennis.py"
Traceback (most recent call last):
File "G:\My Drive\M5\Masterarbeit\implementation_metrics\LENS\run-dennis.py", line 8, in
metric = LENS(model_path, rescale=True)
File "G:\My Drive\M5\Masterarbeit\implementation_metrics\LENS\venv\lib\site-packages\lens\lens_score.py", line 9, in init
self.model = load_from_checkpoint(path)
File "G:\My Drive\M5\Masterarbeit\implementation_metrics\LENS\venv\lib\site-packages\lens\models_init_.py", line 46, in load_from_checkpoint
raise Exception("hparams.yaml file is missing!")
Exception: hparams.yaml file is missing!

Process finished with exit code 1`

I am using Windows 11 and for this project I use Python 3.7 on a PyCharm virtual environment.

Thank you in advance!