iinemo/lm-polygraph

lm-polygraph's Issues

May I know how to run an ensemble-based uncertainty estimation method?

Hi, I have successfully run the information-based and density-based methods.

May I know how to run the ensemble-based uncertainty estimation method?

Get the uncertainty scores without rerun the models (for NumSets, Deg, Ecc)

Thank you for providing the codes for the previously generated text! They have been very helpful, and I've successfully used them for Lexical Similarity analysis. I'm planning to test them for other measurements, including NumSets, Degree matrix (Deg), and Eccentricity.

I noticed that these measurements require two additional statistics: semantic_matrix_entail and semantic_matrix_contra. According to the original paper, I know that these are calculated using DeBERTa over generated samples. I'm wondering if there are any short code snippets available to compute these matrices and feed them into the estimator function.

Thanks!

What is the different between pe_uncertainties and ep_uncertainties?

RT

[Question] Pipeline integration (Langchain)

Good day! Is there any way to use the LM-polygraph in an LLM pipeline, created with the Langchain framework?

Not 100% sure if sampling parameters is correct

https://github.com/IINemo/lm-polygraph/blame/00851972311db837de6358c90dac65616495d4de/src/lm_polygraph/stat_calculators/sample.py#L27

It's now setting both top_k=50 and top_p if we just set top_p, if i understand code correctly. Having both top_k and top_p is kind of weird to me.
Also there seems there is no do_sample; from the hf docs, we could accidentaly get contrastive_search() instead of sample()

Would be nice to test this

Error loading larger models - You shouldn't move a model when it is dispatched on multiple devices

The code

model = WhiteboxModel.from_pretrained(
    "tiiuae/falcon-40b-instruct",
    cache_dir="~/cache/",
    device_map='auto', 
    offload_folder="offload_folder"

Throws the error You shouldn't move a model when it is dispatched on multiple devices.

While

model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-40b-instruct", 
                                             trust_remote_code=True, 
                                             cache_dir="~/cache/",
                                             device_map="auto",
                                             offload_folder="offload_folder")

seems to work fine :/

Possible mismatching max_length and max_new_tokens in example eval script

I was running the polygraph_eval with this example config https://github.com/IINemo/lm-polygraph/blob/main/examples/configs/polygraph_eval_wmt14_ende.yaml

I got a warning about too long string

It didn't failed but i almost sure it would mess up with the results

My wild guess is that stat calculators max_lengths in not connected to max_generated_tokens in ue manager itself. But i didn't really look into it for now

AutoModelForCausalLM max_length

Is there a reason why the max length is equal to 256?

In models.py:

AutoModelForCausalLM.from_pretrained(max_length=256)

Demo doesn't work.

Thank you for the amazing framework! Today when I was trying the following codes (simplest demo), I got the error massage saying that:

return UncertaintyOutput(ue[0], input_text, texts[0], model.model_path, estimator.level)
TypeError: UncertaintyOutput.__init__() takes 5 positional arguments but 6 were given

I am wondering whether the framework is ready to use or you are still implementing them?

from lm_polygraph.utils.model import WhiteboxModel
from lm_polygraph.estimators import *
from lm_polygraph.utils.manager import estimate_uncertainty

ue_method = MeanPointwiseMutualInformation()
estimator = SemanticEntropy()

model = WhiteboxModel.from_pretrained(
    "bigscience/bloom-560m",
    device="cuda:0",
)

input_text = "Who is George Bush?"
estimate_uncertainty(model, ue_method, input_text=input_text)

Get the uncertainty scores without rerun the models

Thanks again for your work!

I noticed that in your framework, we need to first run the model then get the uncertainty scores. While it's perfectly fine when using free models, it could be expensive when working with charging APIs like ChatGPT.

Specifically, I'm curious if there's a way to obtain uncertainty measures for previously generated texts without having to rerun the model.

Any information or suggestions you can offer in this regard would be greatly appreciated. I look forward to hearing from you and learning more about this possibility.

using the openai API for a BlackBox model for non OPENAI hosted platforms

Hi,

thanks for providing the community with this library. I believe uncertainties of LLM queries are an important topic. I tried to play around with this library and am a bit stuck. So I'd like to use a remote model that is accessible through the openai library. For this, I have to provide a custom OPENAI_API_BASE and my OPENAI_API_KEY. However, the library tells me that it doesn't know how to query the remote model?

Here is the code that I drafted given your example:

def main():
    print(f":: black box test, using Mistral-7B-Instruct-v0.2 from {os.environ["OPENAI_API_BASE"]}")
    model = BlackboxModel(openai_api_key=os.environ["OPENAI_API_KEY"], model_path="Mistral-7B-Instruct-v0.2", parameters={"openai_api_base": os.environ["OPENAI_API_BASE"]})

    print(model.parameters)

    print(":: using estimator EigValLaplacian")
    estimator = EigValLaplacian(verbose=True)
    answer = estimate_uncertainty(
        model, estimator, input_text="When did Albert Einstein die?"
    )
    print(">>",answer)

So I get the following error:

:: using estimator EigValLaplacian
Traceback (most recent call last):
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 23, in <module>
    main()
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/examples/./black_box.py", line 16, in main
    answer = estimate_uncertainty(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 166, in estimate_uncertainty
    man()
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 400, in __call__
    batch_stats = self.calculate(batch_stats, self.stat_calculators, inp_texts)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 534, in calculate
    raise e
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/utils/manager.py", line 518, in calculate
    new_stats = stat_calculator(
                ^^^^^^^^^^^^^^^^
  File "/home/steinb95/development/lm-polygraph/lm-polygraph/src/lm_polygraph/stat_calculators/sample.py", line 46, in __call__
    temperature=model.parameters.temperature,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'temperature'

I tried a couple of things, but I am simply unclear where to supply the temperature?

Best
P