Comments (3)
Hey @caiqizh! We're happy to hear that you've been able to make use of our library!
Currently you can change the way these matrix are calculated only by directly changing the code in this module. We will work on improving customizability of implemented methods in the near future, so this will probably become much simpler soon.
from lm-polygraph.
As a temporary solution, you can use the following snippet, which uses SemanticMatrixCalculator to calculate these additional statistics. But we hope to improve the code for your usecase in the nearest future.
from lm_polygraph.estimators import Eccentricity
from lm_polygraph.stat_calculators import SemanticMatrixCalculator
# Put your 5-10 text samples from ChatGPT generation
samples = [
'The capital of France is Paris.',
'Paris is the capital city of France.',
'The capital of France is Paris.',
'In France, the capital is Paris.',
'The capital city of France is Paris.',
]
stats = {
'blackbox_sample_texts': [samples],
'deberta_batch_size': 10,
}
nli_calculator = SemanticMatrixCalculator()
stats.update(nli_calculator(stats, None, None, None))
# Now stats should contain 'semantic_matrix_entail' and 'semantic_matrix_contra'
estimator = Eccentricity()
uncertainty = estimator(stats)[0]
print(uncertainty) # 7.886122580038351e-05
from lm-polygraph.
Thank you for the previous answer! It looks like in the latest version this does not work anymore. Could you please reopen this issue? Thanks!
from lm-polygraph.
Related Issues (18)
- Possible mismatching max_length and max_new_tokens in example eval script HOT 4
- Not 100% sure if sampling parameters is correct HOT 3
- Demo doesn't work. HOT 4
- Get the uncertainty scores without rerun the models HOT 4
- Error loading larger models - You shouldn't move a model when it is dispatched on multiple devices HOT 5
- AutoModelForCausalLM max_length HOT 1
- What is the different between pe_uncertainties and ep_uncertainties? HOT 2
- [Question] Pipeline integration (Langchain) HOT 1
- using the openai API for a BlackBox model for non OPENAI hosted platforms HOT 4
- May I know how to run an ensemble-based uncertainty estimation method? HOT 2
- Entropy calculation maybe wrong? HOT 6
- Dockerfile adjustments HOT 3
- About the generation metric for questions with multiple correct answers HOT 2
- Load custom estimators and stat_calculators in the evaluation script
- Semantic entropy is using probabilites greater than 1 HOT 2
- generate_texts on wbmodel ignores generation parameters and stopping critera.
- Example for normalizaiton HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lm-polygraph.