Comments (9)
mmh actually it’s not straightforward to load the saved report into a new one. it would need something like a from_dict
method that reverses to_dict
(https://github.com/AmenRa/ranx/blob/master/ranx/report.py#L240)
from ranx.
Hi Paul,
Please, give me a more detailed description of the use case.
What do you mean by "tweak the runs"? Do you want to switch the compared runs without re-computing the metric scores, or do you need to expand the compared set of runs for printing tables?
As you pointed out, loading a report
is not straightforward, and other stuff needs to be implemented first to make it doable.
I will add some new functionalities for saving/loading Runs
with pre-computed metric scores (basically, saving/loading the data produced by the evaluate
function, which compare
also uses).
This and the addition of a mechanism to avoid re-computing metric scores during comparisons should make them much smoother.
Could this work in your use case?
from ranx.
I meant "comparing only models A and B instead of A, B and C".
Saving metric scores along with runs sounds great! But the statistical tests also take some time, don’t they?
My idea was basically to separate the computation from the visualization/printing/formatting of the results.
from ranx.
I timed compare
with five runs
and evaluate
with the same runs (metrics=["map@100", "mrr@100", "ndcg@10"]
).
I got 40+ secs for compare
and less than 1 sec in total for evaluate
.
The Fisher's Randomization Test
is very slow, not sure I can make it much faster.
However, I added the Two-sided Paired Student's t-Test
few days ago on my local branch.
Compare
with it takes 2.5 sec on my machine.
I'll commit it in the afternoon so you can use that for faster comparisons.
from ranx.
The new version is out. Hope it helps.
from ranx.
Thanks! So, should I close the issue?
from ranx.
Well, I don't know honestly :D
Do you think the speed of the comparison with the Two-sided Paired Student's t-Test
"solves" your problem?
Do you still need a report
import function?
from ranx.
Well, the issue was not really about speed (I’m patient enough even to run Fisher's Randomization Test)
It was more generally to separate the computation from the visualization/printing/formatting of the results.
But anyway, consider it closed :) (and thanks again for your quick feedback)
from ranx.
You are always welcome! :)
from ranx.
Related Issues (20)
- [Feature Request] Expose DCG as metric HOT 3
- [BUG] dcg and dcg_burges do not work in the compare function HOT 2
- [Feature Request] Use black to indent the code HOT 1
- [BUG] RBP with multiple relevance levels HOT 3
- [Feature Request] Support gzipped files? HOT 3
- [Feature Request] memory issue / make Run more efficient HOT 2
- Incorrect result for f1 score HOT 13
- Zero-scored documents HOT 10
- [BUG] Misleading exception message on dataframe types HOT 2
- [BUG] Issues when storing/loading Qrels from a dataframe and a parquet file. HOT 6
- [Feature Request] Run.from_df and Run.from_parquet does not allow specifying run name HOT 1
- Question on rank aggregation usage HOT 4
- Getting "Segmentation fault (core dumped)" error HOT 2
- [Feature Request] stddev statistic HOT 3
- Couldn't find any documentation about Qrel and run score range HOT 2
- [Feature Request] Propensity-scored Metrics HOT 1
- How do we compare different runs with multiple folds per run? HOT 1
- [Question] About the correction among multiple hypotheses HOT 1
- [Question] How to compute precision for a retriever operating at passage-level HOT 1
- JIT compilation on serverless (i.e. Modal Labs) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ranx.