Comments (8)
Yup, you're right, there's a bug here.
from pyserini.analysis.pyanalysis import get_lucene_analyzer, Analyzer
analyzer = get_lucene_analyzer(stemming=False, stopwords=False)
index_utils.get_term_counts('hobbies:photographi', analyzer)
# fails: (0, 0)
index_utils.get_term_counts('hobbies\:photographi', analyzer)
# works: (1, 1)
index_utils.get_term_counts('hobbies:photography')
# fails: (0, 0)
index_utils.get_term_counts('hobbies\:photography')
# works: (1, 1)
What's happening is that a:b
is getting interpreted by Lucene as a field query, i.e., where "a" is the field name.
This is because we run the query through a query parser:
https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexReaderUtils.java#L210
We shouldn't.
Although this does the right thing:
postings_list = index_utils.get_postings_list('hobbies:photography')
for posting in postings_list:
print(f'docid={posting.docid}, tf={posting.tf}, pos={posting.positions}')
This requires a batch to Anserini and then a new maven artifact deploy. I'll get on it.
Thanks for catching the bug!
from pyserini.
@PepijnBoers please take a look: castorini/anserini#1135
+1 with it if you're happy.
from pyserini.
Per this: https://github.com/castorini/pyserini/#usage-of-the-index-reader-api
# Fetch the document vector:
doc_vector = index_utils.get_document_vector('FBIS4-67701')
# Result is a dictionary where the keys are analyzed terms (i.e., the stemmed form that
# was actually indexed) and the values are the term frequencies.
print(doc_vector)
Yes, "hobbies:photographi" is a (poorly) stemmed (i.e., analyzed) form.
from pyserini.
A bit more detail:
analyzer = pyanalysis.get_lucene_analyzer(stemming=False, stopwords=False)
df = {term: (index_utils.get_term_counts(term, analyzer=analyzer))[1] for term in tf.keys()}
The above code snippet isn't going to work because the index wasn't (at least by default) built using the analyzer config... so it's not going to find the term... and if it does, it's a coincidence of stemmed/non-stemmed forms matching.
from pyserini.
Thank you for your response! I'm still not getting it completely though. As far as I understand; the document vector contains the stemmed form of each term, so by iterating over the keys in the document vector I'm looping over the stemmed versions.
You mention that here as well:
doc_vector = index_utils.get_document_vector('FBIS4-67701')
# Result is a dictionary where the keys are analyzed terms (i.e., the stemmed form that
# was actually indexed) and the values are the term frequencies.
Then when I'm calling the index_utils.get_term_counts(term, analyzer)
, I'm searching the index, which to my understanding also contains stemmed terms, therefore I do not want to stem the term again and use a dummy analyzer.
Maybe the misunderstanding originates from whether the index contains stemmed terms or not. I've built my index using the following flags:
./target/appassembler/bin/IndexCollection -collection WashingtonPostCollection \
-input /Volumes/Samsung_T5/WashingtonPost.v2/data -generator WashingtonPostGenerator \
-index lucene-index.core18.pos+docvectors+rawdocs_all \
-threads 1 -storePositions -storeDocvectors -storeRaw -optimize
from pyserini.
Hrm, I think you're right! What's the docid in core18
? Let me take a look at it.
from pyserini.
docid: 1c7fe012-df9d-11e3-9743-bb9b59cde7b9
from pyserini.
Closed with #86 and in v0.9.1.0 release.
from pyserini.
Related Issues (20)
- Add onboarding doc w/ OpenAI embeddings
- Resolve unittest issues HOT 6
- Add test coverage for impact searcher
- Error for SPLADE on-the-fly encoding with pytorch HOT 3
- Integration failures on linux.cs
- 2CR and regression failures due to underlying system libraries (BLAS?) HOT 3
- MIRACL and Mr. TyDi reproduction failures HOT 4
- Unittest warning - should fix HOT 1
- MS MARCO V1 Passage in Linux Student Environment (Encountered Issues with SLIM) HOT 4
- Could you please release the indexes of TCT-ColBERTv2 for MS MARCO V2 Collections? HOT 2
- Where in the codebase are the outputs of TctColBertDocumentEncoder/TctColBertQueryEncoder combined to create the MaxSim score?
- Error with SLIM pre-built index: fails consistency check HOT 1
- Better characterize running times of jobs in onboarding docs
- Request for Dense Indexed Data
- Failure with BEIR regressions
- Upgrade to Python 3.10? HOT 1
- SLIM regressions on MS MARCO v2 passage HOT 1
- DPR encoder seemingly missing attention mask HOT 3
- Is it possible to seperate bm25 search module in pyserini?
- ODQA NQ regression error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyserini.