Comments (3)
romanchyla
commented on July 28, 2024
btw: the old solr seems to have 5,281,144 docs with fulltext
from adsfulltext.
romanchyla
commented on July 28, 2024
when run as run.py -f /proj/ads/abstracts/config/links/fulltext/all.links --max_queue_size 0 -s
it is submitting everything, this should be the default
from adsfulltext.
romanchyla
commented on July 28, 2024
SOLR has 2.4M docs (the old one 5.2M)
there are some errors in the log queue of adsft
eg.
./logs/adsft.checker.log.2:2017-10-30 17:24:13,527 ERROR [469:MainThread:checker.py:229] Bibcode '1999BAAS...31.1185L' is linked to a zero byte size file '/proj/ads/fulltext/sources/downloads/cache/ADS/articles.adsabs.harvard.edu/full/BAAS/0031/1999BAAS...31.1185L.ocr' ./logs/adsft.checker.log.2:2017-10-30 17:24:13,901 ERROR [478:MainThread:checker.py:229] Bibcode '1999BAAS...31Q1185C' is linked to a zero byte size file '/proj/ads/fulltext/sources/downloads/cache/ADS/articles.adsabs.harvard.edu/full/BAAS/0031/1999BAAS...31Q1185C.ocr'
and
./logs/adsft.extraction.log:2017-10-30 17:13:28,428 ERROR [470:MainThread:extraction.py:897] Fulltext extraction failed for bibcode '1997EL.....38..675R': '/proj/ads/articles/fulltext/sources/EL/0038/epl_38_9_675.pdf' ./logs/adsft.extraction.log:2017-10-30 17:13:31,423 ERROR [480:MainThread:extraction.py:897] Fulltext extraction failed for bibcode '1997EL.....38..423S': '/proj/ads/articles/fulltext/sources/EL/0038/epl_38_6_423.pdf'
from adsfulltext.
Related Issues (20)
- Do not remove facilities from acknowledgements HOT 1
- XML articles with valid empty body should end up with an empty body field in solr
- False positive OCR files for "NASA" queries
- Invalid empty body resulting from <!-- body endbody --> syntax
- Handle facility tags without xlink:href
- HTML extractor cannot decode certain characters
- html5lib parser failing due to less than symbol in XML files
- Facilities appear outside of acknowledgements in XML
- Fulltext query for "Mars" and "HiRISE" returns less papers than before
- Extract useful data from tables
- Some parsers are only extracting the acknowledgements from XML files
- Facilities not properly tokenized
- fulltext contains serialized json in the body
- Use database instead of filesystem HOT 2
- Improve use of file close
- Support encodings other than utf-8
- Multi-file parse HOT 5
- Update parser for Nature XML HOT 2
- Fix missing appendix text HOT 2
- Document Similarity
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adsfulltext.