Comments (3)
using cython.parallel
should help in implementing this
from lsh.
++
from lsh.
Sample of code helped me to speed up the hashing:
def _cache_texts_parallel(self, docs: Iterable[str]):
encoded_docs = (doc.encode("utf8") for doc in docs)
with ProcessPoolExecutor(max_workers=self.workers) as executor:
for i, fingerprint in enumerate(executor.map(self.lsh_cache.hasher.fingerprint, encoded_docs)):
self.lsh_cache.add_fingerprint(fingerprint, i)
from lsh.
Related Issues (20)
- Need help installing this HOT 2
- Check buckets of LSH MinHash
- Is it possible to extend this project for document file formats like PDF, DOCX HOT 1
- Unable to install HOT 1
- storing cache HOT 1
- Unable to install on Mojave (10.14.2) or Ubuntu HOT 5
- ModuleNotFoundError: No module named 'lsh.cMinhash' HOT 5
- Unable to install on Windows HOT 1
- Unable to install in Python3 HOT 6
- Jaccard should be performed on sets, but appears to be given numpy arrays HOT 3
- How to make minhash scalable
- A few questions about the `len` argument in the function `MurmurHash3_x86_32`.
- ModuleNotFoundError: No module named 'LSH.lsh.cMinhash' HOT 1
- error: command 'cc' failed with exit status 1 HOT 1
- Python 3.10 compatiblity
- allow other backends for storing duplicate documents HOT 2
- Add support for SimHash
- Create PyPi package HOT 2
- Is it possible to extend LSH to detect near duplicate images? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lsh.