Comments (4)
YairHalberstadt
commented on May 27, 2024
I've opened a draft PR at #226 to show the general direction I would go in to fix this. I don't have time to do much more at the moment I'm afraid :-(
Anyway good luck!
from abot.
sjdirect
commented on May 27, 2024
Unfortunately I don't either. The problem is that interface has been around since the initial implementation of using threads directly. We decided to keep the abstraction to avoid a major breaking change.
from abot.
sjdirect
commented on May 27, 2024
Can you demonstrate a unit test that proves it does nothing? The current implementation holds the maxconcurrenthttprequests to the configured amount and also provides the concurrency of processing of those completed requests.
from abot.
sjdirect
commented on May 27, 2024
Closing until a test can be submitted in regards to my last comment.
from abot.
Related Issues (20)
- Subdomains links are treated as external links HOT 1
- How to implement the decision maker function HOT 1
- User Agent config value appears to be getting split on spaces and sending requests with multiple user-agent headers HOT 2
- Update Readme file? HOT 1
- Collection of links found HOT 1
- Sitemap.xml parsedlinks is empty HOT 5
- Bot stop crawling without any error
- Handling pagination HOT 1
- Best way to do page-by-page crawling HOT 2
- How do I direct it to crawl, specifc paged content in a site directory? HOT 2
- The PageRequester downloads the content by default HOT 1
- Exception is thrown when redirect URI is relative HOT 1
- Base URI logic of the HyperLinkParser breaks on Linux when faced with a root-relative base tag value HOT 2
- Base URI logic of the HyperLinkParser doesn't respect terminated relative base tag values HOT 2
- Question HOT 1
- Control maximum number of concurrent pages to be crawled HOT 1
- Are 304 responses properly handled? HOT 2
- Crawl .jsp sites
- Content.Text empty despite response code OK and Content stream contains data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abot.