Comments (3)
What is the concern of not changing the user agent? If you want it to be based on the app then you could probably set the default value dynamically, based on the host process name.
from abot.
The issue is that abot is starting to getting listed on "bad bot" sites
like the following which means that polite abot instances could get blocked
by security software even though that specific instance or parent product is
doing nothing wrong.
http://acronymsonline.com/lists/bad_bot.asp
http://www.brontobytes.com/knowledgebase/145/Block-Bad-Bots-and-Spiders-using-htaccess.html
Steven
On Sat, Dec 27, 2014 at 1:24 AM, James Tharpe [email protected]
wrote:
What is the concern of not changing the user agent? If you want it to be
based on the app then you could probably set the default value dynamically,
based on the host process name.—
Reply to this email directly or view it on GitHub
#49 (comment).
from abot.
Fixed this issue with f209280
from abot.
Related Issues (20)
- How to implement the decision maker function HOT 1
- User Agent config value appears to be getting split on spaces and sending requests with multiple user-agent headers HOT 2
- Update Readme file? HOT 1
- Collection of links found HOT 1
- Sitemap.xml parsedlinks is empty HOT 5
- Bot stop crawling without any error
- Handling pagination HOT 1
- Best way to do page-by-page crawling HOT 2
- Incorrect usage of ThreadManager means it does nothing HOT 4
- How do I direct it to crawl, specifc paged content in a site directory? HOT 2
- The PageRequester downloads the content by default HOT 1
- Exception is thrown when redirect URI is relative HOT 1
- Base URI logic of the HyperLinkParser breaks on Linux when faced with a root-relative base tag value HOT 2
- Base URI logic of the HyperLinkParser doesn't respect terminated relative base tag values HOT 2
- Question HOT 1
- Control maximum number of concurrent pages to be crawled HOT 1
- Are 304 responses properly handled? HOT 2
- Crawl .jsp sites
- Content.Text empty despite response code OK and Content stream contains data
- Cancellation is not working properly and integration test is wrong HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abot.