Comments (6)
Can you submit a snippet that shows how/where you are setting the dynamic value and where you expect to get the value back out?
from abotx.
Sorry for the delay I wanted to be sure the error wasn't getting triggered elsewhere.
Here's an example of it going awry for me..
globalCrawlEngine.parallelCrawler.CrawlerInstanceCreatedAsync += (sender, e) =>
{
e.Crawler.PageCrawlStartingAsync += ProcessPageCrawlStarting;
e.Crawler.PageCrawlCompletedAsync += ProcessPageCrawlCompleted;
e.Crawler.PageCrawlDisallowedAsync += PageCrawlDisallowed;
e.Crawler.PageLinksCrawlDisallowedAsync += PageLinksCrawlDisallowed;
e.Crawler.CrawlBag.siteID = e.SiteToCrawl.SiteBag.siteID;
e.Crawler.CrawlBag.siteGrabID = e.SiteToCrawl.SiteBag.siteGrabID;
var testID1 = e.Crawler.CrawlBag.siteID;
var testUri1 = e.SiteToCrawl.Uri.AbsoluteUri;
}
The variables all match up here. I can add a break point and see that the URL matches the id that has been passed in. But then later in code when the PageCrawlStartingAsync
event is triggered the variables no longer match.
public void ProcessPageCrawlStarting(object sender, PageCrawlStartingArgs e)
{
var testID2 = e.crawlContext.crawlBag.siteID;
var testUri2 = e.pageToCrawl.Uri.AbsoluteUri;
}
Although testID1 and testID2 match (they are generated new for each crawl), I can see a difference in the host in testUri1 and testUri2.
As mentioned originally though, I can only trigger this error after a fresh compile and only if two additions are made closely together via the following code
SiteToCrawl siteAddition = new SiteToCrawl();
siteAddition.Uri = new Uri(siteGrab.domain);
siteAddition.SiteBag.siteGrabID = siteGrab.id;
siteAddition.SiteBag.siteID = siteGrab.siteID;
siteAddition.SiteBag.baseUri = siteAddition.Uri;
_siteToCrawlProviderX.AddSitesToCrawl(new List<SiteToCrawl> { siteAddition });
globalCrawlEngine.parallelCrawler.StartAsync();
All of the data at this point is also matching and correct.
from abotx.
Can you try something for me? Subscribe to the non async version and see if that solves your issue.
globalCrawlEngine.parallelCrawler.CrawlerInstanceCreated
instead of
globalCrawlEngine.parallelCrawler.CrawlerInstanceCreatedAsync
I suspect that maybe the since the initialization of those values is happening async that it may cause a scenario where it may fire the ProcessPageCrawlStarting event when the CrawlerInstanceCreatedAsync event is halfway done completing.
from abotx.
Thanks for getting back to me.
OK. I've tried that and that wasn't it. I did find a dirty way around it but then realised that it was inconsistent. It appears that rootUri and originalRootUri are often set to null the first time through the set to the first site's url on the second pass. The crawler always takes this url as it's base url for the crawl and the isInternal check comes back incorrectly. I have overridden this check now and manually check in the decision maker. as well as passing the base url down in the SiteBag so i can manually re-add it during the instanceCreated function. This seems to work but is a very dirty fix and sometimes drops out (different results each test, even from fresh compile). I am about to try an even dirtier fix by adding a delay in the init of the parallel crawler to simply prevent any additions if the singleton crawler has only recently been setup. I'll let you know how I get on and if you have any other ideas, I'm all ears.
from abotx.
When I try to set this up with your snippets I'm still unable to repro. Could you create a fully runnable example? Ie.. a console app that registers everything and add a condition in the SiteCrawlStarting that detects when they are mismatched so I can add a breakpoint there and wait for the condition to happen?
from abotx.
Closing this issue since I cannot repro and have not heard from the reporter in almost 2 weeks
from abotx.
Related Issues (20)
- User agent HOT 5
- Robots.txt is not reloaded when uri scheme is changed (http/https) HOT 1
- custom implementations for IThreadManager doesn't work HOT 2
- Problems with Javascript rendering (phantomjs.exe not found)
- AbotX not respecting CrawlConfigurationX MaxPagesToCrawlPerDomain and MaxCrawlDepth HOT 5
- What happened to some of the properties that were in v1? HOT 1
- Parallel engine not working HOT 8
- If paid feature is enabled, Abotx doesn't do anything and no warning or exception is present HOT 1
- How would I crawl a single site with multiple pages in parallel? HOT 2
- crawledPage.HttpRequestException. on https://aanhangwagenspattyn.be/ while ok in browser or Postman HOT 1
- Incompatibility with net5.0 HOT 5
- abotx.org SSL certificate failing HOT 2
- Javascript rendering - detecting window.location changes HOT 3
- Crawl through a proxy server HOT 2
- AbotX produces huge amount of warnings on Linux HOT 1
- crash when stop
- Current AbotX license does not include rendering of javascript. HOT 1
- Add Elapsed property on the AllCrawlsCompleted event
- Constructing wrong URLs to crawl from anchor tags without scheme
- Releasing the source code for Abotx HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abotx.