Separate crawlerX's crawl each others sites

[sjdirect]

Separate crawlerX's crawl each others sites

[sjdirect]

Can you submit a snippet that shows how/where you are setting the dynamic value and where you expect to get the value back out?

[MossP]

Sorry for the delay I wanted to be sure the error wasn't getting triggered elsewhere.

Here's an example of it going awry for me..

globalCrawlEngine.parallelCrawler.CrawlerInstanceCreatedAsync += (sender, e) =>
{
    e.Crawler.PageCrawlStartingAsync += ProcessPageCrawlStarting;
    e.Crawler.PageCrawlCompletedAsync += ProcessPageCrawlCompleted;
    e.Crawler.PageCrawlDisallowedAsync += PageCrawlDisallowed;
    e.Crawler.PageLinksCrawlDisallowedAsync += PageLinksCrawlDisallowed;

    e.Crawler.CrawlBag.siteID = e.SiteToCrawl.SiteBag.siteID;
    e.Crawler.CrawlBag.siteGrabID = e.SiteToCrawl.SiteBag.siteGrabID;

    var testID1 = e.Crawler.CrawlBag.siteID;
    var testUri1 = e.SiteToCrawl.Uri.AbsoluteUri;
}

The variables all match up here. I can add a break point and see that the URL matches the id that has been passed in. But then later in code when the PageCrawlStartingAsync event is triggered the variables no longer match.

public void ProcessPageCrawlStarting(object sender, PageCrawlStartingArgs e)
{
    var testID2 = e.crawlContext.crawlBag.siteID;
    var testUri2 = e.pageToCrawl.Uri.AbsoluteUri;
}

Although testID1 and testID2 match (they are generated new for each crawl), I can see a difference in the host in testUri1 and testUri2.

As mentioned originally though, I can only trigger this error after a fresh compile and only if two additions are made closely together via the following code

SiteToCrawl siteAddition = new SiteToCrawl();
siteAddition.Uri = new Uri(siteGrab.domain);
siteAddition.SiteBag.siteGrabID = siteGrab.id;
siteAddition.SiteBag.siteID = siteGrab.siteID;

siteAddition.SiteBag.baseUri = siteAddition.Uri;
 _siteToCrawlProviderX.AddSitesToCrawl(new List<SiteToCrawl> { siteAddition });
globalCrawlEngine.parallelCrawler.StartAsync();

All of the data at this point is also matching and correct.

[sjdirect]

Can you try something for me? Subscribe to the non async version and see if that solves your issue.

globalCrawlEngine.parallelCrawler.CrawlerInstanceCreated

instead of

globalCrawlEngine.parallelCrawler.CrawlerInstanceCreatedAsync

I suspect that maybe the since the initialization of those values is happening async that it may cause a scenario where it may fire the ProcessPageCrawlStarting event when the CrawlerInstanceCreatedAsync event is halfway done completing.

[MossP]

Thanks for getting back to me.

OK. I've tried that and that wasn't it. I did find a dirty way around it but then realised that it was inconsistent. It appears that rootUri and originalRootUri are often set to null the first time through the set to the first site's url on the second pass. The crawler always takes this url as it's base url for the crawl and the isInternal check comes back incorrectly. I have overridden this check now and manually check in the decision maker. as well as passing the base url down in the SiteBag so i can manually re-add it during the instanceCreated function. This seems to work but is a very dirty fix and sometimes drops out (different results each test, even from fresh compile). I am about to try an even dirtier fix by adding a delay in the init of the parallel crawler to simply prevent any additions if the singleton crawler has only recently been setup. I'll let you know how I get on and if you have any other ideas, I'm all ears.

[sjdirect]

When I try to set this up with your snippets I'm still unable to repro. Could you create a fully runnable example? Ie.. a console app that registers everything and add a condition in the SiteCrawlStarting that detects when they are mismatched so I can add a breakpoint there and wait for the condition to happen?

[sjdirect]

Closing this issue since I cannot repro and have not heard from the reporter in almost 2 weeks