Pasting a list of 100 urls and only 43 were scanned

[semperaye]

Pasting a list of 100 urls and only 43 were scanned

[nazuke]

Hello @semperaye,

Thank you for opening this issue.

There is a very good chance that the problem that you are encountering was due to a malformed User-Agent HTTP Header that I just released a fix for last night.

If v1.7.1.0 of SEO Macroscope still exhibits this issue, please say. And if possible, could you provide 2 or 3 URLs that definitely fail to be crawled from that list; it would help me to debug the problem.

Other than that, there are no hard limits to the maximum number of URLs that can be crawled, other than available RAM, etc.

Many thanks

[semperaye]

Thanks for the reply. Similar thing happened this evening (with the new version) I was scanning a batch of around 1400 urls and just when it got to about 1370ish the program crashed lol....unfortunately I didn't grab the error code :( I have 8gb of RAM idk could be an issue. I think I'm going to reduce my scans to around 1050ish and see if I still have issues.

[nazuke]

Hi @semperaye,

A lot has changed in v1.7.6 since this bug was first reported.

I'm closing this ticket for now, but please report a new issue if you have the same problem occurring again.

many thanks