Proposal, save scan in the background already

[csaeum]

Proposal, save scan in the background already

[csaeum]

YES, that's what I mean.

[nazuke]

Many thanks for this feedback @csaeum.

Essentially, you mean something like:

1. save the session and quit
2. start the program again the next day
3. reload the saved session and continue from where it left off?

Initially, I did not build any form of persistence into this application; which in hindsight would be very nice.

I've done some experimentation, and it does look like it is simple enough to use C#'s serialization features to dump out the state of a current crawl, and then reload it again.

However, there are also a lot of other details that need to be fixed up when a saved crawl is reloaded; so I'm going to have to out this to one side for a while, and come back to it.

If I eventually build a version 2.0 of this, I would probably build this using something like SQLite to store the crawl data, instead of simply having everything in memory.

[nazuke]

Many thanks @csaeum,

The most recent v1.7.6 release that came out yesterday now includes an experimental "Save Session" feature.

This uses C#'s serialization functionality, and basically dumps out the current session to a file.

For example, unless something goes wrong, a crawl can be started, paused, the session saved and the program exited. Then, the program can be started up back, the session loaded, and the crawl resumed.

Some of the Task Parameters should also be saved/restored; but if these are used, then it is worth double-checking each time.

Please note that the session files can be VERY LARGE, and can take an indeterminate amount of time to save and reload. For example, an 11,000 page crawl will produce a 300MB session file. So be patient if it looks like it's taking a very long time to save and load later; it hopefully hasn't crashed.

In practice, this feature seems to be quite useful for generating reports on a crawl at a later date.

The saved session files are not guaranteed to work with earlier/later versions of SEO Macroscope, because some of the internal details can/will change between versions.