helgeho/web2warc

An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)

web2warc's Issues

OutOfMemoryError

Running this program:

  val maxLevel = 2
  val outPath = "/tmp/out"
  val domain = "epfl.ch"
  val seedUrl = "http://www.epfl.ch"

  Web2Warc.crawl.name = domain
  Web2Warc.writer.path = outPath
  Web2Warc.spec.maxLevel = maxLevel
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + java.util.regex.Pattern.quote(domain) + "(/.*|$)"
  Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + domain + "(/.*|$)"
  Web2Warc.spec.preserveUrlRegex = ".*" // preserve all crawled URLs (default)
  Web2Warc.spec.followRedirects = true
  Web2Warc.spec.increaseLevelOnRedirect = true
  Web2Warc.seeds += seedUrl

  Web2Warc.run()

Fails with OutOfMemoryError:

::Queue state:: current level: 1, URLs in queue: 114
::Queue state:: current level: 1, URLs in queue: 137
...
::Queue state:: current level: 2, URLs in queue: 1020
::Queue state:: current level: 2, URLs in queue: 1019
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Any idea?
Is the program scalable, or the url queue is keep in memory? Still, 2000 urls do not seem too much to cause an OutOfMemoryError.

sbt assembly fails

I am trying to use Web2Warc from the repo source (8e8e309) and am running into issues getting started.

$ git clone https://github.com/helgeho/Web2Warc
$ cd Web2Warc
$ cd project (per the README setup instructions)
$ sbt assembly
[info] Updated file /private/tmp/Web2Warc/project/project/build.properties: set sbt.version to 1.2.6
[info] Loading project definition from /private/tmp/Web2Warc/project/project
[info] Updating ProjectRef(uri("file:/private/tmp/Web2Warc/project/project/"), "project-build")...
[info] Done updating.
[info] Loading settings for project project from assembly.sbt,plugins.sbt ...
[info] Set current project to project (in build file:/private/tmp/Web2Warc/project/)
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':'
[error] Not a valid key: assembly
[error] assembly
[error]         ^
$

No web2warc-assembly-1.0.jar appears to have been created. I am using Scala 2.12.7 on macOS 10.14.1. Please advise.

publish it to maven central or some other repository

publish it to maven central or some other repository?

followRedirects = true does not work on seeds

here, the seeds contains only one page, which is a "301 Moved Permanently" page.
even if followRedirects is set to true, the crawl stops here.
it works if I replace the seed url with a 200 page (then it crawls with depth 5, as expected).

  Web2Warc.writer.path = "temp.warc.gz"
  Web2Warc.spec.maxLevel = 5
  Web2Warc.spec.followRedirects = true
  Web2Warc.spec.increaseLevelOnRedirect = false
//  Web2Warc.seeds = Set("https://epfl.ch")   // this crawler crawls until depth 5.
  Web2Warc.seeds = Set("http://epfl.ch")  // the crawler only archives this page
  Web2Warc.run()