helgeho / web2warc Goto Github PK
View Code? Open in Web Editor NEWAn easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
License: MIT License
An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
License: MIT License
Running this program:
val maxLevel = 2
val outPath = "/tmp/out"
val domain = "epfl.ch"
val seedUrl = "http://www.epfl.ch"
Web2Warc.crawl.name = domain
Web2Warc.writer.path = outPath
Web2Warc.spec.maxLevel = maxLevel
Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + java.util.regex.Pattern.quote(domain) + "(/.*|$)"
Web2Warc.spec.urlRegex = "[^:]+://([^/]*.|)" + domain + "(/.*|$)"
Web2Warc.spec.preserveUrlRegex = ".*" // preserve all crawled URLs (default)
Web2Warc.spec.followRedirects = true
Web2Warc.spec.increaseLevelOnRedirect = true
Web2Warc.seeds += seedUrl
Web2Warc.run()
Fails with OutOfMemoryError:
::Queue state:: current level: 1, URLs in queue: 114
::Queue state:: current level: 1, URLs in queue: 137
...
::Queue state:: current level: 2, URLs in queue: 1020
::Queue state:: current level: 2, URLs in queue: 1019
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
Any idea?
Is the program scalable, or the url queue is keep in memory? Still, 2000 urls do not seem too much to cause an OutOfMemoryError.
I am trying to use Web2Warc from the repo source (8e8e309) and am running into issues getting started.
$ git clone https://github.com/helgeho/Web2Warc
$ cd Web2Warc
$ cd project (per the README setup instructions)
$ sbt assembly
[info] Updated file /private/tmp/Web2Warc/project/project/build.properties: set sbt.version to 1.2.6
[info] Loading project definition from /private/tmp/Web2Warc/project/project
[info] Updating ProjectRef(uri("file:/private/tmp/Web2Warc/project/project/"), "project-build")...
[info] Done updating.
[info] Loading settings for project project from assembly.sbt,plugins.sbt ...
[info] Set current project to project (in build file:/private/tmp/Web2Warc/project/)
[error] Not a valid command: assembly
[error] Not a valid project ID: assembly
[error] Expected ':'
[error] Not a valid key: assembly
[error] assembly
[error] ^
$
No web2warc-assembly-1.0.jar
appears to have been created. I am using Scala 2.12.7 on macOS 10.14.1. Please advise.
publish it to maven central or some other repository?
here, the seeds contains only one page, which is a "301 Moved Permanently" page.
even if followRedirects is set to true, the crawl stops here.
it works if I replace the seed url with a 200 page (then it crawls with depth 5, as expected).
Web2Warc.writer.path = "temp.warc.gz"
Web2Warc.spec.maxLevel = 5
Web2Warc.spec.followRedirects = true
Web2Warc.spec.increaseLevelOnRedirect = false
// Web2Warc.seeds = Set("https://epfl.ch") // this crawler crawls until depth 5.
Web2Warc.seeds = Set("http://epfl.ch") // the crawler only archives this page
Web2Warc.run()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.