machawk1/warcreate

Chrome extension to "Create WARC files from any webpage"

warcreate's Issues

JavaScript files only have request records, which prevents the WARC file from replaying

Change over to using chrome.storage.local for binary image data

This is necessary for the move of the logic to onload from warc creation time.

http://developer.chrome.com/extensions/storage

Content before the HTML tag is not saved to the WARC file

This might have to do with

Example can be seen in the doctype declaration in the produced WARC.

With the additions of #8, the initial content record now has the content of the first HTTP request header duplicated.

JavaScript files aren't captured

The method uses to capture CSS can probably be used here.

CSS Files are not captured

The host looks to not be prepended but the path portion is included in the WARC record.

An example:
WARC/1.0
WARC-Type: request
WARC-Target-URI: _css/azul_movil.css
WARC-Date: 2013-03-20T17:58:32.560Z
WARC-Concurrent-To: urn:uuid:02d04100-1a67-a427-559c-24f7889dd8ce
WARC-Record-ID: urn:uuid:72dff350-fd0d-95b2-a778-1ed78556a0ce
Content-Type: application/http; msgtype=request
Content-Length: 325

GET /s HTTP/1.1
Host: undefined
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7
Cache-Control: no-cache
Accept-Language: de,en;q=0.7,en-us;q=0.3

WARC/1.0
WARC-Type: response
WARC-Target-URI: _css/azul_movil.css
WARC-Date: 2013-03-20T17:58:32.560Z
WARC-Record-ID: urn:uuid:6ddadec2-be01-8694-304e-62a3d37ac3fa
Content-Type: application/http; msgtype=response
Content-Length: 604

HTTP/1.1 200 OK
Content-Type: text/css
Date: Wed Mar 20 2013 13:58:32 GMT-0400 (EDT) GMT
Last-Modified: Wed Mar 20 2013 13:58:32 GMT-0400 (EDT) GMT
Server: Apache/2.2.17 (Unix) PHP/5.3.5 mod_ssl/2.2.17 OpenSSL/0.9.8q
Accept-Ranges: bytes
Content-Type: text/css

<title>404 Not Found</title>

Not Found

The requested URL /getThatText.php was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

Give Feedback for when options have been saved

Provide option to allow user to customize the output filename scheme

outlink content of metadata record likely isn't correct

Images embedded within CSS files aren't captured

Create ability for plugin to upload WARCs to a user-specified location/repo

Limit WARC header/content capture to current tab only

Currently, asynchronous requests from other tabs pollute the WARC if they occur during the WARC creation process.

Special characters in a web page are mangled when saved to WARC

For example, in Mediawiki, the →‎ character is saved as character with hex 92.

App crashes on certain websites (see comments)

nasa.gov
cnn.com

When clicking restore defaults button in options, the save button is not re-enabled

mailto links are incorrectly converted to base URI prepended form for outlinks

Generated WARCs don't validate via cdx-indexer or ukwa's warc-explorer

CSS Request headers are not written to WARC (but response headers are)

DNS records are not retained

Archiving a single resource results in the browser's encapsulated representation being preserved without the binary content

e.g., if I go to http://matkelly.com/_images/logo.png and say "Create WARC", the WARC created contains an HTML page with the image, likely created by the browser to display the image. Further, the binary information is not present.

When changing the URI target for upload, saving the preference then reloading the extension, the target URI is not restored in options

Switch to Semantic Versioning after 1.0 or when it proves to be more useful than date encoded versioning

http://semver.org/

Images obtained from JS capture are not those that are received via curl

Which means the JS versions are not correct.

Two WARC records are generated for the 'seed URI' (HTML), the latter having no body, causing blank replay

Thanks for the discovery, @nlevitt

Document the code using something along the lines of doxygen

Currently, there is little more than inline documentation.

Robots.txt record is not included in the WARC record

It's questionable whether this should even be included but for the sake of comprehensive preservation maybe it should be fetched

All CSS Records' data is prepended with the "undefined" literal

Contents of WARC capture are from all open tabs, not just the current

If filename in WARC does not match actual filename, regardless of validation, the file is never indexed and replayable

The faux naming is a remnant of the reconstructions for image data acquisition and encoding in the WARC file.

HTTP headers for images are not included

This was only applicable to images that required a subsequent Ajax call, e.g., embedded in CSS.

If a URI is present on a page multiple times, multiple duplicate outlinks will be written in metadata records

Wording on WARCreate hover about "clicking for options" is deceptive (there are no options) and obsolete to a previous UI, revise

Outlinks, as implemented, are not describing embedded resources

Content length in WARC response records for responses that contain binary image data is incorrect

Fixing this would probably fix a few other issues down the line.

Create a test suite for WARC Correctness

Rely on WARC data contained within the extension (or on the website) with some diff threshold to allow for (e.g.) date discrepancies .

When creating a second consecutive WARC from a different webpage, the extension crashes

Body of 3XX redirects not captured

Images in CSS files are not captured

After Install, show an intro and options page

A la the Ghostery plugin. This will give folks an intro to what it is an initial options to set.

Add feature to show, prior to creating WARC, which resources are currently ready and which still need to be fetched

Possibly implement this in a contextual dropdown option, which would allow filtering based on the current tab's context.

When restoring default options, example preview does not update

Provide ability to create WARCs that are tar.gz

As suggested by Noah Levitt @ internet archive.

Verify that the results from a variety of complexities of website match almost byte-for-byte to Heritrix output

Resources that required a subsequent request require their HTTP status code to be concatenated for creation

See bug #14 .

Status code and status message are separate variables. It doesn't look like the true HTTP status line returned exists in the jsxhr object, so I've had to manually add HTTP/1.1, which might not always be the case.

Image WARC record content length reporting is incorrect for images that required secondary (Ajax) fetching

Create digest entries for the WARC records to serve as better identifying unique content being captured

As suggested by Noah Levitt @ internet archive.

This would be handy for creating revisit records.

WARC response header is duplicated for initial HTML

Don't Build Blobs, Construct Them

http://updates.html5rocks.com/2012/06/Don-t-Build-Blobs-Construct-Them
BlobBuilder is deprecated with some revision to the HTML5 File API. Move over to Blob interface from BlobBuilder

Provide support for Archive-It collections

Resources embedded in resources require extra fetch to be retained, not accessible from webrequest