machawk1/warcreate

Chrome extension to "Create WARC files from any webpage"

warcreate's Issues

When restoring default options, example preview does not update

Two WARC records are generated for the 'seed URI' (HTML), the latter having no body, causing blank replay

Thanks for the discovery, @nlevitt

App crashes on certain websites (see comments)

nasa.gov
cnn.com

Outlinks, as implemented, are not describing embedded resources

Resources embedded in resources require extra fetch to be retained, not accessible from webrequest

mailto links are incorrectly converted to base URI prepended form for outlinks

Wording on WARCreate hover about "clicking for options" is deceptive (there are no options) and obsolete to a previous UI, revise

Change over to using chrome.storage.local for binary image data

This is necessary for the move of the logic to onload from warc creation time.

http://developer.chrome.com/extensions/storage

Verify that the results from a variety of complexities of website match almost byte-for-byte to Heritrix output

All CSS Records' data is prepended with the "undefined" literal

Resources that required a subsequent request require their HTTP status code to be concatenated for creation

See bug #14 .

Status code and status message are separate variables. It doesn't look like the true HTTP status line returned exists in the jsxhr object, so I've had to manually add HTTP/1.1, which might not always be the case.

Body of 3XX redirects not captured

Provide support for Archive-It collections

Create digest entries for the WARC records to serve as better identifying unique content being captured

As suggested by Noah Levitt @ internet archive.

This would be handy for creating revisit records.

Content length in WARC response records for responses that contain binary image data is incorrect

Fixing this would probably fix a few other issues down the line.

Add feature to show, prior to creating WARC, which resources are currently ready and which still need to be fetched

Possibly implement this in a contextual dropdown option, which would allow filtering based on the current tab's context.

Special characters in a web page are mangled when saved to WARC

For example, in Mediawiki, the →‎ character is saved as character with hex 92.

With the additions of #8, the initial content record now has the content of the first HTTP request header duplicated.

Give Feedback for when options have been saved

Images in CSS files are not captured

Images embedded within CSS files aren't captured

Create ability for plugin to upload WARCs to a user-specified location/repo

Archiving a single resource results in the browser's encapsulated representation being preserved without the binary content

e.g., if I go to http://matkelly.com/_images/logo.png and say "Create WARC", the WARC created contains an HTML page with the image, likely created by the browser to display the image. Further, the binary information is not present.

outlink content of metadata record likely isn't correct

JavaScript files aren't captured

The method uses to capture CSS can probably be used here.

Images obtained from JS capture are not those that are received via curl

Which means the JS versions are not correct.

Robots.txt record is not included in the WARC record

It's questionable whether this should even be included but for the sake of comprehensive preservation maybe it should be fetched

If a URI is present on a page multiple times, multiple duplicate outlinks will be written in metadata records

If filename in WARC does not match actual filename, regardless of validation, the file is never indexed and replayable

The faux naming is a remnant of the reconstructions for image data acquisition and encoding in the WARC file.

JavaScript files only have request records, which prevents the WARC file from replaying

Image WARC record content length reporting is incorrect for images that required secondary (Ajax) fetching

When clicking restore defaults button in options, the save button is not re-enabled

When creating a second consecutive WARC from a different webpage, the extension crashes

When changing the URI target for upload, saving the preference then reloading the extension, the target URI is not restored in options

After Install, show an intro and options page

A la the Ghostery plugin. This will give folks an intro to what it is an initial options to set.

CSS Request headers are not written to WARC (but response headers are)

Limit WARC header/content capture to current tab only

Currently, asynchronous requests from other tabs pollute the WARC if they occur during the WARC creation process.

Switch to Semantic Versioning after 1.0 or when it proves to be more useful than date encoded versioning

http://semver.org/

Generated WARCs don't validate via cdx-indexer or ukwa's warc-explorer

DNS records are not retained

Document the code using something along the lines of doxygen

Currently, there is little more than inline documentation.

CSS Files are not captured

The host looks to not be prepended but the path portion is included in the WARC record.

An example:
WARC/1.0
WARC-Type: request
WARC-Target-URI: _css/azul_movil.css
WARC-Date: 2013-03-20T17:58:32.560Z
WARC-Concurrent-To: urn:uuid:02d04100-1a67-a427-559c-24f7889dd8ce
WARC-Record-ID: urn:uuid:72dff350-fd0d-95b2-a778-1ed78556a0ce
Content-Type: application/http; msgtype=request
Content-Length: 325

GET /s HTTP/1.1
Host: undefined
Connection: close
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7
Cache-Control: no-cache
Accept-Language: de,en;q=0.7,en-us;q=0.3

WARC/1.0
WARC-Type: response
WARC-Target-URI: _css/azul_movil.css
WARC-Date: 2013-03-20T17:58:32.560Z
WARC-Record-ID: urn:uuid:6ddadec2-be01-8694-304e-62a3d37ac3fa
Content-Type: application/http; msgtype=response
Content-Length: 604

HTTP/1.1 200 OK
Content-Type: text/css
Date: Wed Mar 20 2013 13:58:32 GMT-0400 (EDT) GMT
Last-Modified: Wed Mar 20 2013 13:58:32 GMT-0400 (EDT) GMT
Server: Apache/2.2.17 (Unix) PHP/5.3.5 mod_ssl/2.2.17 OpenSSL/0.9.8q
Accept-Ranges: bytes
Content-Type: text/css

<title>404 Not Found</title>

Not Found

The requested URL /getThatText.php was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

WARC response header is duplicated for initial HTML

Content before the HTML tag is not saved to the WARC file

This might have to do with

Example can be seen in the doctype declaration in the produced WARC.

Don't Build Blobs, Construct Them

http://updates.html5rocks.com/2012/06/Don-t-Build-Blobs-Construct-Them
BlobBuilder is deprecated with some revision to the HTML5 File API. Move over to Blob interface from BlobBuilder

Contents of WARC capture are from all open tabs, not just the current

Create a test suite for WARC Correctness

Rely on WARC data contained within the extension (or on the website) with some diff threshold to allow for (e.g.) date discrepancies .

Provide ability to create WARCs that are tar.gz

As suggested by Noah Levitt @ internet archive.

HTTP headers for images are not included

This was only applicable to images that required a subsequent Ajax call, e.g., embedded in CSS.

Provide option to allow user to customize the output filename scheme