Support for WARCs based on version 1.1 of the spec?

[webrecorder]

Support for WARCs based on version 1.1 of the spec?

[ikreymer]

Support for reading and writing WARC 1.1 added in warcio 1.6.0

[ikreymer]

@sebastian-nagel yeah, i think you're right, while we've been cautious to start writing 1.1 WARCs, we should definitely support reading WARC/1.1. We can look into this soon.

[machawk1]

In selectively importing parts of the warcio API, I can persuade the below to process examples with the above scenario:

from warcio.archiveiterator import ArchiveIterator
from warcio.recordloader import ArcWarcRecordLoader
ArcWarcRecordLoader.WARC_TYPES.append('WARC/1.1')

warc11 = '(pathtomywarc)'

with open(warc11, 'rb') as stream:
    for record in ArchiveIterator(stream):
        if record.rec_type == 'response':
            print(record.rec_headers.get_header('WARC-Date'))

...but this is a dirty hack and does not account for the other features of the 1.1 spec. With that also in mind, WARC-Dates that are invalid per the WARC/1.0 spec but legal per WARC/1.1 (e.g., 2014-02-10T00:00:01.000000002Z) do not throw any sort of validation error when processed with warcio 95d5dcd.

My above question (plans?) still remains. I am hoping to finally get around to integrating warcio into ipwb for oduwsdl/ipwb#380 and oduwsdl/ipwb#374.

[sebastian-nagel]

Similar issue: warcio index fails on a WARC file of version 1.1:

warcio.recordloader.ArchiveLoadFailed: Unknown archive format, first line: ['WARC/1.1']

The mentioned work-around (add WARC/1.1 to WARC_TYPES) is not applicable to command-line tools.
The only 1.1 feature I plan to use is the WARC-Refers-To-Date header in revisit records. Warcio does not seem to have issues with unknown headers. If there is already partial support for WARC/1.1 (simply because the differences to 1.0 are small), why not claim to support it?

[nicholasamorim]

Any news on Warc 1.1 support?