ArchiveIterator is adding bytes to payload HTTP header without updating Content-length

[webrecorder]

ArchiveIterator is adding bytes to payload HTTP header without updating Content-length

[ikreymer]

Looks like this is related to to the Content-Disposition re-encoding that warcio is doing...

But what is it that you're trying to do.. remove the non-ascii headers altogether?
I believe to_ascii_bytes() already catches encoding errors, so that should not throw.

The issue you have can actually just be reproed with just:

with open(warcfilebasename + ".warc", 'rb') as f_in:
        with open(warcfilebasename + ".warc.gz", 'wb') as f_out:
            writer = WARCWriter(f_out, gzip=True)
            for record in ArchiveIterator(f_in):
                writer.write_record(record)

The issue happens because:

Content-Disposition: attachment; filename="BIZKAIKO-AIZKORA-TXAPELKETA-2ª-eus.pdf";

is canonicalized to:

Content-Disposition: attachment; filename*=UTF-8''BIZKAIKO-AIZKORA-TXAPELKETA-2%C2%AA-eus.pdf;

automatically, but for some reason the size is not adjusted.
This happens during the write_record, the iterator does not do the adjustment on read.
It should adjust the size though.

[ikreymer]

Now fixed on develop, thanks!

[ikreymer]

and deployed in 1.7.2