Peeker is zero-length after parsing a file

[Drakulix]

Peeker is zero-length after parsing a file

[Drakulix]

After debugging with gdb it seems that peeker is a zero-length slice after parsing the file. I have no idea, how that might happen.

[Drakulix]

Returning early correctly parses the upload, as expected:

Requested: Put: Url { scheme: "http", host: Ipv4(127.0.0.1), port: 8080, path: ["api", "v2", "package", ""], username: None, password: None, query: None, fragment: None } (Headers { Content-Length: 4259, Content-Type: multipart/form-data; boundary=---------------------------8d31b75ee271ce8, User-Agent: NuGet Command Line/2.8.60717.93 (Unix 4.3.3.2), Host: 127.0.0.1:8080, })
[Switching to Thread 0x7fffe79fe700 (LWP 6994)]

Python Exception <class 'OverflowError'> signed integer is greater than maximum: 
Breakpoint 1, params::run_state_machine<std::io::buffered::BufReader<&mut iron::request::Body>> (boundary=, 
    reader=0x7fffe79f9980, form_data=0x7fffe79f9940, mode=FormData)
    at /home/drakulix/.multirust/toolchains/stable/cargo/registry/src/github.com-0a35038f75765ae4/formdata-0.7.8/src/lib.rs:84
84      let boundary = boundary.into_bytes();
(gdb) c
Continuing.

Python Exception <class 'OverflowError'> signed integer is greater than maximum: 
Breakpoint 2, params::run_state_machine<std::io::buffered::BufReader<&mut iron::request::Body>> (boundary=, 
    reader=0x7fffe79f9980, form_data=0x7fffe79f9940, mode=FormData)
    at /home/drakulix/.multirust/toolchains/stable/cargo/registry/src/github.com-0a35038f75765ae4/formdata-0.7.8/src/lib.rs:104
104                     let peeker = try!(reader.fill_buf());
(gdb) c
Continuing.

Python Exception <class 'OverflowError'> signed integer is greater than maximum: 
Breakpoint 2, params::run_state_machine<std::io::buffered::BufReader<&mut iron::request::Body>> (boundary=, 
    reader=0x7fffe79f9980, form_data=0x7fffe79f9940, mode=FormData)
    at /home/drakulix/.multirust/toolchains/stable/cargo/registry/src/github.com-0a35038f75765ae4/formdata-0.7.8/src/lib.rs:104
104                     let peeker = try!(reader.fill_buf());
(gdb) ret
Make params::run_state_machine<std::io::buffered::BufReader<&mut iron::request::Body>> return now? (y or n) y
Python Exception <class 'OverflowError'> signed integer is greater than maximum: 
#0  0x00005555558d0d14 in params::parse_multipart<iron::request::Body> (stream=0x7fffe79fc220, boundary=)
    at /home/drakulix/.multirust/toolchains/stable/cargo/registry/src/github.com-0a35038f75765ae4/formdata-0.7.8/src/lib.rs:50
50      try!(run_state_machine(boundary, &mut reader, &mut form_data, MultipartSubLevel::FormData));
(gdb) c
Continuing.
Ok({"package": File(UploadedFile { path: "/tmp/formdata.9DjNRTXyLg4q/7ymVVgAAAADE40kASMm0XZZ6T1zCPD1d", filename: Some("package"), content_type: Mime(Application, Ext("octet-stream"), []), size: 4057, tempdir: "/tmp/formdata.9DjNRTXyLg4q" })})

[mikedilger]

Thanks for the report and the debugging.

I've tried to create a test case that replicates your data so I can consistently reproduce the incorrect behaviour. It currently has a wrong (made-up) content length, but it is nonetheless useful because it is currently reporting "The request body ended prior to reaching the expected terminating boundary," so I do believe we have a test case that shows the failure.

It's on the nuget_test branch.

Now, onwards towards solving this...

[Drakulix]

I can provide you with a complete request dump, if that helps. I just shortened it for the report. Also that request does get send as a PUT (different to your test case, which is POST), although that probably does not make any difference for parsing.

[mikedilger]

Yes, please supply the full request. I suspect that the Content-Length header does not match the body content length exactly, perhaps due to character encoding surprises.

[mikedilger]

... or the final boundary is not preceded by a CRLF.

I've verified that boundaries starting with lots of hyphens is not an issue.

[Drakulix]

The nuget client is very picky about the Servers it accepts and requires a chain of requests to succeed, before it will finally send the failing request. Iron does not give me the Raw Request, so I can only give you the complete Raw Body and all Headers in human-readable format, without writing a new server with another framework. I hope that is sufficient.

The Request logged (including Headers):

Put: Url { scheme: "http", host: Ipv4(127.0.0.1), port: 8080, path: ["api", "v2", "package", ""], username: None, password: None, query: None, fragment: None } (Headers { Content-Length: 4259, Content-Type: multipart/form-data; boundary=---------------------------8d31b628b7d7b04, User-Agent: NuGet Command Line/2.8.60717.93 (Unix 4.3.3.2), Host: 127.0.0.1:8080, })

The Body (Raw File https://www.dropbox.com/s/yq9tb5kipjfvvqv/tmp?dl=0 ):

-----------------------------8d31b628b7d7b04
Content-Disposition: form-data; name="package"; filename="package"
Content-Type: application/octet-stream

PK��
�i9G�   T�����
              �_rels/.rels ��(��<?xml version="1.0" encoding="utf-8"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Type="http://schemas.microsoft.com/packaging/2010/07/manifest" Target="/openssh.nuspec" Id="R2c08be68be154d1d" /><Relationship Type="http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties" Target="/package/services/metadata/core-properties/fb41da37a0ab40a2a731fdcc57dea506.psmdcp" Id="R94fbe8d8c8a34b2a" /></Relationships>PK����i9G󝷋l����openssh.nuspec ��(��}RMO�0
                                                                                                  �#�����t\&���p���0X���*x�u̕�|�j��7Ų�Ŋ�[P���,�1&,DY�(�(ź*}R�XB���&���sT
                                      ��<�)Km��|)����u4P������m!$�IW���E�'��e����|�W����sD����
�$�}����bp���{�Q�M�F��*�        ���OZ%"�)m�NAYK� ��8�W�*�6Ø��k'��d��M����z��S@��~I���!e쬕�뿩Δ�������t7?������������#u��O���tools/chocolateyinstall.ps1 ��(�����J�0�ůW�;���v�.�l�9z!*"�)~\�1�6k�Y���9T��|�/|�_�W0��pz�\���������s<e�.8����&�1�X����W�g��X��g[`V�`z�P�2��VͶ*S�����w��1��9E�8�8|��S�͖���a-�~��Or,��\������͍���۸��[�_�E���U�U�bX�0�+l؁5vY�<Y���e ����]ps�;��,���U$�S�`�c[�Q�_C����Q+E�LiLip���S����'8#�rA�R�
                                                         .y".T��R�Q�h�7�H��Q����n���Z�#�h!�d��OD1
FD҂�}VO  J��Q���"L`Ъ�"
                     �z�k[?PK����i9GP.��Q�E���tools/chocolateyinstall.ps1~ ��(�����J�0�ůW�;���vԶ�l�9r!*"�)~\�1�.k�]��l:T��|�/|�_�W0��pz�\���������s<e���AF�@����#�#��϶%���϶���e����e:���mU�ŧ2%���w:�1��9������@�i��oXK�zi�7�k�7�׵���1��w�Vs㋓d�Ѷ������E
               s���mXc��˓���|�-��������e�L&�"��J����Z������?��Z)�`J�Nr���k2�����\��T��K��
                                                                                          �Hp�Q�@���MD$$�)"P
                                                                                                            �m��\kq�E�A�F��A���Ȃ2������i LL�
                        c�qX��PK����i9G1�!PO��tools/chocolateyUninstall.ps1 ��(��{�{�pIbQ��sF~r~NbIje@Q~rjq�c�cJnf���~���z�U
        P<�(1W�-3'�8ƿ 5/8�#�4/3��$1'G/�"U�PK��
�i9G�´7����Q�package/services/metadata/core-properties/fb41da37a0ab40a2a731fdcc57dea506.psmdcp ��(��<?xml version="1.0" encoding="utf-8"?><coreProperties xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.openxmlformats.org/package/2006/metadata/core-properties"><dc:creator>brekenfeld</dc:creator><dc:description>OpenSSH Binaries for Windows with Windows Server Services</dc:description><dc:identifier>openssh</dc:identifier><version>7.1</version><keywords>openssh ssh ssh-client ssh-server sshd ssh-service</keywords><dc:title>OpenSSH Server and Client</dc:title><lastModifiedBy>choco, Version=0.9.9.8, Culture=neutral, PublicKeyToken=79d02ea9cad655eb;Microsoft Windows NT 6.1.7601 Service Pack 1;.NET Framework 4</lastModifiedBy></coreProperties>PK��
�i9G�R
P������[Content_Types].xml ��(��<?xml version="1.0" encoding="utf-8"?><Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml" /><Default Extension="nuspec" ContentType="application/octet" /><Default Extension="ps1" ContentType="application/octet" /><Default Extension="ps1~" ContentType="application/octet" /><Default Extension="psmdcp" ContentType="application/vnd.openxmlformats-package.core-properties+xml" /></Types>PK��-
�i9G�   T�����
u��O����tools/chocolateyinstall.ps1PK��-��i9GP.��Q�E���tools/chocolateyinstall.ps1~PK��-��i9G1�!PO�9tools/chocolateyUninstall.ps1PK��-
�i9G�´7����Q�package/services/metadata/core-properties/fb41da37a0ab40a2a731fdcc57dea506.psmdcpPK��-
�i9G�R
P�����f
       [Content_Types].xmlPK����
-----------------------------8d31b628b7d7b04--

[Drakulix]

It should be a CRLF after the file end for various reasons:

• Vim shows the ^M sign for the \r (so it is not a unix line break)
• There is no reason the State Machine should go back to ParsingHeaders from CapturingFile, if the boundary would be incorrect. The stream_until_token call should abort instead of reading all data to the EOF, right?

[mikedilger]

The raw file has the following bytes preceding the final boundary: 0D 00 00 00 00 0A
The four NUL bytes separating the CR from the LF make it an invalid CRLF.

[mikedilger]

stream_until_token reads right up to the end, fails to find the token, and returns Ok(n) (the comments above the function say it returns Ok(n) even if the token was not found). Then it goes back to ParsingHeaders, but the buffer is now empty.

[Drakulix]

I hope I did not dump it wrong, as this seems really weird and I am not sure, why the client should do this. I will try to verify that the actual request is this messed up

[Drakulix]

I am suspecting, that the actual line ending might be an LF instead of a CRLF and the CR is actually still part of the binary file.

[Drakulix]

Okay so the actual code in the nuget client, that constructs the request is the following:
https://github.com/NuGet/NuGet2/blob/2.11/src/Core/Http/MultipartWebRequest.cs

As one can see in line 17 and 74 the "\r\n" is hardcoded (which means it's not unix, that is inserting just an "\n") and correctly added to the request. I also verified that in my Mono Runtime the byte representation of \r\n is actually 0x0D0A.

But even wireshark is showing the "0D 00 00 00 00 0A", so I am very confused how that happens.
Thanks anyway.

[Drakulix]

Quick note, if anybody encounters this: The 0D is actually not part of the CRLF (tested with other files), so its probably still a unix issue.