htp_connp_(res|req)_data_consumed do not include CONNECT

[victorjulien]

htp_connp_(res|req)_data_consumed do not include CONNECT

[catenacyber]

I do not see in the shared pcaps a CONNECT request followed by a regular HTTP request
After the CONNECT requests (and the response), I see TLS...
Am I missing something ?

About your question, If you have 2 pipelined requests together in the same packet, I think you should have a hook in hook_request_start which will try to get the position.
htp_connp_req_data_consumed is "reset" on each call of htp_connp_req_data
So, you should track them, and them together I guess.
Or you can use the in_data_counter which looks accessible, it will tell how many bytes in total were provided...

[victorjulien]

Ah yes, I misdiagnosed what is happening. So what I'm seeing is that the response start hook is called for non-HTTP data

#5  0x00000000009c7151 in HTPCallbackResponseStart (tx=0x7ffff07b6b40) at app-layer-htp.c:2117
#6  0x00007ffff731052a in htp_hook_run_all (hook=0x2483310, user_data=user_data@entry=0x7ffff07b6b40) at htp_hooks.c:127
#7  0x00007ffff7319c76 in htp_tx_state_response_start (tx=0x7ffff07b6b40) at htp_transaction.c:1501
#8  0x00007ffff7316995 in htp_connp_res_data (connp=0x7ffff07b67f0, timestamp=<optimized out>, data=<optimized out>, len=<optimized out>) at htp_response.c:1327
#9  0x00000000009c4e44 in HTPHandleResponseData (f=0x352be90, htp_state=0x7ffff07b6760, pstate=0x7ffff07b6720, stream_slice=..., local_data=0x0) at app-layer-htp.c:943
#10 0x00000000009ef45f in AppLayerParserParse (tv=0x359ca00, alp_tctx=0x7ffff0531b80, f=0x352be90, alproto=1, flags=8 '\b', input=0x7ffff07b7d2f "\026\003\003", input_len=1460) at app-layer-parser.c:1289
#11 0x000000000097d3bf in AppLayerHandleTCPData (tv=0x359ca00, ra_ctx=0x7ffff052c430, p=0x7ffff04f3790, f=0x352be90, ssn=0x7ffff060c020, stream=0x7ffff6211200, data=0x7ffff07b7d2f "\026\003\003", 

(gdb) p stream_slice
$1 = {input = 0x7ffff07b7d2f "\026\003\003", input_len = 1460, flags = 8 '\b', offset = 95}

Guess I wasn't expecting this as its not HTTP anymore. Is that correct you think?

[victorjulien]

I suppose it makes sense if suri isn't doing the protocol upgrade.

in_data_counter is set before the callbacks are called, so it includes all data including what hasn't been processed yet. If we get more than one request in a single blob of data, we'll set in_data_counter to the whole blob. Then at the start of the 2nd request, I'm still unsure how to get an accurate stream offset. Perhaps I need to keep a tracker at the request_complete stage, which uses htp_connp_req_data_consumed somehow.

[victorjulien]

Ok I found a way to do this inside Suricata. I don't think libhtp provides a way to see the number of bytes the current TX has received, but I can work around that.

[catenacyber]

👏