Comments (19)
vel21ripn
commented on August 16, 2024
1
I will try to fix this problem. An error can be reproduced on a short fragment:
tcpdump -r lineVideoCall1.pcap -c 23 -w lineVideoCall2.pcap udp
from ndpi.
vel21ripn
commented on August 16, 2024
Don't use "cat /proc/net/xt_ndpi/flows". Use "ndpi_flow_dump -m flows -s"
The command "cat /proc/net/xt_ndpi/flows" is reset to the counters, and streams with zero values are not displayed.
I have a tool that allows you to skip traffic via xt_ndpi and give it to ndpiReader in which the difference is visible for each packet. However, that this process is time-consuming.
Perhaps I can do this next week.
from ndpi.
sharonenoch
commented on August 16, 2024
Thanks for the quick response.. Will perform the test once more and use ndpi_flow_dump command to print the flows..
We actually enabled the debug and flows in nDPI driver only after we saw that the firewall logs were not getting printed for the UDP flows of Zoom and Line.
Is there any other debug that we can enable to get more logs, so that it would helpful in debugging this flow
from ndpi.
sharonenoch
commented on August 16, 2024
So we did 2 tests yesterday
- The latest code in flow_info-4. This did not detect zoom traffic. Below is the ndpi_flow_dump as you requested. You can check the destination port 8801 showing up as Unknown and not detected as Zoom traffic. This code base had 308 IPs as part of the zoom ip database
bash-4.4# ./ndpi_flow_dump -m flows -s
TIME 1668781095
1668781095 1668781095 4 17 172.9.0.100 8184 25.34.217.37 46722 2352 0 21 0 I=6,5 SN=10.200.3.109,8184 P=Unknown
1668781055 1668781094 4 17 172.9.0.100 8184 144.195.16.253 3478 1440 0 20 0 I=6,5 SN=10.200.3.109,8184 P=stun
1668781055 1668781094 4 17 172.9.0.100 14465 144.195.16.253 3478 1440 0 20 0 I=6,5 SN=10.200.3.109,14465 P=stun
1668781045 1668781045 4 6 172.9.0.100 50107 118.214.137.233 80 399 395 4 3 I=6,5 SN=10.200.3.109,50107 P=http H=x1.c.lencr.org
1668781040 1668781042 4 6 172.9.0.100 50106 170.114.15.101 443 2752 5883 8 9 I=6,5 SN=10.200.3.109,50106 P=zoom,tls H=us04logfiles.zoom.us
1668781002 1668781092 4 17 172.9.0.100 63915 144.195.17.165 8801 8410 7404 74 76 I=6,5 SN=10.200.3.109,63915 P=Unknown
1668781001 1668781070 4 6 172.9.0.100 50101 3.19.157.213 443 9479 6502 13 14 I=6,5 SN=10.200.3.109,50101 P=zoom,tls H=evt-us.ds.corp.zoom.us
1668781001 1668781092 4 17 172.9.0.100 50596 144.195.17.165 8801 6674 5964 53 55 I=6,5 SN=10.200.3.109,50596 P=Unknown
1668781001 1668781095 4 17 172.9.0.100 50595 144.195.17.165 8801 5294019 4528460 5681 4211 I=6,5 SN=10.200.3.109,50595 P=Unknown
1668780999 1668781095 4 6 172.9.0.100 50098 144.195.17.165 443 84455 46214 386 421 I=6,5 SN=10.200.3.109,50098 P=zoom,tls H=zoomsjcar165mmr.sjc.zoom.us
1668780997 1668781090 4 6 172.9.0.100 50090 170.114.52.5 443 3907 18507 19 25 I=6,5 SN=10.200.3.109,50090 P=zoom,tls H=us05www3.zoom.us
1668780024 1668780973 4 6 172.9.0.100 50012 170.114.14.73 443 2950 6472 24 16 I=6,5 SN=10.200.3.109,50012 P=zoom,tls H=us04zpns.zoom.us
1668779968 1668781094 4 6 172.9.0.100 49984 170.114.15.169 443 22636 24268 145 120 I=6,5 SN=10.200.3.109,49984 P=zoom,tls H=us04xmpp1.zoom.us
1668779969 1668781091 4 6 172.9.0.100 49995 170.114.52.5 443 10596 16234 179 180 I=6,5 SN=10.200.3.109,49995 P=zoom,tls H=us05web.zoom.us
1668780029 1668781095 4 6 172.9.0.100 50014 170.114.52.5 443 9176 13560 167 168 I=6,5 SN=10.200.3.109,50014 P=zoom,tls H=us05www3.zoom.us
1668780029 1668781095 4 6 172.9.0.100 50015 170.114.52.5 443 9174 14039 167 168 I=6,5 SN=10.200.3.109,50015 P=zoom,tls H=us05www3.zoom.us
- We reverted back to a code base in Feb 2022(commit b15f1f6) and Zoom was detected always in this code base. Below are the ndpi_flow_dump logs which show the detected Zoom on port 8801. Here we did update all the latest zoom ips to get it to the same 308 ips via the ip_proto proc as that old code base had only 52 ips.
bash-4.4# ./ndpi_flow_dump -m flows -s
TIME 1668777895
1668777858 1668777894 4 17 172.8.0.100 11275 144.195.77.213 3478 1368 0 19 0 I=8,7 SN=10.200.3.122,11275 P=zoom,stun
1668777858 1668777894 4 17 172.8.0.100 9949 144.195.77.213 3478 1368 0 19 0 I=8,7 SN=10.200.3.122,9949 P=zoom,stun
1668777828 1668777894 4 6 172.8.0.100 50101 170.114.15.104 443 2795 5994 9 11 I=8,7 SN=10.200.3.122,50101 P=zoom,tls H=us04logfiles.zoom.us
1668777784 1668777893 4 17 172.8.0.100 59238 144.195.76.87 8801 9259 8283 81 83 I=8,7 SN=10.200.3.122,59238 P=zoom
1668777784 1668777858 4 6 172.8.0.100 50093 3.19.157.213 443 9479 6502 13 14 I=8,7 SN=10.200.3.122,50093 P=zoom,tls H=evt-us.ds.corp.zoom.us
1668777784 1668777893 4 17 172.8.0.100 65398 144.195.76.87 8801 7904 7104 63 65 I=8,7 SN=10.200.3.122,65398 P=zoom
1668777783 1668777895 4 17 172.8.0.100 65397 144.195.76.87 8801 4893142 4029779 4499 3769 I=8,7 SN=10.200.3.122,65397 P=zoom
1668777782 1668777895 4 6 172.8.0.100 50091 144.195.76.87 443 84812 47573 394 432 I=8,7 SN=10.200.3.122,50091 P=zoom,tls H=zoomsjccy87mmr.sjc.zoom.us
1668777779 1668777887 4 6 172.8.0.100 50083 170.114.52.5 443 3957 18470 20 24 I=8,7 SN=10.200.3.122,50083 P=zoom,tls H=us05www3.zoom.us
1668777747 1668777886 4 6 172.8.0.100 50077 170.114.52.5 443 3254 8047 19 19 I=8,7 SN=10.200.3.122,50077 P=zoom,tls H=us05www3.zoom.us
1668777747 1668777886 4 6 172.8.0.100 50076 170.114.52.5 443 3216 7157 18 18 I=8,7 SN=10.200.3.122,50076 P=zoom,tls H=us05www3.zoom.us
1668777699 1668777880 4 6 172.8.0.100 50066 54.192.219.252 443 2796 13547 16 20 I=8,7 SN=10.200.3.122,50066 P=zoom,tls H=marketplacecontent.zoom.us
1668777694 1668777876 4 6 172.8.0.100 50058 205.251.222.254 443 10167 540109 214 382 I=8,7 SN=10.200.3.122,50058 P=zoom,tls H=marketplacefront.zoom.us
1668777685 1668777895 4 6 172.8.0.100 50038 170.114.15.160 443 12876 16106 67 50 I=8,7 SN=10.200.3.122,50038 P=zoom,tls H=us04xmpp1.zoom.us
1668777684 1668777886 4 6 172.8.0.100 50035 170.114.52.5 443 4615 9736 24 23 I=8,7 SN=10.200.3.122,50035 P=zoom,tls H=us05web.zoom.us
1668777683 1668777821 4 6 172.8.0.100 50032 170.114.14.67 443 2300 6127 13 12 I=8,7 SN=10.200.3.122,50032 P=zoom,tls H=us04zpns.zoom.us
1668777682 1668777894 4 6 172.8.0.100 50029 170.114.52.5 443 13714 70520 54 75 I=8,7 SN=10.200.3.122,50029 P=zoom,tls H=us05web.zoom.us
from ndpi.
vel21ripn
commented on August 16, 2024
I fixed the bug due to missing ndpi_detection_giveup() call ce5cca1.
It's not clear to me yet whether LRU zoom is being used.
I see in the output of ndpiReader confidence "DPI (partial cache)", and in /proc/net/xt_ndpi/flows confidence 2 "Match by IP". This needs to be dealt with.
from ndpi.
IvanNardi
commented on August 16, 2024
It's not clear to me yet whether LRU zoom is being used.
@vel21ripn, speaking only of the upstream-nDPI, if you find the word "cache" in the confidence description it means some kind of LRU has been used.
from ndpi.
vel21ripn
commented on August 16, 2024
IMHO Commit d238ff6 must be resolve this bug.
from ndpi.
vel21ripn
commented on August 16, 2024
Fixed cf017cc
from ndpi.
sharonenoch
commented on August 16, 2024
Thankyou for the quick fix. I will get back to you on Monday as I am currently travelling and don't have access to a setup to test the same..
from ndpi.
sharonenoch
commented on August 16, 2024
Zoom is detecting properly now.. Thankyou
Linecall seems to still have some issues.. Also I noticed you added 1 more fix from the initial fix.. Will add that in also in the testing.. Once I gather a proper pcap will keep you updated
from ndpi.
sharonenoch
commented on August 16, 2024
For Line we are seeing the behavior to be not so consistent in the driver, but the ndpiReader is always detecting the same.
lineVideoCall1.zip..
Attached pcap has a linecall where the UDP flow is from 172.8.0.100 to 147.92.169.21
Following filter can be used in Wireshark
ip.dst == 147.92.128.0/17 || ip.src == 147.92.128.0/17
Below is the nDPI reader output.. The flow in question is below..
UDP 172.8.0.100:63719 <-> 147.92.169.21:16673
root@kickseed:~/sharonnDPI/nDPI# ./example/ndpiReader -i /root/lineVideoCall1.pcap -v2 -v3
-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel
* free to extend it and send us the patches for inclusion
------------------------------------------------------------
Using nDPI (4.5.0-3885-2d6403c7) [1 thread(s)]
Using libgcrypt version 1.8.6internal
Reading packets from pcap file /root/lineVideoCall1.pcap...
Running thread 0...
nDPI Memory statistics:
nDPI Memory (once): 36.59 KB
Flow Memory (per flow): 872 B
Actual Memory: 13.43 MB
Peak Memory: 13.43 MB
Setup Time: 48 msec
Packet Processing Time: 11 msec
Traffic statistics:
Ethernet bytes: 7902540 (includes ethernet CRC/IFC/trailer)
Discarded bytes: 0
IP packets: 10000 of 10000 packets total
IP bytes: 7662540 (avg pkt size 766 bytes)
Unique flows: 6
TCP Packets: 352
UDP Packets: 9648
VLAN Packets: 0
MPLS Packets: 0
PPPoE Packets: 0
Fragmented Packets: 0
Max Packet size: 1480
Packet Len < 64: 319
Packet Len 64-128: 1371
Packet Len 128-256: 1000
Packet Len 256-1024: 6179
Packet Len 1024-1500: 1131
Packet Len > 1500: 0
nDPI throughput: 876.50 K pps / 5.16 Gb/sec
Analysis begin: 29/Nov/2022 05:04:05
Analysis end: 29/Nov/2022 05:07:07
Traffic throughput: 54.72 pps / 337.81 Kb/sec
Traffic duration: 182.760 sec
Guessed flow protos: 0
DPI Packets (TCP): 27 (6.75 pkts/flow)
DPI Packets (UDP): 28 (14.00 pkts/flow)
Confidence: DPI 6 (flows)
Detected protocols:
Line packets: 352 bytes: 156269 flows: 4
LineCall packets: 9648 bytes: 7506271 flows: 2
Protocol statistics:
Acceptable 7662540 bytes
Risk stats [found 4 (66.7 %) flows with risks]:
TLS (probably) Not Carrying HTTPS 4 [100.0 %]
NOTE: as one flow can have multiple risks set, the sum of the
last column can exceed the number of flows with risks.
JA3C/JA3S Host Stats:
IP JA3C JA3S
1 172.8.0.100 ca75ea4a95a9164cc96e372d7d075183
2 172.8.0.100 9f65608af90d890731d8863bc910d713
3 147.92.242.232 567bb420d39046dbfd1f68b558d86382
4 147.92.165.68 15af977ce25de452b96affa2addb1036
5 147.92.191.86 e35df3e00ca4ef31d42b34bebaa2f86e
IP/JA3 Distribution:
JA3 IP
1 JA3C ca75ea4a95a9164cc96e372d7d075183 172.8.0.100
2 JA3C 9f65608af90d890731d8863bc910d713 172.8.0.100
3 JA3S 567bb420d39046dbfd1f68b558d86382 147.92.242.232
4 JA3S 15af977ce25de452b96affa2addb1036 147.92.165.68
5 JA3S e35df3e00ca4ef31d42b34bebaa2f86e 147.92.191.86
1 UDP 172.8.0.100:63719 <-> 147.92.169.21:16673 [proto: 316/LineCall][IP: 315/Line][Encrypted][Confidence: DPI][cat: VoIP/10][7730 pkts/6828431 bytes <-> 1908 pkts/675896 bytes][Goodput ratio: 95/88][49.38 sec][bytes ratio: 0.820 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/0 4/18 809/9153 13/262][Pkt Len c2s/s2c min/avg/max/stddev: 62/62 883/354 1098/1089 295/340][Plen Bins: 1,16,31,14,9,10,2,0,0,0,1,0,0,0,0,0,12,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
2 TCP 172.8.0.100:59382 <-> 147.92.165.68:443 [proto: 91.315/TLS.Line][IP: 315/Line][Encrypted][Confidence: DPI][cat: Chat/9][112 pkts/13678 bytes <-> 159 pkts/116839 bytes][Goodput ratio: 56/92][180.21 sec][Hostname/SNI: gwz.line.naver.jp][bytes ratio: -0.790 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 2133/1262 59585/59585 9257/7420][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 122/735 1494/1514 162/679][Risk: ** TLS (probably) Not Carrying HTTPS **][Risk Score: 10][Risk Info: No ALPN][TLSv1.3][JA3C: 9f65608af90d890731d8863bc910d713][JA3S: 15af977ce25de452b96affa2addb1036][Firefox][Cipher: TLS_AES_256_GCM_SHA384][Plen Bins: 0,17,11,7,3,4,1,2,0,1,0,1,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,47,0,0]
3 TCP 172.8.0.100:59398 <-> 147.92.191.86:443 [proto: 91.315/TLS.Line][IP: 315/Line][Encrypted][Confidence: DPI][cat: Chat/9][11 pkts/2747 bytes <-> 12 pkts/7378 bytes][Goodput ratio: 78/91][8.43 sec][Hostname/SNI: lan.line.me][bytes ratio: -0.457 (Download)][IAT c2s/s2c min/avg/max/stddev: 1/0 154/118 278/239 118/118][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 250/615 571/1514 235/574][Risk: ** TLS (probably) Not Carrying HTTPS **][Risk Score: 10][Risk Info: No ALPN][TLSv1.2][JA3C: ca75ea4a95a9164cc96e372d7d075183][ServerNames: *.line.me,line.me][JA3S: e35df3e00ca4ef31d42b34bebaa2f86e][Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign RSA OV SSL CA 2018][Subject: C=JP, ST=Tokyo-to, L=Shinjuku-ku, O=LINE Corporation, CN=*.line.me][Certificate SHA-1: 72:BB:C0:78:1C:91:A0:DF:F3:49:32:BC:6F:56:F6:BE:AE:48:14:03][Firefox][Validity: 2022-08-08 08:36:05 - 2023-09-09 08:36:04][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 7,0,0,7,0,0,0,7,7,0,0,7,0,0,0,21,7,0,7,0,0,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,21,0,0]
4 TCP 172.8.0.100:59406 <-> 147.92.242.232:443 [proto: 91.315/TLS.Line][IP: 315/Line][Encrypted][Confidence: DPI][cat: Chat/9][15 pkts/3997 bytes <-> 19 pkts/5157 bytes][Goodput ratio: 79/79][33.60 sec][Hostname/SNI: uts-front.line-apps.com][bytes ratio: -0.127 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 1/0 2988/202 29999/1100 8544/315][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 266/271 627/1514 232/435][Risk: ** TLS (probably) Not Carrying HTTPS **][Risk Score: 10][Risk Info: No ALPN][TLSv1.2][JA3C: ca75ea4a95a9164cc96e372d7d075183][ServerNames: *.line-apps.com,line-apps.com][JA3S: 567bb420d39046dbfd1f68b558d86382][Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign RSA OV SSL CA 2018][Subject: C=JP, ST=Tokyo-to, L=Shinjuku-ku, O=LINE Corporation, CN=*.line-apps.com][Certificate SHA-1: C8:EB:57:1A:17:32:27:8D:C4:3A:D3:27:58:FF:64:A8:5C:FE:88:51][Firefox][Validity: 2022-10-13 17:31:02 - 2023-11-14 17:31:01][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 5,11,5,0,0,16,5,0,5,16,0,0,0,0,0,0,11,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,11,0,0]
5 TCP 172.8.0.100:59379 <-> 147.92.191.86:443 [proto: 91.315/TLS.Line][IP: 315/Line][Encrypted][Confidence: DPI][cat: Chat/9][15 pkts/1929 bytes <-> 9 pkts/4544 bytes][Goodput ratio: 57/89][123.87 sec][Hostname/SNI: lan.line.me][bytes ratio: -0.404 (Download)][IAT c2s/s2c min/avg/max/stddev: 1/0 10157/1248 88558/7000 23990/2575][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 129/505 571/1514 166/569][Risk: ** TLS (probably) Not Carrying HTTPS **][Risk Score: 10][Risk Info: No ALPN][TLSv1.2][JA3C: ca75ea4a95a9164cc96e372d7d075183][ServerNames: *.line.me,line.me][JA3S: e35df3e00ca4ef31d42b34bebaa2f86e][Issuer: C=BE, O=GlobalSign nv-sa, CN=GlobalSign RSA OV SSL CA 2018][Subject: C=JP, ST=Tokyo-to, L=Shinjuku-ku, O=LINE Corporation, CN=*.line.me][Certificate SHA-1: 72:BB:C0:78:1C:91:A0:DF:F3:49:32:BC:6F:56:F6:BE:AE:48:14:03][Firefox][Validity: 2022-08-08 08:36:05 - 2023-09-09 08:36:04][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 11,0,0,11,0,0,0,11,11,0,0,0,0,0,11,0,11,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,22,0,0]
6 UDP 172.8.0.100:54010 <-> 147.92.169.21:16673 [proto: 316/LineCall][IP: 315/Line][Encrypted][Confidence: DPI][cat: VoIP/10][5 pkts/898 bytes <-> 5 pkts/1046 bytes][Goodput ratio: 77/80][8.08 sec][bytes ratio: -0.076 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 1999/1999 2011/2011 2044/2043 19/19][Pkt Len c2s/s2c min/avg/max/stddev: 174/198 180/209 202/254 11/22][Plen Bins: 0,0,0,0,80,10,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
Source Ports Stats:
1 Port 63719 [1 IP address(es)/1 flows/7730 pkts/6828431 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
2 Port 59382 [1 IP address(es)/1 flows/112 pkts/13678 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
3 Port 59406 [1 IP address(es)/1 flows/15 pkts/3997 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
4 Port 59379 [1 IP address(es)/1 flows/15 pkts/1929 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
5 Port 59398 [1 IP address(es)/1 flows/11 pkts/2747 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
6 Port 54010 [1 IP address(es)/1 flows/5 pkts/898 bytes]
Top IP Stats:
172.8.0.100 ~ 100.00%
Destination Ports Stats:
1 Port 16673 [1 IP address(es)/2 flows/1913 pkts/676942 bytes]
Top IP Stats:
147.92.169.21 ~ 100.00%
2 Port 443 [3 IP address(es)/4 flows/199 pkts/133918 bytes]
Top IP Stats:
147.92.191.86 ~ 50.00%
147.92.242.232 ~ 25.00%
147.92.165.68 ~ 25.00%
The driver ndpi_flow_dump output is as follows
bash-4.4# ./ndpi_flow_dump -m flows -s
TIME 1669716380
1669716378 1669716379 4 17 172.8.0.100 63719 147.92.169.21 16673 10195 10054 19 19 I=8,7 SN=10.200.3.141,63719 P=Unknown
1669716369 1669716373 4 6 172.8.0.100 59406 147.92.242.232 443 3747 4692 14 16 I=8,7 SN=10.200.3.141,59406 P=line,tls H=uts-front.line-apps.com
1669716369 1669716369 4 17 172.8.0.100 63705 1.1.1.1 53 69 85 1 1 I=8,7 SN=10.200.3.141,63705 P=line,dns H=uts-front.line-apps.com
1669716350 1669716350 4 6 172.8.0.100 59401 23.50.244.182 443 1267 5166 7 9 I=8,7 SN=10.200.3.141,59401 P=line,tls H=desktop.line-scdn.net
1669716349 1669716350 4 17 172.8.0.100 61358 1.1.1.1 53 67 203 1 1 I=8,7 SN=10.200.3.141,61358 P=line,dns H=desktop.line-scdn.net
1669716341 1669716350 4 6 172.8.0.100 59398 147.92.191.86 443 2593 7118 11 10 I=8,7 SN=10.200.3.141,59398 P=line,tls H=lan.line.me
1669716341 1669716341 4 17 172.8.0.100 51788 1.1.1.1 53 57 129 1 1 I=8,7 SN=10.200.3.141,51788 P=line,dns H=lan.line.me
1669716245 1669716376 4 6 172.8.0.100 59382 147.92.165.68 443 11996 113651 110 143 I=8,7 SN=10.200.3.141,59382 P=line,tls H=gwz.line.naver.jp
bash-4.4# ./ndpi_flow_dump -m flows -s
TIME 1669716421
1669716378 1669716421 4 17 172.8.0.100 63719 147.92.169.21 16673 5540436 391891 6389 1400 I=8,7 SN=10.200.3.141,63719 P=Unknown
1669716369 1669716403 4 6 172.8.0.100 59406 147.92.242.232 443 3787 4803 15 18 I=8,7 SN=10.200.3.141,59406 P=line,tls H=uts-front.line-apps.com
1669716369 1669716369 4 17 172.8.0.100 63705 1.1.1.1 53 69 85 1 1 I=8,7 SN=10.200.3.141,63705 P=line,dns H=uts-front.line-apps.com
1669716350 1669716411 4 6 172.8.0.100 59401 23.50.244.182 443 1387 5277 10 11 I=8,7 SN=10.200.3.141,59401 P=line,tls H=desktop.line-scdn.net
1669716349 1669716350 4 17 172.8.0.100 61358 1.1.1.1 53 67 203 1 1 I=8,7 SN=10.200.3.141,61358 P=line,dns H=desktop.line-scdn.net
1669716341 1669716350 4 6 172.8.0.100 59398 147.92.191.86 443 2593 7118 11 10 I=8,7 SN=10.200.3.141,59398 P=line,tls H=lan.line.me
1669716341 1669716341 4 17 172.8.0.100 51788 1.1.1.1 53 57 129 1 1 I=8,7 SN=10.200.3.141,51788 P=line,dns H=lan.line.me
1669716245 1669716376 4 6 172.8.0.100 59382 147.92.165.68 443 11996 113651 110 143 I=8,7 SN=10.200.3.141,59382 P=line,tls H=gwz.line.naver.jp
Please let me know if there is any other output that we can provide to help debug the same
from ndpi.
vel21ripn
commented on August 16, 2024
We need change parameters for module xt_ndpi "max_unk_udp=32 max_unk_tcp=32 max_unk_other=32"
You can change options via sysfs
echo 32 >/sys/module/xt_ndpi/parameters/max_unk_udp
from ndpi.
sharonenoch
commented on August 16, 2024
Thanks @vel21ripn . We tested this change and it is detecting Linecall now.. I feel based on our earlier tests also, analyzing 32 packets should be sufficient to detect line. I am marking this issue as fixed
bash-4.4# ./ndpi_flow_dump -m flows -s | grep 192.168.64.100
1669975118 1669975782 4 17 192.168.64.100 56121 147.92.169.18 24273 22580643 11854103 46984 48696 I=8,7 SN=10.200.3.42,56121 P=linecall
1669975066 1669975736 4 6 192.168.64.100 50010 147.92.165.68 443 20168 144609 177 225 I=8,4 SN=100.64.236.70,50010 P=line,tls H=gwz.line.naver.jp
1669975020 1669975782 4 6 192.168.64.100 49951 203.17.244.49 443 6075 6900 150 150 I=8,7 SN=10.200.3.42,49951 P=tls
from ndpi.
sharonenoch
commented on August 16, 2024
@vel21ripn So we were retesting zoom and it stopped getting detected, while nDPIreader shows it properly in all the pcaps.
We have narrowed it down to when we add the 'max_unk_udp=32 max_unk_tcp=32 max_unk_other=32" during driver load time to increase the packets which are traversed for the line, is when zoom stops getting detected. Can you please check..
We performed 4 runs.. Below are the results
modprobe xt_ndpi max_unk_udp=32 max_unk_tcp=32 max_unk_other=32 ndpi_enable_flow=1 ndpi_flow_limit=500 -> Does not detect zoom. it detect Line fine.
Run 2 -> modprobe xt_ndpi max_unk_udp=32 max_unk_tcp=32 max_unk_other=32 -> Does not detect zoom. it detect Line fine.
Run 3 -> modprobe xt_ndpi ndpi_enable_flow=1 ndpi_flow_limit=500 -> It detects zoom but does not detect Line
Run 4 -> modprobe xt_ndpi -> It detects zoom but does not detect Line
We also checked with the latest tree also which had a commit for improved zoom detection.
from ndpi.
vel21ripn
commented on August 16, 2024
Can you give me a sample of the traffic that shows the error?
from ndpi.
sharonenoch
commented on August 16, 2024
Attached pcap. I trimmed the tcpdump based on the following filter ip.src == 144.195.22.81 || ip.dst == 144.195.22.81 and captured the first 10000 packets as the call was more than 10 minutes long.. Hopefully the behavior remains the same when simulated with those parameters (max_unk_udp=32 max_unk_tcp=32 max_unk_other=32).. if not we will capture one more dump.
Below is the ndpi reader output.
-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel
* free to extend it and send us the patches for inclusion
------------------------------------------------------------
Using nDPI (4.5.0-4208-dcff3ef8) [1 thread(s)]
Using libgcrypt version 1.8.6internal
Reading packets from pcap file zoom_new.pcap...
Running thread 0...
nDPI Memory statistics:
nDPI Memory (once): 36.86 KB
Flow Memory (per flow): 912 B
Actual Memory: 8.25 MB
Peak Memory: 8.25 MB
Setup Time: 71 msec
Packet Processing Time: 6 msec
Traffic statistics:
Ethernet bytes: 3839694 (includes ethernet CRC/IFC/trailer)
Discarded bytes: 0
IP packets: 7056 of 7056 packets total
IP bytes: 3670350 (avg pkt size 520 bytes)
Unique flows: 8
TCP Packets: 1217
UDP Packets: 5826
VLAN Packets: 0
MPLS Packets: 0
PPPoE Packets: 0
Fragmented Packets: 0
Max Packet size: 1480
Packet Len < 64: 909
Packet Len 64-128: 1974
Packet Len 128-256: 698
Packet Len 256-1024: 1063
Packet Len 1024-1500: 2412
Packet Len > 1500: 0
nDPI throughput: 1.14 M pps / 4.63 Gb/sec
Analysis begin: 05/Jan/2023 11:56:55
Analysis end: 05/Jan/2023 12:01:07
Traffic throughput: 28.02 pps / 119.13 Kb/sec
Traffic duration: 251.798 sec
Guessed flow protos: 6
DPI Packets (TCP): 6 (6.00 pkts/flow)
DPI Packets (UDP): 118 (19.67 pkts/flow)
DPI Packets (other): 1 (1.00 pkts/flow)
Confidence: DPI (partial cache) 6 (flows)
Confidence: DPI 2 (flows)
Detected protocols:
ICMP packets: 13 bytes: 1134 flows: 1
Zoom packets: 7043 bytes: 3669216 flows: 7
from ndpi.
vel21ripn
commented on August 16, 2024
Please check commit 59c0a10
from ndpi.
sharonenoch
commented on August 16, 2024
Thankyou... Will check with that fix
from ndpi.
sharonenoch
commented on August 16, 2024
Thanks @vel21ripn .. We tested this and it detects both zoom and linecall. Closing the issue
from ndpi.
Related Issues (20)
- При попытке установить ndpi-netfilter на debian 9 возникает ошибка (при этом, библиотека nDPI собирается успешно). HOT 6
- flow_info-4 build errors HOT 1
- how to enable remotescan, qqlive protocols detection (not initialized by default)? HOT 2
- Cant compile HOT 1
- libxt_ndpi.so link error after iptables version 1.8.9 HOT 1
- The --clevel options can not work HOT 2
- nDPI Flow Risk feature HOT 10
- Request for clarification on installing nDPI Netfilter with xtables on Debian 11 HOT 4
- Make throws an error that skbuff.h could not be found HOT 5
- make modules_install with kernel headers of different version HOT 10
- Kernel panic на ядре 4.9 HOT 12
- block icmp
- Block icmp HOT 1
- Risk add command doesn't match actual rule. HOT 5
- protocol line and linecall not recognised by iptables HOT 2
- modify iptables extension for better performance HOT 5
- Policy Based Routing does not work. HOT 10
- Flow Risk ID 27 - Risky Domain Name - Default behavior HOT 6
- New build issues of master HOT 1
- Валятся тесты из папки tests после сборки HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ndpi.