apnadkarni / iocp Goto Github PK
View Code? Open in Web Editor NEWImplements Tcl channels based on Windows I/O completion ports.
Home Page: https://iocp.magicsplat.com
License: BSD 2-Clause "Simplified" License
Implements Tcl channels based on Windows I/O completion ports.
Home Page: https://iocp.magicsplat.com
License: BSD 2-Clause "Simplified" License
iocp 2.0a0
when compiling with Visual Studio 2022, I get warnings:
tclWinIocpBT.c
.\win\tclWinIocpBT.c(750): warning C4090: "Funktion": Unterschiedliche "const"-Qualifizierer
.\win\tclWinIocpBT.c(768): warning C4090: "Funktion": Unterschiedliche "const"-Qualifizierer
(sorry for the german messages :-)
I can fix these by adding const
to the function signature of IocpNewWincharObj
, see patch attached.
R'
Setting -nagle option to 1 sets the TCP_NODELAY flag.
Shouldn't that be the opposite? In fact, setting TCP_NODELAY disable the Nagle algorithm.
This is rather a placeholder to me (I'll investigate deeper later)...
Following script illustrate the freeze:
if {![llength $::argv]} {
set ::argv [list ::iocp::inet::socket -async localhost 9999]
}
if {"::iocp::inet::socket" in $::argv} {
load tcliocpsock iocp
}
puts "WARN: ensure there is no listener on port [lindex $::argv end]"
puts "connect: \[$::argv\]"
set ch [{*}$::argv]
chan configure $ch -blocking 0
#chan event $ch writable ...; chan event $ch readable ...
puts "send test data"
chan puts -nonewline $ch zzz; chan flush $ch
# after 100 {puts "done."; set done 1}; vwait done
# puts "close"; close $ch
puts "exit."
Output with some debug (marked as **
):
WARN: ensure there is no listener on port 9999
connect: [::iocp::inet::socket -async localhost 9999]
send test data
exit.
** global NS deleted.
** interp deleted.
!FROZEN!
Prerequisites:
exit()
without proper cleanup);This is sporadic using tcl 8.6 in normal case, but pretty reproducible with tcl 8.5.
The reason for this difference is the TIP#398, so 8.6 simply doesn't try to flush by default, to force it one'd need to set environment var TCL_FLUSH_NONBLOCKING_ON_EXIT, for example:
set TCL_FLUSH_NONBLOCKING_ON_EXIT=1 && tclsh86 test-sock-async.tcl
Supplying socket -async localhost 9999
to script (so evaluating using tcl sockets) doesn't show the freeze at all.
Also execution in different thread/interp doesn't show it - looks like it could be some "conflicting" or deadlocking clean-up handler with the tcl-API finalization stage.
Moved from https://sourceforge.net/p/twapi/bugs/191/
Dear Ashok,
thank you for great twapi and iocp.
With the following lines, I get a block wish window (not responding any more) :
package require iocp_inet
rename ::socket ::socket_ori
rename ::iocp::inet::socket ::socket
package require twapi
package require http
proc ::http::Log {args} {
puts "[http]:$args"
}
http::register https 443 {::twapi::tls_socket}
http::geturl https://www.elmicron.de/labelers.csv
This is TWAPI (developper Version) 4.4.0, http 2.9.1 and wish 8.6.10 (VC6 self compiled), iocp 1.0.0.
I get the following console output (it is tricked, as the console stalls and copy&paste is not possible) (screenshot attached):
(bin) 6 % package require iocp_inet
1.0.0
(bin) 7 % rename ::socket ::socket_ori
(bin) 8 % rename ::iocp::inet::socket ::socket(bin) 1 % package require twapi
4.4.0
(bin) 2 % package require http
2.9.1
(bin) 3 % proc ::http::Log {args} {
puts "[http]:$args"
}
(bin) 4 % http::register https 443 {::twapi::tls_socket}
443 ::twapi::tls_socket
(bin) 5 % http::geturl https://www.elmicron.de/labelers.csv
[http]:^A1 URL https://www.elmicron.de/labelers.csv - token ::http::1
[http]:{Using rc0 for www.elmicron.de:443 - token ::http::1} {}
[http]:^B1 begin sending request - token ::http::1
[http]:^C1 end sending request - token ::http::1
[http]:^D1 begin receiving response - token ::http::1
When I remove the first 3 lines (iocp initialization), it works correctly:
(bin) 1 % package require twapi
4.4.0
(bin) 2 % package require http
2.9.1
(bin) 3 % proc ::http::Log {args} {
puts "[http]:$args"
}
(bin) 4 % http::register https 443 {::twapi::tls_socket}
443 ::twapi::tls_socket
(bin) 5 % http::geturl https://www.elmicron.de/labelers.csv
[http]:^A1 URL https://www.elmicron.de/labelers.csv - token ::http::1
[http]:{Using rc0 for www.elmicron.de:443 - token ::http::1} {}
[http]:^B1 begin sending request - token ::http::1
[http]:^C1 end sending request - token ::http::1
[http]:^D1 begin receiving response - token ::http::1
[http]:^E1 end of response headers - token ::http::1
[http]:^F1 end of response body (unchunked) - token ::http::1
[http]:{Closing socket rc0 (no connection info)}
::http::1
I am sorry for this report. Come to me if I may be of any help.
Thank you,
Harald
Package seems to be affected by same issue as Tcl - [b6d0d8cc2c]:
proc stats {} {
upvar port port; set state TIME_WAIT; # note it is localized on windows
puts [regexp -all ":$port\\s+$state" [exec netstat -n]]
}
set port 80; stats
timerate { time { close [socket localhost $port] } 50; after 100; stats }
results to this:
% set port 80; stats
0
% timerate { time { close [socket localhost $port] } 50; after 100; stats }
50
100
150
200
250
This test and test in next comment expecting replacement for socket
with ::iocp::inet::socket
or following inject for every interpreter (thread) used in test:
# inject default socket command:
namespace eval ::iocp::inet { namespace export socket }; rename ::socket {}; namespace import ::iocp::inet::socket
May be similar fix like [b960d1b71e] or some option (like -linger
) can be implemented to avoid such TIME-WAIT floods.
Or even both, so -linger auto
may be set as default, so decision whether lingering or graceful shutdown is expected can follow the implementation like [b960d1b71e].
Tcl 8.6.10 and earlier had a bug in running callbacks in the wrong thread on non-Windows platforms - https://core.tcl-lang.org/tcl/tktview/f583715154c6f33d6214
Copy the tests for the fix and ensure iocp is not susceptible to the same.
It looks like iocp package seems to miss an error (code) reporting stuff at several places (constructs like TclWinConvertError(WSAGetLastError())
or IocpSetTclErrnoFromWin32(WSAGetLastError())
...
Anyway, at least by server socket it is indeed NONE instead of EADDRINUSE:
+% set s [::socket -server dummy 80]
-% set s [::iocp::inet::socket -server dummy 80]
couldn't open socket: address already in use
% puts $::errorCode
+POSIX EADDRINUSE {address already in use}
-NONE
This is actually not a IOCP issue but rather a Tcl issue and common to all channels including Tcl core sockets. Details at https://core.tcl-lang.org/tcl/tktview?name=75d525d37c
Here as a placeholder.
Hello Ashok,
First of all, many thanks for your library - it literally saved my project. I find it a shame that your implementation of IOCP-based sockets still is not merged into core Tcl. However, I've run into a difficult to reproduce problem, and maybe you'll be able to help me.
My setup is a scientific instrument sending telemetry at a fairly high rate via 3 TCP connections (~650Mbit/s in total - standard Tcl sockets on Windows can't sustain such load).
It is received by 3 client processes (one process per connection), all running in a pure tclsh on Windows 10 Pro server (2x Xeon CPUs, hyperthreading off). Bitrate per connection is between 180-260 Mbit/s and doesn't change over time.
The traffic contains packets, that start with a standard PCAP header followed by some payload, sized between 5-10K each.
The incl_len field in the PCAP header tells me the size of the payload. I extract it using a simple
binary scan $pcapHdr @8i
Then the payload itself is simply written to an SSD disk using [chan copy]
The problem is that sometimes this incl_len field is extracted completely wrong (a negative value). In such case I log the whole PCAP header, and it doesn't look like a PCAP header. Instead it looks like somewhere in the middle of my payload, every time with the same offset. For now, it's dummy payload that looks like 0x0A0B0C0D etc.
So, it seems like part of a packet get duplicated or truncated, and so I loose sync on my incoming traffic, and can't read the next PCAP header. It happens quite rarely - maybe once or twice over the session of 8h.
There's a similar application written in C# that doesn't manifest this bug, so this rules out problems at the hardware/Windows level.
This is how I configure the socket:
chan configure $wzlSock -translation binary -buffersize 1000000 -keepalive 1
chan event $wzlSock readable [list WIZIF::readSocket $wzlSock]
proc readSocket reads one packet at a time. The socket is in the blocking mode. While troubleshooting this, I've added extra checks that every [read] returns as many bytes as requested (or throws an error in case of EOF, that shuts down the whole process)
sorcvbuf & sosndbuf are left at default 65K. Is it worth increasing them to 1M too?
If my description rings any bell for you, I'd like to hear.
Thanks in advance,
Petro
apn: just btw, found a small bug (or inconsistency either) - vtblPtr->getoption can be called with interp = NULL, e. g. https://github.com/apnadkarni/iocp/blob/master/win/tclWinIocp.c#L1472, but some of getoption callback uses interp handle to set error without to check it is NULL
See if zero- byte reads offer any performance benefits
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.