anacrolix / utp Goto Github PK
View Code? Open in Web Editor NEWUse anacrolix/go-libutp instead
Home Page: https://github.com/anacrolix/go-libutp
License: Mozilla Public License 2.0
Use anacrolix/go-libutp instead
Home Page: https://github.com/anacrolix/go-libutp
License: Mozilla Public License 2.0
As with the torrent package, this needs a license if it's to be usable by anyone else. :)
If i want to listen on utp packets, what i need to do?
As i see package does not have Listen method, so i need to listen udp specific port and use NewSocketFromPacketConn?
Does does it possible to create Listen method?
I'm not sure if this is actually an issue, but the LocalAddr
and RemoteAddr
methods on a Conn return an address whose network identifies as udp. I think these should return utp{4,6} instead, as its more accurate.
I currently wrap the utp conn from your package to change this functionality, but if you could add it upstream that would be very helpful.
the readme says:
It opts for simplicity and reliability over strict adherence to the (poor) spec
Could you please note somewhere which things you're talking about specifically? That way we can make sure to watch out for them. (I've found many bugs in transport protocol implementations, and usually notes like these helped find them).
By the way, uTP should be LEDBAT (RFC 6817), so it should be well specced out. Wikipedia notes:
but the details of the µTP implementation are different from those of the draft.[19][broken citation]
I haven't dived deeply into either to know what's different, but would also be relevant to document somewhere.
I suspect that Apple's LEDBAT extensions to the TCP stack (used for software updates) may also be helpful: https://opensource.apple.com/source/xnu/xnu-1699.32.7/bsd/netinet/tcp_ledbat.c
skt, err := utp.NewSocket("udp", ":0")
skt.SetReadDeadline(time.Now().Add(time.Second))
_, _, err = skt.ReadFrom(nil)
blocks forever.
Not sure what is the reason you do not call SO_REUSEADDR or lazyDestroy() does the job but I unable to restart application. Especially It hurts on Android.
Can you make Socket.destroy() exported if it helps? Or how to wait until socket will be destoryed?
Using this benchmark:
import (
"net"
"testing"
"github.com/anacrolix/utp"
)
func getTCPConnectionPair() (net.Conn, net.Conn, error) {
lst, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
return nil, nil, err
}
var conn0 net.Conn
var err0 error
done := make(chan struct{})
go func() {
conn0, err0 = lst.Accept()
close(done)
}()
conn1, err := net.Dial("tcp", lst.Addr().String())
if err != nil {
return nil, nil, err
}
<-done
if err0 != nil {
return nil, nil, err0
}
return conn0, conn1, nil
}
func getUTPConnectionPair() (net.Conn, net.Conn, error) {
lst, err := utp.NewSocket("udp", "127.0.0.1:0")
if err != nil {
return nil, nil, err
}
var conn0 net.Conn
var err0 error
done := make(chan struct{})
go func() {
conn0, err0 = lst.Accept()
close(done)
}()
conn1, err := utp.Dial(lst.Addr().String())
if err != nil {
return nil, nil, err
}
<-done
if err0 != nil {
return nil, nil, err0
}
return conn0, conn1, nil
}
func benchConnPair(b *testing.B, c0, c1 net.Conn) {
b.ReportAllocs()
b.SetBytes(128 << 10)
b.ResetTimer()
request := make([]byte, 52)
response := make([]byte, (128<<10)+8)
pair := []net.Conn{c0, c1}
for i := 0; i < b.N; i++ {
if i%2 == 0 {
pair[0] = c0
pair[1] = c1
} else {
pair[0] = c1
pair[1] = c0
}
if _, err := pair[0].Write(request); err != nil {
b.Fatal(err)
}
if _, err := pair[1].Read(request[:8]); err != nil {
b.Fatal(err)
}
if _, err := pair[1].Read(request[8:]); err != nil {
b.Fatal(err)
}
if _, err := pair[1].Write(response); err != nil {
b.Fatal(err)
}
if _, err := pair[0].Read(response[:8]); err != nil {
b.Fatal(err)
}
if _, err := pair[0].Read(response[8:]); err != nil {
b.Fatal(err)
}
}
}
func BenchmarkTCP(b *testing.B) {
conn0, conn1, err := getTCPConnectionPair()
if err != nil {
b.Fatal(err)
}
defer conn0.Close()
defer conn1.Close()
benchConnPair(b, conn0, conn1)
}
func BenchmarkUTP(b *testing.B) {
conn0, conn1, err := getUTPConnectionPair()
if err != nil {
b.Fatal(err)
}
defer conn0.Close()
defer conn1.Close()
benchConnPair(b, conn0, conn1)
}
I get the following:
PASS
BenchmarkTCP-4 5000 263246 ns/op 497.91 MB/s 27 B/op 0 allocs/op
BenchmarkUTP-4 2016/05/26 23:21:51 error resending packet: write udp [::]:56345->127.0.0.1:56344: use of closed network connection
2016/05/26 23:21:51 error resending packet: write udp [::]:56345->127.0.0.1:56344: use of closed network connection
2016/05/26 23:21:51 error resending packet: write udp [::]:56345->127.0.0.1:56344: use of closed network connection
100 134251556 ns/op 0.98 MB/s 1195857 B/op 4016 allocs/op
If you tweak the numbers a bit, you might get it to 0.2MB/s
I did manage to get it to a point where it just mostly never completes and fails with "connection closed", but that was using syncthings protocol wrappers which do async read/writes. In cases where it did complete it was around 0.3MB/s
calling Dial
opens a new socket, then calls dial on that socket. closing the returned connection doesnt appear to close the opened socket, leading to a file descriptor leak.
I have been consistently hitting this panic in my application, and curious as to the reasoning of it, and how I may be triggering it?
https://github.com/anacrolix/utp/blob/master/socket.go#L484
The stack trace goes back to this call:
https://github.com/anacrolix/torrent/blob/master/client.go#L383
My application is handling tens of thousands of torrents, and it works very well, but eventually (<hr) hits this panic. I am not quite sure how to handle this. I am unable to produce sample code to trigger this, as although I am reliably producing it, it is infrequent and hard to track down which action is causing it.
Edit: I also am getting this output fair frequently when dealing with this load.
resetting conflicting syn
So that we could do the STUN dance or whatever is required before wrapping it in UTP.
Similarly how you can wrap and unwrap a TCP connection with tls.Server or tls.Client.
I'm having some of my ipfs nodes die of OOM, and looking at the heap profile I see that utp was using 100MB (of my machines 256MB of RAM). It all appears to have been allocated by the sendBufferPool
's allocation function. That seems a bit much, and i'm sure some of this is user error on my part. Any ideas how to debug the issue?
Hey @anacrolix,
FYI when trying to use anacrolix/torrent
with cgo
disabled, the following error pops up:
# github.com/securityscorecard/tm-torrents-discovery/vendor/github.com/anacrolix/torrent
vendor/github.com/anacrolix/torrent/utp_go.go:10: cannot use *utp.Socket as type utpSocket in return argument:
*utp.Socket does not implement utpSocket (missing DialContext method)
Hi
I tested with go 1.5 on Mac Air:
$ go get github.com/anacrolix/utp
$ go get github.com/anacrolix/utp/cmd/ucat
$ ./bin/ucat -l -p 9876 >/dev/null &
$ dd if=/dev/zero bs=1m count=100 | ./bin/ucat 127.0.0.1 9876
100+0 records in
100+0 records out
104857600 bytes transferred in 18.542269 secs ( *** 5655058 bytes/sec *** )
2015/09/07 03:51:28 wrote 104857600 bytes
2015/09/07 03:51:28 received 0 bytes
2015/09/07 03:51:28 received 104857600 bytes
This is compared to native build of ucat from https://github.com/bittorrent/libutp:
$ ./build/Release/ucat -l -p 9876 >/dev/null &
$ dd if=/dev/zero bs=1m count=1000 | ./build/Release/ucat -B $((256*1024)) 127.0.0.1 9876
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 11.888499 secs ( *** 88200875 bytes/sec *** )
That's a big drop..
Cheers
If i create a socket for accepting new connections on, I want to keep that open even if all the connections that have been made to me get closed, That expectation is violated by this line: https://github.com/anacrolix/utp/blob/master/utp.go#L1252
I think that line only needs to be executed if the connections drop to zero AND the socket was created through a 'dial' (or even then probably not). Its probably the case that the right thing to do is to only close the actual socket when 'Close' is called.
This repository currently deprecated in favour of https://github.com/anacrolix/go-libutp. Issues, PRs and comments are still accepted.
I guess this relies on the socket having had at least a single connection?
package main
import(
"time"
"github.com/anacrolix/utp"
)
func main() {
s, err := utp.NewSocket("udp", ":0")
if err == nil {
s.SetReadDeadline(time.Now().Add(time.Second))
}
}
panic: deadline callback is nil
goroutine 1 [running]:
panic(0x6f3120, 0xc08200af40)
C:/Go/src/runtime/panic.go:464 +0x3f4
github.com/anacrolix/utp.(*deadlineCallback).updateTimer(0xc0820ae078)
C:/Go/bin/src/github.com/anacrolix/utp/utp.go:89 +0xde
github.com/anacrolix/utp.(*deadlineCallback).setDeadline(0xc0820ae078, 0xeceae9880, 0x24eb8a24, 0x9dd8a0)
C:/Go/bin/src/github.com/anacrolix/utp/utp.go:96 +0x4e
github.com/anacrolix/utp.(*connDeadlines).SetReadDeadline(0xc0820ae078, 0xeceae9880, 0x24eb8a24, 0x9dd8a0, 0x0, 0x0)
C:/Go/bin/src/github.com/anacrolix/utp/utp.go:116 +0x57
main.main()
C:/Users/Audrius/AppData/Local/liteide/goplay.go:12 +0x14a
exit status 2
I am trying to hunt down an issue I have with this library where I notice that the uTP transfer does not back-off when there is TCP background traffic.
Tracing this issue I noticed that an extensive number of packets are being resend. I noticed this was primarily due to this call to resend
which is caused by this call to ackSkipped
. Using wireshark I noticed that lots of ACK packets do have the Selective ACKs
extension enabled but do not have an extension bitmask:
According to the specification the length should be:
Note that the len field of extensions refer to bytes, which in this extension must be at least 4, and in multiples of 4.
There are also ACK packets that do have this set correctly:
Now as a quick hack I've tried not calling ackSkipped
when there are no bytes in the extension:
switch ext.Type {
case extensionTypeSelectiveAck:
if len(ext.Bytes) > 0 {
c.ackSkipped(h.AckNr + 1)
bitmask := selectiveAckBitmask{ext.Bytes}
for i := 0; i < bitmask.NumBits(); i++ {
if bitmask.BitIsSet(i) {
nr := h.AckNr + 2 + uint16(i)
c.ack(nr)
} else {
c.ackSkipped(h.AckNr + 2 + uint16(i))
}
}
}
}
and I've also changed ackSkipped
to not resend the packets that often:
switch send.acksSkipped {
case 60, 120: // was 3, 60
if logLevel >= 1 {
log.Printf("acksSkipped = %d", send.acksSkipped)
}
ackSkippedResends.Add(1)
send.resend()
send.resendTimer.Reset(c.resendTimeout() * time.Duration(send.numResends))
default:
}
For me this is dramatically decreasing the number of duplicates that are being send.
However it does not fix my issue with uTP traffic throttling back when there is TCP background traffic.
Does this make sense?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.