Segmentation fault starting thread manager

[linuxton]

Segmentation fault starting thread manager

[droe]

No, this is definitely not issue #9. Could you post the output of sslsplit -V, a more complete debug log, plus ideally also a stack trace?

[linuxton]

Sure, here they are:

./sslsplit -V
SSLsplit 0.4.7 (built 2013-08-23)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-0.4.7/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit 0.4.7 (built 2013-08-23)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

• [0.0.0.0]:8443 ssl plain netfilter
  Loaded CA: '/C=US/ST=test/L=test/O=test/OU=test/CN=N/emailAddress=[email protected]'
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  [New Thread 0xb7b87b70 (LWP 16616)]
  [New Thread 0xb7386b70 (LWP 16617)]
  [New Thread 0xb6b85b70 (LWP 16618)]
  Inserted events:
  0x8075160 [fd 11] Read Persist
  0x8077f1c [fd 12] Read Persist
  0x8075050 [fd 10] Read Persist
  0x80751b0 [fd 3] Signal Persist
  0x8078080 [fd 1] Signal Persist
  0x8078160 [fd 2] Signal Persist
  0x8078240 [fd 13] Signal Persist
  Failed to start thread manager

Program received signal SIGSEGV, Segmentation fault.
0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thanks for your help

[linuxton]

Could I just replace proxy.c and pxythrmgr.c in sslsplit directory with the ones updated by you or should I tar the original archive again and use make with the replaced files?

I tried the former but it still gives segmentation fault issue. Is there anything else I'd log to show you?

Thanks for your time

[droe]

I suggest you do a git clone of master in order to test the latest code, that is safer than updating only those two files. Can you post the output of bt full in gdb after each segmentation fault you encounter?

[linuxton]

Ok this is a -V of latest sslsplit git clone:

./sslsplit -V
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

This is a try to run it:
./sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

• [0.0.0.0]:8443 ssl plain netfilter
  Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=[email protected]'
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  Inserted events:
  0x8555198 [fd 7] Read Persist
  0x8556ccc [fd 8] Read Persist
  0x8555088 [fd 6] Read Persist
  0x8556d18 [fd 3] Signal Persist
  0x8556e80 [fd 1] Signal Persist
  0x8556f60 [fd 2] Signal Persist
  0x8557040 [fd 13] Signal Persist
  Failed to start thread manager

And this is a gdb run of it:
(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-fc/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

• [0.0.0.0]:8443 ssl plain netfilter
  Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=[email protected]'
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  [New Thread 0xb7b87b70 (LWP 3970)]
  [New Thread 0xb7386b70 (LWP 3971)]
  [New Thread 0xb6b85b70 (LWP 3972)]
  Inserted events:
  0x8075160 [fd 11] Read Persist
  0x8077d14 [fd 12] Read Persist
  0x8075050 [fd 10] Read Persist
  0x80751b0 [fd 3] Signal Persist
  0x8077e78 [fd 1] Signal Persist
  0x8077f58 [fd 2] Signal Persist
  0x8078038 [fd 13] Signal Persist
  Failed to start thread manager
  [Thread 0xb7b87b70 (LWP 3970) exited]
  [Thread 0xb7386b70 (LWP 3971) exited]
  [Thread 0xb6b85b70 (LWP 3972) exited]
  [Inferior 1 (process 3967) exited normally]

Using a 'bt full' at this point gives a 'No stack.' message. Should I configure a ulimit number to successfully execute a bt full in gdb after running the program or is it normal since no segmentation fault error was in output?

I ran the previous "standard" 0.4.7 version of sslsplit and did a bt full, thought it may have been useful to you since I forgot to add it last time:

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-0.4.7/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit 0.4.7 (built 2013-08-26)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

• [0.0.0.0]:8443 ssl plain netfilter
  Loaded CA: '/C=US/ST=Alabama/L=Some/O=Internet Windows Ltd/OU=On/CN=N/emailAddress=[email protected]'
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  [New Thread 0xb7b87b70 (LWP 6251)]
  [New Thread 0xb7386b70 (LWP 6252)]
  [New Thread 0xb6b85b70 (LWP 6253)]
  Inserted events:
  0x8075160 [fd 11] Read Persist
  0x8077efc [fd 12] Read Persist
  0x8075050 [fd 10] Read Persist
  0x80751b0 [fd 3] Signal Persist
  0x8078060 [fd 1] Signal Persist
  0x8078140 [fd 2] Signal Persist
  0x8078220 [fd 13] Signal Persist
  Failed to start thread manager

Program received signal SIGSEGV, Segmentation fault.
0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
(gdb) bt full
#0 0xb7c527e1 in free () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#1 0x080590c4 in pxy_thrmgr_free ()
No symbol table info available.
#2 0x08054e44 in proxy_free ()
No symbol table info available.
#3 0x080536b6 in main ()
No symbol table info available.

quit
A debugging session is active.

Inferior 1 [process 6293] will be killed.

Quit anyway? (y or n)

Is it normal that executing two different 'run' in gdb gives different LWP numbers (see above 'run' of latest github clone, LWP 3970/3971/3972 while this log gives LWP 4071/4072/4073)?

(gdb) run -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
Starting program: /root/sslsplit-fc/sslsplit -D -l connections.log -j /tmp/sslsplit/ -S logdir/ -k ca.key -c ca.crt ssl 0.0.0.0 8443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Generated RSA key for leaf certs.
SSLsplit fc (built 2013-08-25)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected
proxyspecs:

• [0.0.0.0]:8443 ssl plain netfilter
  Loaded CA: '/C=aa/ST=aa/L=aaa/O=aaaa/OU=aaa/CN=aaa/emailAddress=[email protected]'
  Using libevent backend 'epoll'
  Event base supports: edge yes, O(1) yes, anyfd no
  [New Thread 0xb7b87b70 (LWP 4071)]
  [New Thread 0xb7386b70 (LWP 4072)]
  [New Thread 0xb6b85b70 (LWP 4073)]
  Inserted events:
  0x8075160 [fd 11] Read Persist
  0x8077ccc [fd 12] Read Persist
  0x8075050 [fd 10] Read Persist
  0x8077d18 [fd 3] Signal Persist
  0x8077e80 [fd 1] Signal Persist
  0x8077f60 [fd 2] Signal Persist
  0x8078040 [fd 13] Signal Persist
  Failed to start thread manager
  [Thread 0xb7b87b70 (LWP 4071) exited]
  [Thread 0xb7386b70 (LWP 4072) exited]
  [Thread 0xb6b85b70 (LWP 4073) exited]
  [Inferior 1 (process 4068) exited normally]

Is there anything else I'd log?

Thanks for your time

[edannenberg]

Same error here, seems to be related to the -j option. If i try without -j it starts just fine.

sslsplit -V
SSLsplit 0.4.7-6-gfc29806 (built 2013-11-21)
Copyright (c) 2009-2013, Daniel Roethlisberger [email protected]
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter: IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.21-stable
rtlinked against libevent 2.0.21-stable
2 CPU cores detected

[exvance]

[Comment moved to new separate issue #11 since it is possibly unrelated.]

[antonywu]

If I get Failed to start thread manager without Segmentation fault does that mean it is a different issue?

FYI, This is on a RaspberryPi

$ uname -a
Linux chaitea 3.6.11+ #538 PREEMPT Fri Aug 30 20:42:08 BST 2013 armv6l GNU/Linux
$ sudo ./sslsplit -V
SSLsplit 0.4.7-6-gfc29806 (built 2013-12-12)
Copyright (c) 2009-2013, Daniel Roethlisberger <[email protected]>
http://www.roe.ch/SSLsplit
Features: -DDISABLE_SSLV2_SESSION_CACHE -DHAVE_NETFILTER
NAT engines: netfilter* tproxy
netfilter:  IP_TRANSPARENT SOL_IPV6 !IPV6_ORIGINAL_DST
compiled against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
rtlinked against OpenSSL 1.0.1e 11 Feb 2013 (1000105f)
TLS Server Name Indication (SNI) supported
OpenSSL is thread-safe with THREADID
Using direct access workaround when loading certs
SSL/TLS algorithm availability: RSA DSA ECDSA DH ECDH EC
OpenSSL option availability: SSL_OP_NO_COMPRESSION SSL_OP_NO_TICKET SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION SSL_OP_DONT_INSERT_EMPTY_FRAGMENTS SSL_OP_NO_SESSION_RESUMPTION_ON_RENEGOTIATION SSL_OP_TLS_ROLLBACK_BUG
compiled against libevent 2.0.19-stable
rtlinked against libevent 2.0.19-stable
1 CPU cores detected

[droe]

There are actually two issues; one issue is that the thread manager fails to start (which I cannot reproduce so far), and the other is that sslsniff crashes after the thread manager fails to start (which I can reproduce by simulating a thread manager startup failure). The two commits referenced in this issue fixed many (all?) of potential sources of segmentation faults after thread manager startup failure, so you should not get any segmentation faults anymore if you use latest master. There is no fix for the failing thread manager yet.

[droe]

If you are using latest the latest git master version and still encounter a segmentation fault following Failed to start thread manager - please attach a stack trace to this issue.

If you are able to reproduce the Failed to start thread manager in a stable way, please attach your operating system details to this issue: uname -a and specific information on the distribution and version. Also, details on the compiler and libc used to build SSLsplit would be great.

[droe]

I'm closing the issue since I believe the segmentation faults to be fixed. If you encounter any Failed to start thread manager errors in latest master, please open a new ticket.