Local port forwarding example

[jjsp556]

Local port forwarding example

[pkittenis]

Hi there,

Thanks for the interest.

This is non-trivial - PRs welcome. There is such implementation in parallel-ssh if looking for a ready-made solution. See API documentation.

[Red-M]

I've tried to do this and I can say it is very hard to do.

I've run into some performance issues in doing this that seem to be unavoidable and I'd love a non gevent version of the parallel-ssh implementation that I can run with just pure threads.

I tried to do it but I get 1 thread burning CPU cycles on a single core after a successful connection. I've debugged my code and I can't locate why it is doing this.

If the channel objects had a fileno function for select to use, the example from paramiko for its tunneling could be dropped right in with a single thread for the whole solution.

[pkittenis]

As above, it is non trivial. If it's burning cycles that probably means a missing select calls somewhere, or a select that returns 0 with no check leading to hot loop. Paramiko is not relevant - different API. There is also a C code example at libssh2 documentation.

[Red-M]

Both threads for:

• reading from the downstream socket and writing to the ssh channel
• reading from the ssh channel and writing to the downstream socket

are in a blocked state for my testing.

Paramiko is using in it's example of port forwarding, select.select() which could be compatible from my testing by implementing a chan.fileno() function to allow the builtin select.select() to work as per the example by replacing line 58 with the ssh2-python's own channel object and replacing for send and recv, with the addition suggested above.

However I'd ideally want this in C rather than in Python code.

[pkittenis]

C code example here. Paramiko is not relevant.

For a working implementation see parallel-ssh (note select is used, as is a separate thread). Usual libssh2 rules apply - cannot share sessions/channels across threads. Not surprising if it is getting blocked when the code is attempting to read and write from the same channel in different threads.

[pkittenis]

For reference, @Red-M :

• Copyrighted code may not be used without copyright information intact
• Derivative works of open source licensed code must be distributed under a compatible license with copyright information intact

Copy-pasting copyrighted code that is under license and stripping copyright and license information is a breach of both copyright and terms of license.

For issues regarding tunneling implementation in parallel-ssh, an issue should be raised on that project. This is an issue tracker for ssh2-python, not for debugging ad-hoc or derivative works of this or any other project.

When there is working code for this issue that can be reviewed, a PR may be raised.

[Red-M]

Hey, sorry about all that, I haven't been feeling the best.

I've got a working implementation that should still be reasonably fast but I would like to confirm with you first if it is ok to release. I've had to rewrite a few functions since my project doesn't use gevents.

I could contribute a working example for ssh2-python from such code.

My project uses a Linux kernel style GPL2 license.

[pkittenis]

Hi,

If it's a from scratch implementation there is no need to check with me. If it is based on parallel-ssh code then a compatible license must be used, as well as copyright info retained but as long as that is done the license allows modifications as well as distribution. Both projects are LGPL v2.1 only which allows distribution as GPLv2 only (not later) per GNU license compatibility matrix.

If it resolves the performance issue when a client connects - which I've seen too - can consider contributing the changes to parallel-ssh as well, or I can back port them separately. Thanks for considering to contribute.

[Red-M]

I've got similar implementations for eagain wrappers (read, write and generic functions) that were based from yours except they use my project's implementation of a client for ssh2-python and the stdlib for select and not gevent. I've kept the copyright for that in just those functions as the rest is either my own or another MIT project.
The local tunneling is from paramiko but using eagain wrappers that I mentioned from before.

A sample:

    def _block(self, func, *args, **kwargs):
        # This function was part of parallel-ssh and has been modified to work with RedSSH.
        #
        # Copyright (C) 2014-2018 Panos Kittenis.
        #
        # This library is free software; you can redistribute it and/or
        # modify it under the terms of the GNU Lesser General Public
        # License as published by the Free Software Foundation, version 2.1.
        #
        # This library is distributed in the hope that it will be useful,
        # but WITHOUT ANY WARRANTY; without even the implied warranty of
        # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
        # Lesser General Public License for more details.
        #
        # You should have received a copy of the GNU Lesser General Public
        # License along with this library; if not, write to the Free Software
        # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
        out = func(*args, **kwargs)
        while out==LIBSSH2_ERROR_EAGAIN:
            self._block_select()
            out = func(*args, **kwargs)
        return(out)

[pkittenis]

Thanks, but that's too small to need attribution. There's similar code in examples which are without copyright.

So the tunnelling is using paramiko with ssh2-python to connect to remote host via the tunnel I suppose. That's not using ssh2-python for the tunnel so I don't see what example could be added here for that.

[Red-M]

I used Paramiko's structure for the select.select() so that it could be contained in a single thread instead of requiring two threads.

If you're fine with the other 3 functions that handle LIBSSH2_ERROR_EAGAIN for all my other calls to work, I can remove that statement from them.

    def _block_select(self, timeout=None):
        # This function was part of parallel-ssh and has been modified to work with RedSSH.
        #
        # Copyright (C) 2014-2018 Panos Kittenis.
        #
        # This library is free software; you can redistribute it and/or
        # modify it under the terms of the GNU Lesser General Public
        # License as published by the Free Software Foundation, version 2.1.
        #
        # This library is distributed in the hope that it will be useful,
        # but WITHOUT ANY WARRANTY; without even the implied warranty of
        # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
        # Lesser General Public License for more details.
        #
        # You should have received a copy of the GNU Lesser General Public
        # License along with this library; if not, write to the Free Software
        # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
        block_direction = self.session.block_directions()
        if block_direction==0:
            return()
        rfds = []
        wfds = []
        if block_direction & LIBSSH2_SESSION_BLOCK_INBOUND:
            rfds = [self.sock]
        if block_direction & LIBSSH2_SESSION_BLOCK_OUTBOUND:
            wfds = [self.sock]
        select.select(rfds,wfds,[],timeout)

    def _block(self, func, *args, **kwargs):
        # This function was part of parallel-ssh and has been modified to work with RedSSH.
        #
        # Copyright (C) 2014-2018 Panos Kittenis.
        #
        # This library is free software; you can redistribute it and/or
        # modify it under the terms of the GNU Lesser General Public
        # License as published by the Free Software Foundation, version 2.1.
        #
        # This library is distributed in the hope that it will be useful,
        # but WITHOUT ANY WARRANTY; without even the implied warranty of
        # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
        # Lesser General Public License for more details.
        #
        # You should have received a copy of the GNU Lesser General Public
        # License along with this library; if not, write to the Free Software
        # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
        out = func(*args, **kwargs)
        while out==LIBSSH2_ERROR_EAGAIN:
            self._block_select()
            out = func(*args, **kwargs)
        return(out)

    def _block_write(self, func, data, timeout=None):
        # This function was part of parallel-ssh and has been modified to work with RedSSH.
        #
        # Copyright (C) 2014-2018 Panos Kittenis.
        #
        # This library is free software; you can redistribute it and/or
        # modify it under the terms of the GNU Lesser General Public
        # License as published by the Free Software Foundation, version 2.1.
        #
        # This library is distributed in the hope that it will be useful,
        # but WITHOUT ANY WARRANTY; without even the implied warranty of
        # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
        # Lesser General Public License for more details.
        #
        # You should have received a copy of the GNU Lesser General Public
        # License along with this library; if not, write to the Free Software
        # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
        data_len = len(data)
        total_written = 0
        while total_written<data_len:
            (rc, bytes_written) = func(data[total_written:])
            total_written+=bytes_written
            if rc==LIBSSH2_ERROR_EAGAIN:
                self._block_select(timeout)

    def _read_iter(self, func, timeout=None):
        # This function was part of parallel-ssh and has been modified to work with RedSSH.
        #
        # Copyright (C) 2014-2018 Panos Kittenis.
        #
        # This library is free software; you can redistribute it and/or
        # modify it under the terms of the GNU Lesser General Public
        # License as published by the Free Software Foundation, version 2.1.
        #
        # This library is distributed in the hope that it will be useful,
        # but WITHOUT ANY WARRANTY; without even the implied warranty of
        # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
        # Lesser General Public License for more details.
        #
        # You should have received a copy of the GNU Lesser General Public
        # License along with this library; if not, write to the Free Software
        # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
        pos = 0
        remainder_len = 0
        remainder = b''
        (size, data) = func()
        while size==LIBSSH2_ERROR_EAGAIN or size>0:
            if size==LIBSSH2_ERROR_EAGAIN:
                self._block_select(timeout)
                (size, data) = func()
            if timeout is not None and size==LIBSSH2_ERROR_EAGAIN:
                raise(StopIteration)
            while size>0:
                while pos<size:
                    if remainder_len>0:
                        yield(remainder+data[pos:size])
                        remainder = b''
                        remainder_len = 0
                    else:
                        yield(data[pos:size])
                    pos = size
                (size, data) = func()
                pos = 0
        if remainder_len > 0:
            yield(remainder)

[pkittenis]

These are clearly derived from parallel-ssh, which makes that work a derivative. So usual copyright and license terms apply. If it was just the one function which is pretty general I wouldn't care.

License cannot be applied to blocks of code like that - if the project uses or is derived from licensed code it must be distributed in a compatible license as a whole. See above link for compatibility.

Any further discussion on this can be done on mail group - this issue is for tracking the forwarding example progress.

[Red-M]

Yeah, I use GPL2 but with a linux kernel style header so this should be fine.
I'm just unsure if I need to remove these notices all together.

[Red-M]

I've released a working implementation under one of my projects here:
https://github.com/Red-M/RedSSH/blob/master/redssh/tunneling.py#L75-L115

The only bug here is that it is nearly impossible to tell if the client has disconnected from the local socket so I've put a timeout on that to release the thread spawned from that.

[Red-M]

So I've recently got this working in my library rather well and it requires a lot of work to get this working correctly.

[pkittenis]

Local port forward is done in parallel-ssh using ssh2-python.