Giter Site home page Giter Site logo

hivesolutions / netius Goto Github PK

View Code? Open in Web Editor NEW
119.0 10.0 5.0 5.23 MB

Readable, simple and fast asynchronous non-blocking network apps

Home Page: http://netius.hive.pt

License: Apache License 2.0

Python 100.00%
net library wsgi http http2 asyncio

netius's Introduction

Netius

Fast and readable async non-blocking network apps

Netius is a Python network library that can be used for the rapid creation of asynchronous, non-blocking servers and clients. It has no dependencies, it's cross-platform, and brings some sample netius-powered servers out of the box, namely a production-ready WSGI server.

Simplicity and performance are the main drivers of this project. The codebase adheres to rigorous code standards and is extensively commented on. As far as performance is concerned, it aims to be up to par with equivalent native implementations, where PyPy can provide the extra boost to raise performance up to these standards.

Bear in mind that although netius is non-blocking, it will naturally still block if the operations performed within the event loop are blocking, like reading or writing a file, which are both blocking operations in the Python standard library. Running multiple netius instances in parallel, and having a fast server like NGINX acts as their reverse proxy, which is one way of minimizing the perceptibility of such blockages.

Installation

pip install netius

Or download the source from GitHub.

Netius has no dependencies and is, therefore cross-platform. It's compatible with PyPy, with which its benefits of performance increase up to 1.5x - 2.5x faster in most environments when compared with running it with the CPython interpreter.

Usage

WSGI Server

import netius.servers

def app(environ, start_response):
    status = "200 OK"
    contents = "Hello World"
    content_l = len(contents)
    headers = (
        ("Content-Length", content_l),
        ("Content-Type", "text/plain"),
        ("Connection", "keep-alive")
    )
    start_response(status, headers)
    yield contents

server = netius.servers.WSGIServer(app = app)
server.serve(port = 8080)

HTTP Client

Synchronous usage

import netius.clients
result = netius.clients.HTTPClient.get_s(
    "http://www.flickr.com/",
    asynchronous = False
)
print(result["data"])

Asynchronous usage

import netius.clients

def on_partial(client, parser, data):
    print(data)

def on_message(client, parser, message):
    netius.clients.HTTPClient.cleanup_s()

netius.clients.HTTPClient.get_s(
    "http://www.flickr.com/",
    callback = on_message,
    on_data = on_partial
)

Test servers

The servers that come with netius out-of-the-box, can be tested through the command line:

Class Example
WSGIServer python -m netius.servers.wsgi
FTPServer python -m netius.servers.ftp
HelloServer MESSAGE="Hello Netius" python -m netius.extra.hello
FileServer BASE_PATH=/ python -m netius.extra.file
SMTPServer python -m netius.servers.smtp
RelaySMTPServer python -m netius.extra.smtp_r

Learn more

Basic

Advanced topics

More information can be found in the Advanced Topics page.

License

Netius is currently licensed under the Apache License, Version 2.0.

Build Automation

Build Status Build Status GitHub Coverage Status PyPi Status License

netius's People

Contributors

hugo-gomes avatar joamag avatar timgates42 avatar tsilva avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

netius's Issues

HTTP3 and QUIC ideas

Rationale

The world of HTTP is now constantly migrating and although the evolution from HTTP 1.1 to 2.0 took a lot of time the migration from 2.0 to 3.0 was quite fast.

Netius should follow this migration path and add support for HTTP 3.0 including the implementation of the QUIC transport protocol on top of UDP.

References

HTTP2 stream priority support

Description

While the base HTTP2 server implementation for Netius is developed, a proper support for priorities is still not developed leaving the output channel "unprotected" against large unimportant chunks of data that fill the space of the more important HTML, Javascripts, etc chunks.

Current HTTP2 performance under real browser scenarios is limited by the lack if priority support and the total loading times are often larger than the HTTP1.1 equivalents.

Notes

The delaying of the dependent streams should be performed at two different levels:

  • CPU/RAM, meaning that the on_data should not be triggered while the stream is not ready
  • Bandwidth, meaning that no bytes are sent back to the client while the parent streams are not completed or blocked, a delay operation is done on all of the data frames of the stream until the parent frame unblock its task, this may be done by using part of the available_stream/delay_frame implementation

Note that the most important part point is the one about blocking the bandwidth, as that's the most limited resource

Benchmarking

The page loading performance of the Netius HTTP2 server should be equivalent to the one offered by other HTTP2 servers like nghttp2 for that proper comparison must be created.

References

Protocols support

Description

Since the inception of the asyncio library in Python 3.5, the Python network world has changed, and a lot of "scalability" emerged from that.

As Netius was created before the arrival of asyncio, the tech approach to network diverged and although overall both strategies represent nonblocking asyncio event-loop-based solutions, they have different approaches and abstractions for server and client implementation.

The objective of this issue is to merge (via compatibility layer) both approaches, making it possible for the following scenarios:

  • A netius HTTP should be able to run on the asyncio event loop
  • An asyncio service should be able to run in the netius event loop

Meaning full bi-direction compatibility between Netius way of working and the asyncio one.

Implementation

Architecture

  • Transport - Connection (Stream) vs Connectionless (Datagram)
  • Protocol - Equivalent to part of the current Connection and also part of the current Server/Client (protocol logic)
  • Server - Equivalent to the newly created Service that spawns multiple connections (that should be converted into transports)
  • Parsers - Already exists and should remain pretty much the same
  • Event Loop - Must be improved to distance itself from the server vs client approach, meaning that an event loop is something simpler that only handles the polling strategy adapting details
  • Servers/Clients (Agents) - Must be simple Abstract API entry-points that facilitate some operations (no logic), this abstract logic should be re-use as much as possible

Change ideas

Command-based client

Must rethink very well what's a command-based client (eg: SMTP, IMAP, etc.) as their session is currently very hard-coded, it must be much more similar to command-based. Kind of like a bot-based server (more similar to the FTP server). And their state machine should be more flexible.

Migration strategy

This is the overall idea of how to migrate a client

HTTPConnection -> HTTPProtocol

Principles

  • Retro-compatibility should always be a priority
  • Concepts like auto_close and auto_pause no longer make sense
  • Bi-directional compatibility between netius and asyncio
    • Protocols developed for asyncio should be able to run without problems in netius
    • Netius server implementations should be able to run without problem in the base asyncio event loop

Testing

To test that both strategies are working one can use the ASYNCIO=1 env variable to run in the asyncio event loop. And the COMPAT=1 mode will ensure that the services and clients of Netius will run the compatibility layer meaning that they should be able to run with the asyncio API behavior expectations, calling the proper protocol callbacks etc.

Reference

Transports and protocols (callback-based API)
aiohttp server

Get content of uploaded file in easier way than by slicing 'wsgi.input'

Hi!

At first I want to thank you for great working library, which is easy to use and play nice with use cases in our projects.
When file is sent to server (using Cordova File Transfer plugin) from our hybrid app, content of environment['wsgi.input'].getvalue() is:

--+++++
Content-Disposition: form-data; name="file"; filename="image.jpg"
Content-Type: application/octet-stream

(Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
--+++++--

And when I need to get text content of uploaded file, I have to manually slice environment['wsgi.input'] to get content (to get dispose of boundaries, Content info etc..):

uploadedFileContent = inputWithFile.getvalue()
uploadedFile=StringIO(uploadedFileContent[uploadedFileContent.index(START_OF_FILE_CONTENT):len(uploadedFileContent)])
uploadedFile=StringIO(uploadedFile.getvalue([0:uploadedFile.getvalue().index(BEGINNING_OF_ADDITIONAL_INFO_AT_END_OF_FILE)])
content = str(uploadedFile.getvalue())

Is there a way in your library to get content of uploaded file out of the box, without the need to slice environment['wsgi.input'] maually?

Thank you in advance for reply,
Radek from Techmetria.

HTTP Client timeout

Description

It's important to implement a proper way of "timeouting" an HTTP request so that the control flow is retuned to the caller on an error callback (just like normal HTTP error eg: 500).

This should be a regression from the major event loop refactor taken some weeks ago.

Failover on proxy server

Description

It's important to be able to handle a fallback process when a back-end server is not available.
Our current proxy solutions does not handle the unavailability of the back-end server on a proper way.

Solution

Proper solution would imply

  • Constant "ping" of the back-end server
  • Marking the back-end server as not available
  • Not redirecting request to the unavailable services

Gzip enconding in HTTP Proxy

Description

It's important to be able to re-encode an HTTP based data stream either with chunked encoding or gzip encoding.
This away it's possible to optimize a proxy connection reducing latency.

Performance

This change in code should be done without affecting any of the current structures/workflow of the netius proxy implementation.

HTML parser

Description

Netius should include an HTML/XML parser so that it's possible to create a web crawler if that's desired. The HTML parser would be really important and relevant.

Notice that having a perfect HTML parser is not a priority as such parser would be very complex to implement.

Performance

The parsing performance of the parser is relevant and the structure of the parser should be on track with the HTTP parser in terms of structure (event driven).

Support for server side SSL SNI (Service Name Indication)

Description

Sometimes it's important to have a virtual hosting based serving with a different SSL certificate per each host (proxy usage). For this situations the solution is to user SNI extension to SSL that provides a way to handle this scenarios. SNI is available for Python since 3.3 but only 3.4 handles server side SNI correctly.

Implementation

The corre implementation implies the compatability of netius with 3.4 and the usage of the ssl.SSLContext.set_servername_callback method.

Reference

http://en.wikipedia.org/wiki/Server_Name_Indication
https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_servername_callback

Python type stub files ๐Ÿ“œ

Description

With Python 3.6+, there's been primitive support for type hinting.
We want to be able to offer this richer type of support through the usage of type hinting.

Implementation

Make use of type stubs to add types to the major entry point functions/methods.

There's already some work done d0cc0f4.

HTTP Forward proxy is woking too slow on Edge and Firefox

Description

For some reason the Forward HTTP proxy is working too slowly under Microsoft Edge and Firefox browsers. It seems as if the connections are getting into a stalled state for some reason.

Note that this only happens for HTTP based connections and not the ones with CONNECT.

Better DHT client

Description

The current DHT implementation lacks true support for bootstrap and better peer discovery and mapping. This must be changed with a new implementation.

The current partial implementation is located here.

Related

Issue #8

Debugger client/server

Description

The current netius infra-structure does not have any kind of runtime based debugging capabilities that are able to discover complex runtime CPU usage or RAM consuming.

The new infra-structure (netiusd) should be able to print/debug diagnostics information on the netius infra-structure on demand and using a command line based (interactive) approach.

The impact (performance) on the host server should be minimal to zero.

The debugger should be able to stop the loop and then resume it (operations).

Solution

On the host/server side a flag should be set so that the proper Appier HTTP service is loaded and is able to provide a client with diagnostics information. The service should include a conditional constant stream of information (http chunked) that refreshes on each tick of the main service loop.

On the client side a simple interactive console application should be created using the Appier HTTP client infra-structure.

The configuration values that are related with the diagnostics service (default values) are:

DIAG=0
DIAG_PORT=5050
DIAG_HOST=127.0.0.1

In case the diagnostics service is requested (DIAG=1) and it fails to load an INFO message should be printed but the system should keep running (failing gracefully).

Estimation

2 days

HTTP/2 support

Description

HTTP/1.1 is slow and old and a new standard must be implemented.
It's going to be challenging mostly because of the required implementation of the headers compression HPACK (RFC7541).

Notice that there's already a lot of browsers supporting this specification as seen here.

Inspiration

Currently the best inspiration is to use the nghttp2 library that provides support fot Python through its binding libraries.

There's also the node-http2 (node.js implementation) quite interesting for its sinplicity.

Estimation

2-5 days

References

Attachements

rfc7540.pdf

More Documentation

I just came across this project and it looks promising :)

But I'm missing the most basic instructions about some of the features, like how I enable HTTP2 support and create non-blocking HTTP servers.

Also, I guess there is a pre-fork mode. How does this handle load balancing between children if they each are handling multiple requests?

IPv6 Support

Description

Currently netius does not offer IPv6 support which is going to be a major drawback in the near future, taking into account the migration process.

Implementation

Adding this kind of functionality should not be a major problem for the netius base system.

TFTP Server support

Description

TFTP is still used under a lot of scenarios to allow simple transfer of files. One of the most well known scenarios is the PXE boot infra-structure.

For that a simple server under the netius infra-structure must be created.

Links

Wikipedia
RFC 1350
Client for Windows
PXELINUX

Syslog client/server that implements the standard syslog protocol

Description

It's important to start the implementation of a remote logging mechanism that allows servers and client to log their activity in a distributed way. There are some global standards syslog that already define logging rules.

Authentication/security should be considered one of the main priorities in the implementation of these kind of servers.

Implementation

One of the major problems of the current syslog protocol is the missing authorization/authentication and for that an extension may have to be created.

Another feature that may have to be implemented is the options to serialize the requests as JSON messages for simplicity.

Examples

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed

For a full client implementation in python go here.

Here are some example filters for /etc/rsyslog.conf:

:msg, regex, ".*failed.*" /var/log/failed.log
:syslogtag, isequal, "su" /var/log/su.log

References

Wikipedia
RFC 3164
RFC 5424

SSDP Client

Description

In order to be able to control a router and map/open an outside port into a local device an SSDP/UPnP client must be implemented and used.

This client may be used by the Torrent server for automatic port opening and mapping.

Partial implementation is already created under https://github.com/hivesolutions/netius/blob/master/src/netius/clients/ssdp.py.

Related

Issue #9

References

Netius HTTP2 proxy server is not working correctly under iPad Chrome

Description

While using lugardajoia.com website and HTTP2 under Chrome for IPad some resources fail to load for an unknown reason.
This may be related with #13.

Notes

This issue has yet to be verified under other iOS devices like iPhone.
Ths issue does not occur under Safari for iOS.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.