Cross platform filtering HTTP/S proxy based on .NET Standard 2.0.
This is the open source version of our filtering proxy. It is now in a feature freeze, and will only receive bug fix patches. Development continues privately at Technik Empire.
Cross platform filtering HTTP/S proxy based on .NET Standard 2.0.
License: Mozilla Public License 2.0
Cross platform filtering HTTP/S proxy based on .NET Standard 2.0.
This is the open source version of our filtering proxy. It is now in a feature freeze, and will only receive bug fix patches. Development continues privately at Technik Empire.
When we set the expires header, it already exists. Need to remove it first, then add it (when issuing a block action response).
Obviously there's an issue with the timezone settings when generating the timestamp and validity date(s) of the certs. Easiest solution is to make the dates retroactive. We could hunt down the exact time conversion issue but the former is a guaranteed solution.
Enable super bad protocols on the client side (SSL3 etc). As long as we keep them disabled on the upstream connections we're fine. We'd actually be doing clients a favour by transparently upgrading the connection (as the deprecated protocols get intercepted and handled locally, so never get to leave the client machine).
We don't have any content inspection callbacks available for websockets. We need to implement them, including the upcoming 3.0.x "stream" inspection.
The engine, via header forwarding, wrongly passes negotiated websocket session keys upstream, leading to 400-series responses (access denial and such).
Some standard-hating webapps send headers with HTTP verbs that are not legal, and C#/MS fights us on those, so we came up with a dirty work around. However, this is not accurate. We need to modify the header, if present, on the request.Content.Headers
property of the request if there is actual content, but we need to modify the header directly on the request.Headers
properties when no content has actually been sent.
Right now, we end up breaking these requests that are technically, by the book illegal.
The callbacks for directing the next action are mangled in such a way that:
Possible causes:
Performance drops can be observed on things such as google maps.
We need to add the basic constraints extension to flag the cert as a CA otherwise Firefox, the only browser in the universe that cares about this, will reject the certs we spoof.
Right now there's a rather low limit set on request body sizes. Remove it.
The HTTP and HTTPS endpoint public properties return the same inner private object. Not super critical, but still wrong.
We should have a callback where users can modify headers and rewrite URL's before actually sending the request outbound. Shouldn't be too hard, and would be really powerful for things like forcing safe search on and what have you.
We do not return
from the control block, and therefore block actions without requesting content inspections on responses will fail!
Get rid of m_x
type variables and replace with standard _x
for members.
This seems to be a different issue than #26 as a 502 is not returned. The server closes the connection without warning when connecting via onshape.com.
If I take the websocket URL and paste it into the websocket echo test, it connects fine.
@TechnikEmpire you can contact @montesound if you want to test the behavior.
These issues always arise from the way that .NET splits headers up into different, typed containers in HttpWebRequest/Response
but not in ASP.NET request and response objects.
Right now, whether a user has requested it or not, we fully load request bodies into memory before processing them. Instead, when a user wants to ignore it, we should simply wrap the input stream as StreamContent
so we write-through directly to the server, rather than buffering everything into memory first.
This is with CitadelCore 2.2.3
2018-06-01 09:58:12.8820 INFO Request https://billing.vispnet.ca/projects/piwi/index.php?NEWSESSION/ whitelisted by rule @@||billing.vispnet.ca in category /default/whitelist/rules.
2018-06-01 09:58:12.8820 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:15.8880 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:21.8890 INFO Connection from local:59476 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:23.5133 INFO Connection from local:59477 -> remote:80 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:33.8971 ERROR An error occurred while sending the request. at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__59.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CitadelCore.Net.Handlers.FilterHttpResponseHandler.d__6.MoveNext()Unable to connect to the remote server at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 66.244.214.7:443 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
From FilterHttpResponseHandler.cs::Handle() #263
2018-06-01 09:58:34.1942 INFO Request https://billing.vispnet.ca/favicon.ico whitelisted by rule @@||billing.vispnet.ca in category /default/whitelist/rules.
2018-06-01 09:58:34.1942 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:37.1967 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:43.1977 INFO Connection from local:59478 -> remote:443 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
2018-06-01 09:58:46.2524 INFO RESPONSE WindowsDiverter.cs::HandleNewTcpConnection() #678
2018-06-01 09:58:46.2524 INFO Connection from local:59357 -> remote:59479 outbound is our process. WindowsDiverter.cs::HandleNewTcpConnection() #616
I'm assuming that WsClient inherits from IDisposable, since Coverity has flagged it as a dangling resource/leak. It needs to be explicitly disposed if this is the case. Perhaps we can just wrap the whole thing in a using
statement, or we can do a try/finally
.
Kestrel 2.2.x will include HTTP/2 support, so watch for that and be ready to upgrade so we automatically inherit it.
Title says everything.
We install the root CA into the root X509Store for the local machine in order to establish OS trust. The question is, will this function the same on different platforms? If not, we just have to modify our abstract proxy server to request a platform specific implementation.
We should have the originating/receiving machine-local software readily available in the message callbacks.
We need a configuration class to avoid an ever-bloating constructor.
We pass false
to the leaveInnerStreamOpen
prorperty when constructing SSL streams around our peek streams (SNI capable streams). This causes some requests to randomly fail. It would appear that these are typically requests that have very small payload responses, so I'm not sure if the peek stream is getting closed because we're peeking to the end of the stream.
Anyway, changing leaveInnerStreamOpen
to true
seems to resolve it. If this problem persists, we need to replace our stream peeking mechanism. Looks like first class SNI support is in the pipeline for corefx, but I'm not gonna hold my breath for that. Whenever it does come though, we should use the built in SNI.
Title explains it all.
We should allow the ability to inspect streams as they pass through the proxy. This would involve adding an extra ProxyNextAction
that will allow you to stream-inspect.
A valid requirement/use case that this would address is where people want to inspect large streams such as streamed videos. You can't handle a 300+mb stream in a binary fashion, so you permit it to pass but get to look at the stream as it moves forward. In this case, the user would be empowered to be able pull arbitrary frames out of the video stream and inspect/classify the video content.
This shouldn't be hard. Add an enum, and wherever we use the stream copy utilities internally, we would instead manually loop the stream and then invoke an extra callback when our copy buffer fills up before sending it out. We would also have an out param in this callback that would enable the user to stop or modify the stream. (maybe we don't need modifiability in the callback initially).
In the case of decoding video frames. I've already had feedback from licensing people at MPEG-LA that as long as I leverage platform provided decoders, my use of patented codecs is covered because the provider (say, Microsoft and their media components) pays for patent use which includes the end user, and the software would fall under that "end user" clause. Don't use ffmpeg though, they specifically named it as an example where you are not covered and would thus be infringing and wow nothx.
Coverity complains about accessing a property (getter function) of a variable that is initialized as a null reference.
V6HttpEndpoint = httpListenOptions.IPEndPoint;
httpListenOptions
is declared as null
and gets defined later on based on IPV4/IPV6 settings. We should wrap this assignment in a null check.
On service point manager set max concurrent connections high and we get 10x throughput.
Our response stream automatically decompresses on the upstream. So when we are passing data back we need to both remove the content-encoding header and possibly adjust Content-Length. If we can assume that all connections are http 1.1 or later on the client side then we don't need to set a content length and can just purge that header, and allow kestrel to automatically chunk the response.
If not, that is if it's a http 1.0 connection, then we need to collect the whole response in a buffer, set the content length properly and then pipe out the response in a non-chunked fashion.
This is flagged as a bug because user action on a provided functionality should not error. Must change the object to swap the real stream for a ClosedStream
object just like the TLS SNI adapter does.
aspnet/WebSockets#19
https://github.com/dotnet/corefx/issues/15430
The autobahn test on CitadelCore only fails here but that's just because the whole thing is missing.
This website returns a 502 on current version of CitadelCore.
It tries to connect twice via websockets and then gracefully degrades to XHR.
wss://ws-eu.pusher.com/app/a677eaa2ad4202ba89fc?protocol=7&client=js&version=4.2.1&flash=false
To replicate after first load, run this in the javascript console
delete localStorage.pusherTransportEncrypted;
I fixed this in our own repo by adding Upgrade, Connection, and Sec-WebSocket-Version to ForbiddenWsHeaders as the implementation of websockets we're using adds those headers itself. This seemed to break whatsapp web, however.
Document offline coverity build process:
Microsoft's built in "cross-platform" websocket implementation doesn't work on Windows 7. Need to remove that plugin/addon and replace it with code written by someone competent.
We need a generic error handler callback. Ultimately we want users of the library to have complete control over the internet experience of their users. So, if SSL fails, for example, there should be a callback/handler just like the message begin/end functions where library users can handle it however they would like.
This should go for websocket handshake failures, parsing errors etc. Anywhere in the code where we abandon a connection due to an error, the user should get to handle this (optionally) in a callback.
Given that this handler could be called even for non-http messages (eg. Websockets), we should explore exposing a generic stream or something. Something to abstract away the protocol but give control over content still.
Problems accessing web.whatsapp.com
Right now we can only inject data and then drop the connection. We ideally would like to be able to inject and keep the connection alive.
The volley attempts to shut down the already-shut-down endpoint in both the client and server IO tasks.
We ignore zero-length request and response bodies right now. Unfortunately some web API's can complain about this and break some sites. We have to create zero length payloads in order to attach content related headers because Microsoft is super strict and won't let us attach content related headers when there is no content body itself.
We need to specify the domain and path of cookies. I'm not sure why we're not getting this info out of our http context object but anyway. We can default to "/"
for path and use the DNS host from the request string to populate those fields and this solves that problem at least for now.
HTTP2 is not universally complete in .NET. We should strip all headers and values that advertise it's available, as well as other protocols like QUIC.
We should also strip accept-encoding
headers and let the http client set those by itself.
We should also strip any google specific encoding headers, like their custom compression algos and such.
Some odd requests are giving 500 internal server error, which means the proxy has had an error while handling that request. However, when directly visiting the links of failed requests, there's no error. Perhaps something to do with referer maybe?
We can cut down on copies to all of our callbacks by using Memory. Don't use Span because Span can't survive continuations. On that note, we may want at least some of our current and even planned callbacks to be task based.
Need to review our forbidden headers list and ensure that we're stripping hop-by-hop headers we need to strip, not necessarily all of them, despite the proxy rule that we must.
Presently we just have the built in error handling middleware print the raw error to the response stream. This can be useful, but it'd be nice to be able to expose this to customization, dynamic especially.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.