ngmoco / falcore Goto Github PK
View Code? Open in Web Editor NEWModular HTTP server framework for Go
License: MIT License
Modular HTTP server framework for Go
License: MIT License
While trying to unit test my own request filters, I realized it's very difficult as I can't create my own complete falcore.Request objects.
Is it possible to extract parameters with the router to pass to a pipeline stage ?
From a trace:
[TRAC] 1c7de31dfc server.Init S=0 Tot=0.0003 %=44.39
[TRAC] 1c7de31dfc *auth_filter.Filter S=0 Tot=0.0002 %=39.46
[TRAC] 1c7de31dfc *falcore.Pipeline S=0 Tot=-8143286784.0000 %=-1384912741466112.00
...
Hi guys,
This is more of a heads up, I've not quite got to the bottom of the problem yet.
After some tweaks to our core Go package RPM spec file, I was comfortable enough to re-enable building our Go repos for some other distros. A mime based file_filter test failed for the following distros:
https://build.opensuse.org/project/monitor?project=devel:languages:go
Given that these three distro's are clones, I'll bet there's some problem with mime type detection on RHEL 6. If I find more, I'll update.
Cheers the noo,
Graham
Was wondering if it is possible to use this in combination with the Gorilla web toolkit at http://www.gorillatoolkit.org/
Thanks!
Could you please be so kind to explain how to use the hot restart functionality?
When sending SIGHUP to the hot_restart
example binary, I receive an address already in use
error like so:
./hot_restart
47149 Received SIGHUP. forking.
Forking now with socket: 7
47149 Forked pid: 47220 errno: <nil>
47220 Got socket FD: 7
47220 Could not start server: listen tcp <nil>:8090: address already in use47220 Exiting now
this is on osx 10.8
thanks a lot!
Using this most basic falcore app:
package main
import (
"net/http"
"github.com/ngmoco/falcore"
)
func main() {
pipeline := falcore.NewPipeline()
pipeline.Upstream.PushBack(falcore.NewRequestFilter(func(req *falcore.Request)*http.Response {
return falcore.SimpleResponse(req.HttpRequest, 200, nil, "OK\n")
}))
server := falcore.NewServer(8081, pipeline)
server.ListenAndServe()
}
A simple request is very fast:
Daves-MacBook-Air-2:ka_test dgrijalva$ time curl http://localhost:8081/
OK
real 0m0.015s
user 0m0.008s
sys 0m0.005s
If you add a Connection:Keep-Alive
header, it always takes just over 5 seconds:
Daves-MacBook-Air-2:ka_test dgrijalva$ time curl http://localhost:8081/ -H"Connection:Keep-Alive"
OK
real 0m5.298s
user 0m0.005s
sys 0m0.003s
The Router stuff needs to be updated, clarified, and documented.
Original request:
I've been poring over the source for falcore for a while now and I can't work out how Routes work, and their relationship with pipelines.
How do you add a route to a server? Can a route select a pipeline? etc;
A good clear, but simple example would go a long way to helping newbs like myself understand how all this works.
P.S. While I'm at it, a good clear example of how the Proxy/Upstream stuff works would rule too :)
running go build server.go
in falcore/examples/basic_file_server
produces
server.go:7:2: import "github.com/fitstar/falcore/compression": cannot find package
server.go:8:2: import "github.com/fitstar/falcore/static_file": cannot find package
running go get
in the same directory produces
package github.com/fitstar/falcore/compression
imports github.com/fitstar/falcore/compression
imports github.com/fitstar/falcore/compression: import "github.com/fitstar/falcore/compression": cannot find package
package github.com/fitstar/falcore/compression
imports github.com/fitstar/falcore/static_file
imports github.com/fitstar/falcore/static_file
imports github.com/fitstar/falcore/static_file: import "github.com/fitstar/falcore/static_file": cannot find package
Captain we've been hit!
Tried to install.
go get github.com/ngmoco/falcore
And got:
# github.com/ngmoco/falcore
C:\Go\src\pkg\github.com\ngmoco\falcore\server.go:68: cannot use int(srv.listenerFile.Fd()) (type int) as type syscall.Handle in function
argument
Windows amd64 Go 1
Scott, I think you're right that we should move most of the responders to a subpackage. I propose we keep SimpleResponse, StringResponse, ByteResponse, and PipeResponse in falcore proper and move all others to a new package, responder
.
Thoughts?
README.md refer to a not valid url blog
rather than starting a new falcore server.
can I attach falcore to the started net/http server?
this way I can use falcore with google appengine.
thanks.
not trolling, but ab doc says:
-k
Enable the HTTP KeepAlive feature, i.e., perform multiple requests within one HTTP session. Default is no KeepAlive.
gleecology commented
More to the point, after the first conn, it fails (even whilst adding the Connection: header):
--debug--
ab -v 6 -n 3 -c 1 -k http://localhost:8000/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)...INFO: POST header ==
GET / HTTP/1.0
Connection: Keep-Alive
Host: localhost:8000
User-Agent: ApacheBench/2.3
Accept: /
LOG: header received:
HTTP/1.1 200 OK
Content-Length: 12
hello world!
LOG: Response code = 200
apr_poll: The timeout specified has expired (70007)
...To this ignorant layman, KA looks broken.
Apparently some of my recent issues arise from the smelly armpit of the HTTP spec:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3
I need to falcore to handle requests with this header but this type of request breaks the 1 request to 1 response paradigm that we have use for the pipeline design.
There are a few things we can do:
Created this issue to discuss the possibility/likelyhood of getting a litte bit of refactoring commited. See this branch for the commits: https://github.com/gnanderson/falcore/tree/refactor
Will send pull requests to Dave and Scotts tree's for review.
This started off by a series of benchmarks using weighttpd against one of my own server binaries, I noticed that using weighttp with keep alive enabled gave a very unrealistically good score, but then I noticed that only half the requests were successful so there was something up server side. I tracked it down to the main server loop and the 'Connection' header check. While RFC2616 states that header field names should be case insensitive, it's vague about the header values and the upper case Keep-Alive check in the server loop was failing on every second request using weighttp.
So no problem, easy fix, but when I changed the header check to match lower case I noticed performance fall through the floor for persistent keep alive connections. I tracked that down to a combination of TCP_CORK plus the tweaked TCP stack setting on the linux box. With a bit of fiddling and some hit and miss I was back to my usual throughput.
Having said that, what I really missed was being able to bench the falcore.Server directly in the test suite. I started off trying to add a benchmark for falcore.Server, and this was OK for simple hello world filters but to use the file_filter meant the cyclic dependancy of importing file_filter which imports falcore which imports file_filter... made this impossible.
So one thing led to another, breaking the cyclic dependancy meant having to refactor the package imports, and api calls/references, and then it meant also refactoring the filter tests to work standalone without an http server, otherwise I would have been back to square one.
Anyway, there's a few things to look at in the refactored branch... Let me know what you think. It breaks the package imports for server binaries and there's a few API backwards incompatibilities, such things as each filter had a "Filter" struct which i renamed to FileFilter, CompressionFilter and so on.
Cheers the noo,
Graham
I've noticed an intermittent failure of this test, here's a sample log from the latest failure.
[ 51s] $WORK/github.com/ngmoco/falcore/static_file/_test/static_file.test
[ 52s] --- FAIL: TestBasicFiles (0.02 seconds)
[ 52s] file_filter_test.go:144: custom mime type Error GETting file:unexpected EOF
I first saw the failure on the openSUSE Factory and openSUSE Tumbleweed builders. These are our unstable and rolling release platforms respectivly so after triggering a rebuild and not seeing a direct repeat of the failure I dismissed it as an issue with an unstable platform. Subsequently I have seen the issue maybe two or three time on the current stable release builders and now on an external distro build. Seeing the issue on an external distro prompted me to upstream it.
This intermittently appears on both 32 and 64 bit archs, and across at least the following platforms, but this should be considered with the knowledge that as in intermittent failure, the openSUSE platforms are the ones that get automatically triggered to rebuild more often which increases the likelyhood of occurence.
A first take suggests the error can only originate from the RoundTripper use but poking at the implemtation in the std libs:
http://golang.org/src/pkg/net/http/transport.go?s=3821:3892#L116
I see that there's the use of synch primatives, so this may be a subtle failure and possibly a bug in the test rather than falcore. Additionally we run our Go builds with -p 4 to match the allocated cores on the builders, so I'm not sure if that affects things.
The serve loop is currently very generous with clients that use keep-alive. This is, I think, a good thing, but it means the server can never gracefully shut down if there are long lived keep-alives.
I think the graceful shutdown procedure should be finish up any in flight requests and close connections as gracefully as possible. I can see a very clean approach to this by using a separate (permanent or temporary) go routine for reading. This will double the number of goroutines per connection, but it will also allow us to support http pipelining as we can separate read from write.
Any thoughts? Too much?
I'd like to propose we rename SimpleResponse to StringResponse. We can then add a BytesResponse with similar behavior. I'd also like to add a JSONResponse helper that takes an interface{} and returns (*http.Response, error). There's already a pull request for a RedirectResponse helper.
These are extremely common cases and we should support them. Renaming SimpleResponse might be painful, but it will be more clear. We could leave it as is for now, but it seems better to just go for it...
Thoughts?
This is basically a duplicate of issue #4. I would have re-opened it, but don't have permission.
C:\Users\Ralph\Temp>go version
go version go1
C:\Users\Ralph\Temp>go get github.com/ngmoco/falcore
# github.com/ngmoco/falcore
C:\Go\src\pkg\github.com\ngmoco\falcore\server.go:100: cannot use fd (type int) as type syscall.Handle in function argument
The upstream has a reference to a single *net.TCPConn which it uses to set the deadline just before calling transport.RoundTrip. The transport will actually open new connections to the same server up to MaxIdleConnsPerHost if the current connection is already in use. Whenever a new connection is opened it overwrites the single reference. When an old connection is reused later, it times-out since the deadline was set on the new connection instead of the older one being used.
I can't think of an easy way to fix this. Maybe wrap the connection with one that checks a channel. Send a flag to the channel in the same place as current deadline place and the first connection used will pop the channel and set the deadline only on the first write, otherwise no deadline is set. I don't think this method is "thread" safe though.
Thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.