Comments (7)
This will probably fix the error (see docs):
require 'typhoeus/adapters/faraday'
from her.
That did the trick for the error
from her.
I'm working on an app that communicates with two independent service APIs, and Her is a great ORM, so I've been interested in request parallelization too.
The ActiveRecord pattern in Her makes it more involved to add an asynchronous callback mechanism. The common solution is to return proxy objects for (collections of) model objects. They are then replaced by the mapped data from the response.
Actually, this is the pattern followed by the parallel support in Faraday (see their wiki). It's not fully asynchronous though, as the calling thread is blocked until the request group fully completes.
I'm interested in helping out with an implementation, but I'm relatively new to Ruby so any pointers are welcome.
For the time being, I tried to find a quick-and-dirty solution. If you add the Typhoeus adapter as middleware, all Faraday requests run through Hydra, which is I/O nonblocking:
my_api = Her::API.new.setup(host) do |connection|
connection.use Faraday::Adapter::Typhoeus
end
However, enclosing multiple Her requests in a Faraday in_parallel block is not possible, since Her expects an immediate response.
So I just run every request in its own thread:
f = Thread.new { @features = Feature.all }
a = Thread.new { @articles = Article.all }
f.join; a.join
This works fine, because Hydra is used (providing nonblocking I/O) and while one thread is blocked, it's perfectly possible to start a second request on a second thread.
Except for one important detail: Faraday requests use a shared Hydra instance, which is not thread-safe. So I monkey-patched the adapter to use Typhoeus::Request.run
, which spawns its own, per-request Hydra instance. Problem solved.
module Faraday
class Adapter
class Typhoeus < Faraday::Adapter
def perform_request(env)
read_body env
request(env).run
rescue Errno::ECONNREFUSED
raise Error::ConnectionFailed, $!
end
end
end
end
It's hardly an elegant solution, but it does the job. Runtime equals the longest request time. Note that my use case is a couple of service calls as part of a client Rails app controller action. YMMV if you're doing heavy fetching in a background task.
from her.
Except for one important detail: Faraday requests use a shared Hydra instance, which is not thread-safe. So I monkey-patched the adapter to use
Typhoeus::Request.run
, which spawns its own, per-request Hydra instance. Problem solved.
Which version of Typhoeus are you using? Since 0.5 Typhoeus::Request.run
doesn't use a hydra any more. Instead it really blocks now - so be careful.
from her.
@i0rek
Thanks for the heads up! I was indeed confused by an outdated RDoc+source. However, I'm on the latest version from RubyGems.org (Ethon 0.5.8, Typhoeus 0.5.4).
I see that Typhoeus::Request
is now using the Easy interface as opposed to the Multi one that Hydra uses. But since my request timings were fairly unambiguous, I experimented a bit further:
- Using a per-thread Hydra in the Faraday adapter didn't yield any difference in request times
- Removing any single request didn't affect the time of the other one
So finally, taking the aforementioned controller action and artificially increasing the response time of each service API by 5.0 seconds yields the following: (art
and sci
are the service APIs, front
is the client app)
16:59:17 front.1 | Started GET "/" ...
16:59:17 art.1 | Started GET "/consumer/articles" ...
16:59:22 art.1 | Completed 200 OK in 5098ms (Views: 90.9ms | ActiveRecord: 5.6ms)
16:59:22 front.1 | ETHON: performed EASY url=... total_time=5.130982
16:59:23 sci.1 | Started GET "/features" ...
16:59:23 sci.1 | Completed 200 OK in 5513ms (Views: 75.8ms | ActiveRecord: 8.7ms)
16:59:23 front.1 | ETHON: performed EASY url=... total_time=5.543162
16:59:23 front.1 | Completed 200 OK in 5661ms (Views: 100.8ms | ActiveRecord: 0.0ms)
To rule out any Rails logging inaccuracies, I also timed the page load manually, giving the same result.
There are no mentions of MULTI anywhere, yet the requests are clearly not blocking; otherwise the front
request would have taken at least 10 seconds.
So I guess this is a mystery. I'm not familiar with libcurl, but if you have any idea what could be causing this, that would be great to know.
from her.
Actually - I might have been wrong. Although Request#run
blocks the GIL
is released so that the other thread can run too.
from her.
I’m going to close this issue now since it hasn’t been updated for a few months.
from her.
Related Issues (20)
- after_find called twice HOT 1
- path_prefix option is not used when belongs_to association is set. HOT 5
- add ability to wrap "where" clause HOT 4
- Support for first, limit, group and order HOT 6
- Support for update_attributes missing? HOT 6
- Support for association extensions HOT 1
- undefined method `map' for nil:NilClass for association HOT 6
- Class-level response parsing? HOT 1
- Scopes on JsonApi model
- Find or Create by ID HOT 2
- Have we considered making this gem more SEO friendly? HOT 4
- Attributes don't play nicely with ActiveModel::Dirty in Rails 6
- Safe Navigation Operator &. don't works on nil objects
- I would like to have find_by! in order to raise an exception instead of a nil HOT 1
- HashWithIndifferentUsage is slow, and requires large ActiveSupport dependency
- ruby 2.7, rails 6.0 and faraday 1.0 HOT 3
- Release version 1.1.1 HOT 1
- her-rb.org website inaccessible. HOT 1
- Organizing a fork of her HOT 1
- Execute rescue_from plus her? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from her.