contribsys/einhorn

Einhorn: the language-independent shared socket manager

einhorn's Issues

Any plans to publish v0.7.5 to rubygems.org?

Hi, do you have plans to publish v0.7.5 to rubygems.org?

Thanks.

When new releases will be available in rubygems.org?

I'm using Einhorn installed via gem install einhorn, but it seems new releases after 0.6.4 version are not yet available, do you know when they will be?

einhorn should support partial rollout & rollback

I haven't quite thought through how this feature should work, but something like "spin up one new worker process, let it serve requests for 30 mins, and then finish the upgrade" might be cool. As well, keeping around the old processes in a paused state could be useful for implementing rollback.

einhornsh is not script friendly

Right now, einhornsh is almost scriptable, but the resulting output leaves something to be desired:

evan@caron:~/stripe/einhorn$ echo reload | bin/einhornsh time-srv
Welcome evan! You are speaking to Einhorn Master Process 27289 (time-srv)
Enter 'help' if you're not sure what to do.

Type "quit" or "exit" to quit at any time
> Reloading, as commanded
> 
^C

In particular, it doesn't exit automatically if it gets an EOF from stdin, and it's pretty verbose.

It would be nice if einhornsh checked if stdin is a tty, and interact in a less verbose, more scripting friendly mode.

einhorn should monitor errors on upgrade

Ran into a problem when upgrading its children process to a new version, if the new children error out immediately (due to a bug). Because they error pre-ack, the old children stick around and handle requests (yay!)

However, einhorn continuously tries to start the new children. It should possibly back off, and almost certainly notify somehow. If I hadn't happened to be doing maintenance on the server in question, who knows how long we would have gone without noticing.

JRuby support

Hi to all!

Einhorn cannot run on jruby due to a lack of fork. Can i make a patch for that? I propose replace fork + exec by posix_spawn for MRI and spoon for jruby.

If einhorn will support jruby it can be proposed for puma instead its cluster mode :)

• posix_spawn – http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
• a fork/exec replacement for jruby – https://github.com/headius/spoon
• using spoon in foreman – https://github.com/ddollar/foreman/pull/140/files

When there are no children processes alive, swallows TERM

E.g.

[MASTER 21022] INFO: Sending TERM to []

And then hangs around for a bit.

Support binding UNIX sockets

I've an application that I'd like to run under Einhorn. It currently binds a UNIX socket and then an SSL-terminating nginx sitting on the same box talks to the service over that socket. This is a websocket service so it's designed to have tonnes of open sockets and using UNIX domain sockets for the nginx<->service bridge gives us a lot more headroom. Would it be possible for Einhorn to be able to bind UNIX sockets as well?

I'm happy to write the code for this (have taken a short crack at it already) but I'm a) not great at ruby, and b) would probably need some handholding on making sure the in-place upgrades are OK with the changes. If I do write it would this be something you'd want to merge?

[very low priority] The README says "Welcome gdb"

Since Einhorn is a former Stripe project it makes sense that it still has some Stripe-isms lurking around. The phrase Welcome gdb is one of them, since gdb is a former Stripe. Anyone reading that example in 2022 outside of Stripe, however, probably has no idea and is confused as to why a Ruby shell-like process is invoking a debugger.

how to keep einhorn running in background?

after calling einhorn -b 127.0.0.1:8000 -q -m manual /myapp
it blocks the shell. and i cant exit.
how to keep einhorn running on background
if i try to exit the einhorn will just die.

do i need to use any other program to accomplish this?

Race condition in einhorn?

Saw the following in logs when trying to kick off ~40 einhorns at the same time.

cc @zenazn

[2014-10-16 03:26:03.563102] [WORKER 19807] INFO: About to exec ["my", "command"]
[2014-10-16 03:26:03.564304] /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:163:in `write_nonblock': closed stream (IOError)
[2014-10-16 03:26:03.564331]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:163:in `break_loop'
[2014-10-16 03:26:03.564348]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:188:in `block in trap_async'
[2014-10-16 03:26:03.564364]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:10:in `call'
[2014-10-16 03:26:03.564867]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command/interface.rb:10:in `command_server='
[2014-10-16 03:26:03.564892]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event/command_server.rb:54:in `deregister!'
[2014-10-16 03:26:03.564908]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event/command_server.rb:44:in `close'
[2014-10-16 03:26:03.564924]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:28:in `block (2 levels) in close_all'
[2014-10-16 03:26:03.564937]    from /usr/lib/rbenv/versions/2.1/lib/ruby/2.1.0/set.rb:263:in `each_key'
[2014-10-16 03:26:03.564951]    from /usr/lib/rbenv/versions/2.1/lib/ruby/2.1.0/set.rb:263:in `each'
[2014-10-16 03:26:03.564964]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:27:in `block in close_all'
[2014-10-16 03:26:03.564976]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:26:in `each'
[2014-10-16 03:26:03.564989]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:26:in `close_all' 
[2014-10-16 03:26:03.565003]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/event.rb:34:in `close_all_for_worker'
[2014-10-16 03:26:03.565559]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:276:in `block in spinup'
[2014-10-16 03:26:03.565785]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:265:in `fork'   
[2014-10-16 03:26:03.565985]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:265:in `spinup' 
[2014-10-16 03:26:03.566199]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `block in replenish_immediately'
[2014-10-16 03:26:03.566399]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `times'  
[2014-10-16 03:26:03.566594]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:468:in `replenish_immediately'
[2014-10-16 03:26:03.566790]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn/command.rb:455:in `replenish'
[2014-10-16 03:26:03.566999]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:469:in `run'
[2014-10-16 03:26:03.567200]    from /stuff/vendor/bundle/ruby/2.1.0/gems/einhorn-0.6.3/bin/einhorn:327:in `<top (required)>'  
[2014-10-16 03:26:03.567399]    from /stuff/vendor/bundle/ruby/2.1.0/bin/einhorn:23:in `load'
[2014-10-16 03:26:03.567590]    from /stuff/vendor/bundle/ruby/2.1.0/bin/einhorn:23:in `<main>'

Is there a way to "inject" environment variables when upgrading?

When doing an upgrade using einhornsh -e upgrade is there a way to inject enviroment variables?

I'm using workers in Go, and using the os.Getenv() function without success after upgrading them using the einhornsh command. And since Einhorn has already started cannot use it to pass enviroment data to its children.

License missing from gemspec

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Even for projects that already specify a license, including a license in your gemspec is a good practice, since it is easily
discoverable there without having to check the readme or for a license file. For example, it is the field that rubygems.org uses to display a gem's license.

For example, there is a License Finder gem to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough
issue that even Bundler now generates gems with a default 'MIT' license.

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), github has created a license picker tool.

In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :).

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know. In either case, I'll follow up. Thanks!

p.s. I've written a blog post about this project

Maybe make YAML parsing safer

Need to make sure this doesn't interfere with preloaded code though.

Integrate Github actions CI workflow

We need an active CI which can test old Rubies for support purposes. See the Sidekiq repo and steal its code as necessary.

einhorn is vulnerable to environment changes by preloaded code

Classifying this as a bug, which maybe isn't strictly accurate, but it definitely impacted our use of einhorn.

We had an issue today using einhorn's preloading where the preloaded code would modify environment variables in a way that changed einhorn's behavior when it re-exec'd itself on upgrade (specifically, changing RUBYOPT and BUNDLE_GEMFILE).

I think the right thing for einhorn to do in this situation is to save its environment at startup as part of its state and then restore that environment just before it re-exec's itself.

Prints "Could not find any config for exited child..." upon upgrade

I now see this pretty consistently:

[MASTER 34636] ERROR: Could not find any config for exited child 34672! This probably indicates a bug in Einhorn.

Looks like it was introduced in: 673f5a8. Sounds like we had a race before, but I think this means we have a race still.

@ebroder, what would you think about de-racing by making the state_passer live past the SIGCHLD handler install? I'd guess that would mean either (a) installing the SIGCHLD handle before loading state — though I haven't checked if there's a chicken and an egg there — or (b) having some coordination mechanism between the state_passer and the master. Thoughts?

Socket reuse flag not working?

Hi - I'm running einhorn via upstart and I'm seeing the following exception if there is a stop/start sequence in rapid succeession:

Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Writing PID to /srv/www/channel_api/shared/pids/einhorn.pid
Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Binding to 0.0.0.0:8080 with flags ["rn"]
Mar 12 19:13:13 doughnut goliath-channel_api: [MASTER 29092] INFO: Sending USR2 to []
Mar 12 19:13:13 doughnut goliath-channel_api: /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:205:in `bind': Address already in use - bind(2) for 0.0.0.0:8080 (Errno::EADDRINUSE)
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:205:in `bind'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:356:in `block in socketify_env!'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:355:in `each'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:355:in `socketify_env!'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/lib/einhorn.rb:449:in `run'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/gems/einhorn-0.6.3/bin/einhorn:327:in `<top (required)>'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/bin/einhorn:23:in `load'
Mar 12 19:13:13 doughnut goliath-channel_api: #011from /home/deploy/.bundler/channel_api/ruby/2.1.0/bin/einhorn:23:in `<main>'

When the error is reported, there are no running einhorn processes but netstat -na | grep 8080 reports a few sockets in TIME_WAIT. I thought the r flag should let einhorn bind despite that?

Typo in repo description

"Einhorn: the lanuage-indepedent shared socket manager": "lanuage" should probably be "language".

Einhorn with Goliath

Hello

I do not know how to achieve a seamless restart Goliath.
Please give me some advice, thank you!

einhorn -n 2 -b 127.0.0.1:3000 ruby app.rb --einhorn
[MASTER 16379] INFO: Writing PID to /var/folders/nc/2jhfmrs55fz7qkqddw5cvtkw0000gn/T/einhorn.pid
[MASTER 16379] INFO: Binding to 127.0.0.1:3000 with flags []
[MASTER 16379] INFO: Launching 1 new workers
[MASTER 16379] INFO: ===> Launched 16380 (index: 0)
[WORKER 16380] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "--einhorn"]
[MASTER 16379] INFO: Worker 16380 has been up for 1s, so we are considering it alive.
[MASTER 16379] INFO: Up to 1 / 1 timer ACKs
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/runner.rb:107:in `initialize': invalid option: --einhorn (OptionParser::InvalidOption)
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `new'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `run!'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:133:in `block in <module:Goliath>'
[MASTER 16379] INFO: ===> Exited worker 16380
[MASTER 16379] INFO: ===> Launched 16382 (index: 0)
[WORKER 16382] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "--einhorn"]
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/runner.rb:107:in `initialize': invalid option: --einhorn (OptionParser::InvalidOption)
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `new'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:107:in `run!'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/application.rb:133:in `block in <module:Goliath>'
[MASTER 16379] INFO: ===> Exited worker 16382 before it was ACKed

And

einhorn -n 2 -b 127.0.0.1:3000 ruby app.rb -sv
[MASTER 16434] INFO: Writing PID to /var/folders/nc/2jhfmrs55fz7qkqddw5cvtkw0000gn/T/einhorn.pid
[MASTER 16434] INFO: Binding to 127.0.0.1:3000 with flags []
[MASTER 16434] INFO: Launching 2 new workers
[MASTER 16434] INFO: ===> Launched 16435 (index: 0)
[WORKER 16435] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "-sv"]
[MASTER 16434] INFO: ===> Launched 16436 (index: 1)
[WORKER 16436] INFO: About to exec ["/Users/tumayun/.rvm/rubies/ruby-2.2.3/bin/ruby", "app.rb", "-sv"]
[16435:INFO] 2015-11-30 23:27:34 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
[16436:INFO] 2015-11-30 23:27:34 :: Starting server on 0.0.0.0:9000 in development mode. Watch out for stones.
/Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/eventmachine-4de7d0e00fe8/lib/eventmachine.rb:526:in `start_tcp_server': no acceptor (port is in use or requires root privileges) (RuntimeError)
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/eventmachine-4de7d0e00fe8/lib/eventmachine.rb:526:in `start_server'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/bundler/gems/goliath-5d0b54543705/lib/goliath/server.rb:86:in `block in start'
    from /Users/tumayun/.rvm/gems/ruby-2.2.3/gems/em-synchrony-1.0.4/lib/em-synchrony.rb:38:in `block (2 levels) in synchrony'
[MASTER 16434] INFO: ===> Exited worker 16436 before it was ACKed
[MASTER 16434] INFO: Worker 16435 has been up for 1s, so we are considering it alive.
[MASTER 16434] INFO: Up to 1 / 2 timer ACKs, breaking the streak of 1 consecutive unacked workers dying
^C[MASTER 16434] INFO: ===> Exited worker 16435

Integrate standardrb

• Integrate standardrb
• standard:fix the codebase to use the standard code format

See the Sidekiq repo for an example project which is already done.

Customize graceful shutdown signal

I'd like to use einhorn to do seamless reloading of some processes but I'm using various programs that expect something different than a SIGUSR2 for a graceful shutdown.

It would be nice if the graceful shutdown signal could be specified on the command line.

Thoughts?

`config` command does not work properly

config seems to be a no-op, at least for the max_unacked and max_upgrade_additional settings

To reproduce:
sudo einhornsh -c $SERVICE -e state
sudo einhornsh -c $SERVICE -e 'config max_unacked: 10 (where 10 is any new value)
sudo einhornsh -c $SERVICE -e state (and see that nothing changed)

@evan-stripe pointed out that it looks like config is merging changed values into the wrong place

einhorn should watch for stuck processes

einhorn should notice if it signals a process to die and that process doesn't done so.

Or at least it should give you enough hooks to monitor it yourself (a la #2). I'd settle for that.

Support passing shared socket fd numbers in environment variables

Unrecognized fds should be set to cloexec

This can specifically be an issue with code preloading, which can, e.g., establish database connections and such.

Einhorn should nice itself, but not its children

Right now reloads have the potential to eat up a lot of CPU. I believe that's due to loading code. If so, we should probably have Einhorn nice itself, and renice its children back to normal.

Crash during upgrade

Don't understand exactly what happened here, but einhorn crashed with this exception a while back and has been breakage looping since (on account of workers being left around):

[2013-02-15 02:29:11.596843] /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:9:in `block in unsignaled_workers': undefined method `length' for nil:NilClass (NoMethodError)
[2013-02-15 02:29:11.596882]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:8:in `select'
[2013-02-15 02:29:11.596915]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:8:in `unsignaled_workers'
[2013-02-15 02:29:11.596949]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/worker_pool.rb:57:in `old_workers'
[2013-02-15 02:29:11.596981]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn/command.rb:309:in `cull'
[2013-02-15 02:29:11.597014]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/lib/einhorn.rb:283:in `run'
[2013-02-15 02:29:11.597046]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/gems/einhorn-0.4.2/bin/einhorn:294:in `<top (required)>'
[2013-02-15 02:29:11.597077]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/bin/einhorn:23:in `load'
[2013-02-15 02:29:11.597109]    from /deploy/abba/current/vendor/bundle/ruby/1.9.1/bin/einhorn:23:in `<main>'

@gdb Does that make sense to you? Is this something like a bug where we received a signal before we had the handlers in place for it?

YAML issue with Ruby 3.1

See sidekiq/sidekiq#5400

Is this project abandoned?

Just curious.