Errno::ECONNREFUSED (Connection refused - connect(2)) after OS X Java update

[sunspot]

Errno::ECONNREFUSED (Connection refused - connect(2)) after OS X Java update

[alindeman]

Any useful output if you run rake sunspot:solr:run (foreground)?

[alindeman]

Actually, did you recently do a bundle update against the sunspot gems? @nz, this might be the broken rake task issue I'm talking about where within Rails, config/sunspot.yml is not respected.

[cseydel]

Hi alindeman,
I didn't run a bundle update successfully since we are still using Rails 3.0.10 which depends on builder 2.1.2 and the newest sunspot_rails depends on builder =3.0.0.

Bundler could not find compatible versions for gem "builder":
  In Gemfile:
    sunspot_rails depends on
      builder (= 3.0.0)

    rails (= 3.0.10) depends on
      builder (2.1.2)

so we are using sunspot versions:

Using sunspot_rails (1.3.0.rc6) from https://github.com/sunspot/sunspot.git (at master) 
Using sunspot_solr (1.3.0.rc6) from https://github.com/sunspot/sunspot (at master) 

I copied the raketask output to a gist:
https://gist.github.com/1374599

Thanks for your help!

[alindeman]

Thanks. A couple things are going on here.

1. The recent rsolr release added the builder dependency that's breaking Rails 3.0.10. I've filed a bug here: rsolr/rsolr#30. Sunspot may revert to depending on rsolr 1.0.2 if that isn't fixed soon.

2. The rake tasks in Rails are broken as of f9ccfc2 in sunspot/sunspot_solr. @nz is working on a fix for this.

Could you try replacing your sunspot_solr line in Gemfile with?

gem 'sunspot_solr', :git => 'https://github.com/sunspot/sunspot', :ref => 'ada19e5'

And running:

bundle update sunspot_solr

[cseydel]

I switched to

Using sunspot_solr (1.3.0.rc5) from https://github.com/sunspot/sunspot (at ada19e5)

but now I can't start the server with following trace: https://gist.github.com/1374673

[cseydel]

Ok, I got it running with 1.3.0.rc5
Point was, that all sunspot gems need the reference to the commit hash.

gem 'sunspot_solr', :git => 'https://github.com/sunspot/sunspot', :ref => '79175ea'

Using sunspot (1.3.0.rc5) from https://github.com/sunspot/sunspot (at 79175ea) 
Using sunspot_rails (1.3.0.rc5) from https://github.com/sunspot/sunspot.git (at 79175ea) 
Using sunspot_solr (1.3.0.rc5) from https://github.com/sunspot/sunspot (at 79175ea)

We got two projects running with sunspot_solr and both were broken after our Java update. Two other colleagues here in house with same setup but without Java update never had problems.

Neverthenless this strange behaviour appears first after both of us updating Java on our Mac. I am pretty sure for myself that I didn't run a bundle update.
I appreciated your help and maybe you should get an eye on this issue.

Kind regards,
Christoph

[alindeman]

Hmm, sorry I'm a bit confused now: is this still an issue or not?

[cseydel]

This is still an issue in rc6 but I reverted to rc5 and it works for me.

[alindeman]

What about the 1.3.0 release (just released over the weekend)? Thanks!

[cseydel]

Thanks, 1.3.0 fixes the issue and solr is working fine.

[alindeman]

Awesome! Great to hear.

[gunn]

Getting the same (or similar) problem on 1.3.0:

bundle exec rake sunspot:solr:run doesn't error, but sunspot doesn't seem able to communicate with solr.

Running bundle exec rake sunspot:solr:reindex --trace, I get:

Connection refused - connect(2)
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `initialize'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `open'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/timeout.rb:89:in `timeout'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `connect'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:626:in `start'
/Users/arthur/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:1168:in `request'
/Users/arthur/.rvm/gems/ruby-1.9.2-p290/gems/rsolr-1.0.6/lib/rsolr/connection.rb:15:in `execute'
...

From within the web app I get:

RSolr::Error::Http - 500 Internal Server Error
Error: lazy loading error

org.apache.solr.common.SolrException: lazy loading error
...

[alindeman]

Can you post any output that appears at http://localhost:8981/solr/admin when starting the server in development mode?

[gunn]

@alindeman I take it you mean port 8982 in development mode?
I get what appears to be the full functioning, yellow on grey, admin interface:

Solr Admin (sunspot)
192.168.2.1:8982
cwd=/Users/arthur/.rvm/gems/ruby-1.9.2-p290/gems/sunspot_solr-1.3.0/solr SolrHome=/Users/arthur/Work/git/search_site/solr/ 
Solr
[SCHEMA] [CONFIG] [ANALYSIS] [SCHEMA BROWSER] 
[STATISTICS] [INFO] [DISTRIBUTION] [PING] [LOGGING]
App server:
[JAVA PROPERTIES] [THREAD DUMP]
...

[gunn]

Hmm, I suspect that this is related to the sunspot_cell plugin. I'll let you know what I discover.

[thegrubbsian]

I'm also having this issue on 1.3.0. Is there a safe version to roll back to?

[nz]

Better to figure out what's going on — what do your Solr logs say? Have you
tried removing your solr/data directory in case of index corruption?

[bglusman]

FWIW, I think I'm having the same or similar problem also, and also not fixed by 1.3.0, nor by using the commit hashes referenced above.

My solr log has this error:
INFO: JNDI not configured for solr (NoInitialContextEx)

But I'm not seeing that error generated when the Errno::ECONNREFUSED: Connection refused - connect(2) error occurs, so that may have been from before I properly started the solr instance or something... it was in the rails root log/sunspot-solr-development.log but removing that (and the .lck) it didn't regenerate any errors on connection refused, but I can still connect to the web admin at http://localhost:8982/solr/admin/

[pferrel]

I'm having what looks like the same problem.

There is no error on startup of rails or solr except the "INFO" level message in sunspot_solr_development.log:

INFO: JNDI not configured for solr (NoInitialContextEx)

My solr admin is running fine on 8982 as expected, but any try to communicate with solr through sunspot gets a "connection refused". This happens when I create a blank search object or when I "sunspot:reindex". The following is reported when doing a reindex:

Maclaurin:FinderBots.com pferrel$ rake sunspot:reindex --trace
** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
Re-indexing AmpedPage
rake aborted!
Connection refused - connect(2)
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:762:in initialize' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:762:inopen'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:762:in block in connect' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:54:intimeout'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/timeout.rb:99:in timeout' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:762:inconnect'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:755:in do_start' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:744:instart'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1284:in request' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rsolr-1.0.6/lib/rsolr/connection.rb:15:inexecute'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rsolr-1.0.6/lib/rsolr/client.rb:163:in execute' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rsolr-1.0.6/lib/rsolr/client.rb:158:insend_and_receive'
(eval):2:in post' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rsolr-1.0.6/lib/rsolr/client.rb:67:inupdate'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/rsolr-1.0.6/lib/rsolr/client.rb:131:in delete_by_query' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/indexer.rb:55:inremove_all'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:in block in remove_all' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:ineach'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:in remove_all' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot.rb:464:inremove_all'
/usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot_rails-1.3.0/lib/sunspot/rails/searchable.rb:178:in solr_remove_all_from_index' /usr/local/Cellar/ruby/1.9.3-p0/lib/ruby/gems/1.9.1/gems/sunspot_rails-1.3.0/lib/sunspot/rails/searchable.rb:197:insolr_reindex'
/Users/pferrel/Projects/FinderBots.com/Rakefile:47:in block (3 levels) in <top (required)>' /Users/pferrel/Projects/FinderBots.com/Rakefile:45:ineach'
/Users/pferrel/Projects/FinderBots.com/Rakefile:45:in `block (2 levels) in <top (required)>'
...

nz, I have tried starting from a clean/no index solr to no avail. This was working at one point since I have a bunch of indexed content.

Does anyone have an idea how to track this down?

I'm getting the same connection refused. This is from my Rails console:

irb(main):023:0> Employer.search { keywords 'hello' }
Errno::ECONNREFUSED: Connection refused - connect(2)
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `initialize'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `open'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/timeout.rb:89:in `timeout'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `connect'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:626:in `start'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:1168:in `request'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/connection.rb:15:in `execute'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:166:in `execute'
    from /Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:161:in `send_and_receive'

This is from my solr session:

(solr *) $ bundle exec rake sunspot:solr:run
2012-02-26 10:50:55.627::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2012-02-26 10:50:55.888::INFO:  jetty-6.1.3
2012-02-26 10:50:56.000::INFO:  Extract jar:file:/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot_solr-1.3.0/solr/webapps/solr.war!/ to /var/folders/5l/b8w61x0d6651k7l965dc335h0000gn/T/Jetty_0_0_0_0_8982_solr.war__solr__-2c5peu/webapp
2012-02-26 10:50:57.970::INFO:  Started SocketConnector @ 0.0.0.0:8982

Same for reindex:

(solr *) $ bundle exec rake sunspot:reindex --trace
** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
rake aborted!
Connection refused - connect(2)
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `initialize'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `open'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `block in connect'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/timeout.rb:44:in `timeout'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/timeout.rb:89:in `timeout'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:644:in `connect'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:626:in `start'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/net/http.rb:1168:in `request'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/connection.rb:15:in `execute'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:166:in `execute'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:67:in `update'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/rsolr-1.0.7/lib/rsolr/client.rb:131:in `delete_by_query'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/indexer.rb:55:in `remove_all'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:in `block in remove_all'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:in `each'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot/session.rb:181:in `remove_all'
/Users/bhilkert/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/sunspot-1.3.0/lib/sunspot.rb:464:in `remove_all'

http://localhost:8982/solr/admin/ works as expected but doesn't have any index data to return results.

[pferrel]

When this happened to me, I found that the port assumed in sunspot was not the same as the port solr was launched on. This can happen if you are not pointing to 8982 in sunspot.yml. For me there was an error running the config code so the sunspot.yml was not being executed. If you run the ruby code in irb check the config to see which port is being assumed. You can hack around the problem by launching solr to use that port.

For a real fix I had to find why the sunspot.yml was not being executed. My problem was related to sunspot_mongoid, which I had to modify.

It looks like my app was looking for 8983, like you said. I started the local instance up on that port and it connected. I'm using mongomapper variant, so good to know it's just a miscommunication in the port. I look through the mongo_mapper version and it does have anything to do with reading configs, so I wonder if it has to do with the main sunspot gem.

[pferrel]

I am using mongoid and sunspot_mongoid. This requires sunspot_rails and the interaction between s_m and s_r was causing the config in sunspot.yml to not get executed, which I fixed in a fork of sunspot_mongoid. Not sure how it all works with mongo_mapper but if your sunspot.yml is not getting used you'll want to fix that. Sorry I don't recall what I fixed.

I have not looked into what sunspot_rails does for the mongoid intergeneration but it is required so you might want to look into it.

Are you able to use the sunspot:reindex rake task? Is search working? I had trouble with these. I had to add the rake task to my rakefile since installing the gems did not create the correct tasks. Also I had trouble with a new method in active record "find_in_batches" which was not implemented in my brew of gems. The creation of a search object also would not work as advertized, I had to employ an alternate method. Let me know if you need for info on any of these.

To my knowledge none of this was a sunspot problem so if you agree perhaps we should take this to email. Drop me a line.

The code to support another ORM is actually pretty minimal. The MM version does actually write the rake task, which once the server was started on the production port (8983), works fine. In production, we're using heroku and will connect to a URI in our config, so it seems to work fine just making the dev search go to the production port.

Thanks for the insight, it definitely helped me get to the bottom of it. For now, I'll just live with it and remember what port to start the dev server on.