pat/thinking-sphinx

Sphinx/Manticore plugin for ActiveRecord/Rails

Thinking Sphinx

Thinking Sphinx is a library for connecting ActiveRecord to the Sphinx full-text search tool, and integrates closely with Rails (but also works with other Ruby web frameworks). The current release is v5.6.0.

Upgrading

Please refer to the changelog and release notes for any changes you need to make when upgrading. The release notes in particular are quite good at covering breaking changes and more details for new features.

The documentation also has more details on what’s involved for upgrading from v4 to v5, v3 to v4, and v1/v2 to v3.

Installation

It’s a gem, so install it like you would any other gem. You will also need to specify the mysql2 gem if you’re using MRI, or jdbc-mysql if you’re using JRuby:

gem 'mysql2',          '~> 0.4',    :platform => :ruby
gem 'jdbc-mysql',      '~> 5.1.35', :platform => :jruby
gem 'thinking-sphinx', '~> 5.5'

The MySQL gems mentioned are required for connecting to Sphinx, so please include it even when you’re using PostgreSQL for your database.

You’ll also need to install Sphinx – this is covered in the extended documentation.

Usage

Begin by reading the quick-start guide, and beyond that, the documentation should serve you pretty well.

Requirements

The current release of Thinking Sphinx works with the following versions of its dependencies:

Library	Minimum	Tested Against
Ruby	v2.4	v2.4, v2.5, v2.6, v2.7, v3.0, v3.1, v3.2
Sphinx	v2.2.11	v2.2.11, v3.4.1
Manticore	v2.8	v4.0, v6.0
ActiveRecord	v4.2	v4.2..v7.0

It might work with older versions of Ruby, but it’s highly recommended to update to a supported release.

It should also work with JRuby, but the test environment for that in CI has been unreliable, hence that’s not actively tested against at the moment.

Sphinx or Manticore

If you’re using Sphinx, v2.2.11 is recommended even though it’s quite old, as it works well with PostgreSQL databases (but if you’re using MySQL – or real-time indices – then v3.3.1 should also be fine).

If you’re opting for Manticore instead, v2.8 or newer works, but v4 or newer is recommended as that’s what is actively tested against. The v4.2 and 5.0 releases had bugs with facet searching, but that’s been fixed in Manticore v6.0.

Rails and ActiveRecord

Currently Thinking Sphinx is built to support Rails/ActiveRecord 4.2 or newer. If you’re using Sinatra and ActiveRecord instead of Rails, that’s fine – just make sure you add the :require => 'thinking_sphinx/sinatra' option when listing thinking-sphinx in your Gemfile.

If you want ActiveRecord 3.2-4.1 support, then refer to the 4.x releases of Thinking Sphinx. Or, for ActiveRecord 3.1 support, then refer to the 3.0.x releases. Anything older than that, then you’re stuck with Thinking Sphinx v2.x (for Rails/ActiveRecord 3.0) or v1.x (Rails 2.3). Please note that these older versions are no longer actively supported.

Ruby

You’ll need either the standard Ruby (v2.4 or newer) or JRuby (9.1 or newer).

Database Versions

MySQL 5.x and Postgres 8.4 or better are supported.

Contributing

Please note that this project has a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

To contribute, clone this repository and have a good look through the specs – you’ll notice the distinction between acceptance tests that actually use Sphinx and go through the full stack, and unit tests (everything else) which use liberal test doubles to ensure they’re only testing the behaviour of the class in question. I’ve found this leads to far better code design.

All development is done on the develop branch; please base any pull requests off of that branch. Please write the tests and then the code to get them passing, and send through a pull request.

In order to run the tests, you’ll need to create a database named thinking_sphinx:

# Either fire up a MySQL console:
mysql -u root
# OR a PostgreSQL console:
psql
# In that console, create the database:
CREATE DATABASE thinking_sphinx;

You can then run the unit tests with rake spec:unit, the acceptance tests with rake spec:acceptance, or all of the tests with just rake. To run these with PostgreSQL, you’ll need to set the DATABASE environment variable accordingly:

DATABASE=postgresql rake

Licence

Copyright © 2007-2024, Thinking Sphinx is developed and maintained by Pat Allan, and is released under the open MIT Licence. Many thanks to all who have contributed patches.

thinking-sphinx's Issues

Sphinx 0.9.9 prefers 'listen' over 'address & 'port' in configuration

peterhaza@Shrek lokale-guider (git::master)$ searchd -c config/development.sphinx.conf --console
Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file 'config/development.sphinx.conf'...
WARNING: key 'address' is deprecated in config/development.sphinx.conf line 7; use 'listen' instead.
WARNING: key 'port' is deprecated in config/development.sphinx.conf line 8; use 'listen' instead.
listening on 127.0.0.1:3312

:with time range not working

I have some items that are updated periodically and I use the :with functionality to only show results from updated items. However, the behavior has been unpredictable and not working.

At times it returns no results when it should. Other times it returns results when it shouldn't.

define_index do
  indexes title, :sortable => :true
  indexes description
  indexes retailer(:name), :as => :retailer_name

  # attributes that are not strings
  has retailer_id, created_at, updated_at

  set_property :delta => true
end

and

Item.search 'search terms', :match_mode => :all, :field_weights => { :title => 20, :description => 5 }, :with => { :updated_at => 1.day.ago..Time.now }

Deletion test with alternate primary keys

May not be a reproducible error, but doesn't hurt to add a scenario to cover this. See this thread on the group.

init.rb breaks rake db:schema:load

If you have an empty database and want to load the initial structure with rake db:schema:load (or rake db:create), you'll get an error that it cannot find the table TS is looking for. Of course it can't, it's not created yet! Somehow TS needs to not look for this in rake db tasks.

Search#total_pages - undefined method `/' for nil:NilClass

If you are indexing an ActiveRecord model field but none of those models in the database have a value for that field (they can be nil or an empty string) then starting the sphinx daemon results in these warnings (for a question model, in this case):

WARNING: index 'question_core': preload: failed to load /path/to/project/db/sphinx/development/question_core.spa: bad size 0 (at least 20 bytes expected); NOT SERVING
WARNING: index 'question': no such local index 'question_core' - SKIPPING LOCAL INDEX
WARNING: index 'question': no valid local/remote indexes in distributed index - NOT SERVING

The sphinx daemon continues to start and you can search that model without any problems (Question.search #=> []). However, the ThinkingSphinx::Search object that you get back will raise an exception when you call the total_pages method.

I've created a failing test, and simple/naive fix (in this diff[1]), but when I git cloned the project just now I get other errors (output here[2]) which is why I've just displayed the diff rather than committing back in the normal way.

[1] http://pastie.org/663261
[2] http://pastie.org/663260

Facet.to_param suggestion

Xapit has a very convenient solution for facet url creation, which allows you to get your faceting up and running extremely fast (it just works):

From http://github.com/ryanb/xapit

BEGIN QUOTE
Facets allow you to further filter the result set based on certain attributes.

  <% for facet in @articles.facets %>
    <%= facet.name %>
    <% for option in facet.options %>
      <%= link_to option.name, :overwrite_params => { :facets => option } %>
      (<%= option.count %>)
    <% end %>
  <% end %>

The to_param method is defined on option to return an identifier which will be passed through the URL. Use this in the search.

  Article.search("phone", :facets => params[:facets])

You can also list the applied facets along with a remove link.

  <% for option in @articles.applied_facet_options %>
    <%=h option.name %>
    <%= link_to "remove", :overwrite_params => { :facets => option } %>
  <% end %>

END QUOTE

Their to_param implementation allows it to work for both adding the facet if it doesn't exist, and if it does exist, then depending on breadcrumb_facets? will either remove just that facet, or remove all back to a given facet (breadcrumb_facets? == true):
http://github.com/ryanb/xapit/blob/master/lib/xapit/facet_option.rb

One downside to this implementation is that it doesn't allow you to have breadcrumb navigation links and a remove facet link.

Shop > Shoes (X) > Blue[breadcrumb mode] (X)[remove mode]

It would be nice to be able to configure this "on the fly":

  <% for option in @articles.applied_facet_options %>
    <%= link_to option.name, :overwrite_params => { :facets => option } %>
    <%= link_to "remove", :overwrite_params => { :facets => option.remove } %>
  <% end %>

Having the facet params encoded as a single param enables the breadcrumbs to be displayed in the order the user selects them. This means you don't have to use cookies, etc. to track this ordering. It also means that the controller code is very simple and just has to pass a single param into the Model.search options.

What do you think? :D

facets loads records when it doesn't need to

When faceting, the search query loads activerecord objects but only uses them to get the facets list. This list can be retrieved from the activerecord class.

That allows us to use an ids_only sphinx query and avoid the database queries. This can be a significant gain for models with many facets (as each facet loads its own result set).

There's probably a better way to accomplish this with a refactoring of class_from_crc, but this is my hack for now:

    --- a/thinking-sphinx/lib/thinking_sphinx/facet_search.rb
    +++ b/thinking-sphinx/lib/thinking_sphinx/facet_search.rb
    @@ -55,6 +55,7 @@ def facet_search_options
           max    = config.configuration.searchd.max_matches || 1000

           options.merge(
    +        :ids_only       => true,
             :group_function => :attr,
             :limit          => max,
             :max_matches    => max,
    @@ -105,8 +106,14 @@ def add_from_results(facet, results)
           self[name]  ||= {}

           return if results.empty?
    -      
    -      facet = facet_from_object(results.first, facet) if facet.is_a?(String)
    +
    +      if facet.is_a?(String)
    +        first_result = results.results[:matches].first
    +        result_class = results.send(:class_from_crc, first_result[:attributes]['class_crc'])
    +        facet = facet_from_object(result_class, facet)
    +      end

           results.each_with_groupby_and_count { |result, group, count|
             facet_value = facet.value(result, group)

has_many :through not interpreted correctly

Originally posted on StackOverflow, but asked to copy over here, too.

# Model code...
class Publication < ActiveRecord::Base
  has_many :issues
  has_many :articles, :through => :issues
end

class Issue < ActiveRecord::Base
  belongs_to :publication
  has_many :articles
end

class Article < ActiveRecord::Base
  belongs_to :issue
  define_index do
    has issue(:publication_id), :as => :publication_id
  end
end

The following line, which seems like it would work,

Publication.first.articles.search 'foobar'

returns:

RuntimeError: Missing Attribute for Foreign Key publication_id
    from /home/matchu/rails/torch/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/active_record/has_many_association.rb:18:in `search'

That seems to imply that publication_id isn't set correctly in the index. But it is.

Article.search :with => {:publication_id => 1} # acts as expected

Not epically urgent, since using the syntax above is a functional workaround, but that syntax above is, additionally, no fun.

rake ts:config does not create any sources when cache_classes = true

I was not sure if this is the as the threadsafe! problem, but I cannot get the rake task to output any sources when running in production mode (or turning on cache_classes).
I run the most recent code and I saw in the commit that the following lines have been added less than a day ago:

return if defined?(Rails) && Rails.configuration.cache_classes

When I remove the line of code, everything works just fine.

In a define_index block, provide a way to force a join in the generated SQL

This is needed when you want to define an attribute using a SQL fragment that references a column in another table. Current work around is to define an attribute on a column in the other table using the association in your model, but this isn't a very elegant solution, especially when you don't want the additional attribute.

An example of a real-world need for this can be found in this post from the google group.

geo search broken since 1b0c1ed7

The anchor attribute keys changed names in ThinkingSphinx::Search, but Riddle can't handle the new names. This causes an obscure exception about calling length on nil.

This code:

    lat && lon ? {
      :latitude_attribute   => lat_attr.to_s,
      :latitude             => lat,
      :longitude_attribute  => lon_attr.to_s,
      :longitude            => lon
    } : nil

was replaced by this:

  {
    :latitude       => options[:geo] ? options[:geo].first : options[:lat],
    :longitude      => options[:geo] ? options[:geo].last  : options[:lng],
    :latitude_attr  => latitude_attr,
    :longitude_attr => longitude_attr
  }

When it should have been:

  {
    :latitude       => options[:geo] ? options[:geo].first : options[:lat],
    :longitude      => options[:geo] ? options[:geo].last  : options[:lng],
    :latitude_attribute  => latitude_attr.to_s,
    :longitude_attribute => longitude_attr.to_s
  }

Erroneous facets when used on attribute

See http://groups.google.com/group/thinking-sphinx/msg/8dae7cc604261e30 for history

When switching from

indexes tags.name, :as => :tag, :facet => true 

to

has tags.name, :as => :tag, :facet => true 

then Item.facets returns { ..., :tag => { :nil =>
43 }} instead of anything usable like { ... :tag => { "video" =>
10, ... }}

I'll try to get more information if I can.

• Thibaut

ThinkingSphinx::Search.total_entries is not capped by max_matches

Not sure if this is a bug or intentional behaviour, but if the number of search results exceeds the current max_matches setting, total_pages et al. are adjusted properly, but total_entries still reports the original number.

i.e., for 1500 Jobs and a max_matches setting of 1000:

results = Job.search
results.per_page #=> 20
results.total_pages #=> 50
results.total_entries #=> 1500

Shouldn't total_entries be 1000? As you can only retrieve 50 pages (with each 20 entries)? If this is indeed intentional behaviour, how can I retrieve the correct number of search results (i.e., that are actually retrievable)?

Explicit :classes option in searches should limit searches to related indexes.

To ensure errors are avoided for missing attributes on classes that aren't actually being searched, the indexes that are required should be passed through to the Riddle client instance. The format is a single string separated by commas.

scopes for facets

Is there a way to reuse the same scope for search and for
facet search?

Here is example what I mean (from online tutorial)..
scope = Article.latest_first.by_name('Puck')
articles = scope.search
facets = scope.facets

multi-field search should allow :match_mode => :extended2

The :conditions version of Model.search doesn't take a :match_mode argument because it requires :extended match mode. Perhaps in the sphinx-0.9.9 branch of TS :extended could be replaced with :extended2 across the board and this problem would be eliminated... otherwise, :match_mode => :extended2 could be made a valid option for Model.search :conditions => { ... }

private method `new' called for ThinkingSphinx::Configuration:Class

Getting this error with a new install of the plugin. Seems to work in development mode but borks in staging..

Stack trace: http://gist.github.com/122075

Having a one-to-many-to-many association throws error when trying to run facets, but indexing works.

Example:

House ==> has many bedrooms
Each bedroom has many doors, of a certain color.

I don't have a :through relationship for house and bedrooms because it's one to many, so house_id is stored on the bedroom table.

It is defined as:
class House << ActiveRecord::Base
has_many :bedrooms

define_index do
indexes bedrooms.doors.color, :as => :bedroom_door_color, :facet => true
end

end

That does index properly, but when I try to run facets using House.facets, I get the
following error:

NoMethodError: undefined method 'doors' for #Array:0x271ad5c
from /opt/local/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/
active_record/associations/association_proxy.rb:148:in send' from /opt/local/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/ active_record/associations/association_proxy.rb:148:insend'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet.rb:95:in translate' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/facet.rb:94:ineach'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet.rb:94:in translate' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/facet.rb:75:invalue'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:112:in
add_from_results' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/search.rb:183:in each_with_groupby_and_count'
from /opt/local/lib/ruby/gems/1.8/gems/activerecord-2.3.2/lib/
active_record/attribute_methods.rb:211:in each_with_index' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/search.rb:182:ineach'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/search.rb:182:in
each_with_index' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/search.rb:182:in each_with_groupby_and_count'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:111:in
add_from_results' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:47:in populate'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:45:in
each' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:45:in populate'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/facet_search.rb:11:in
initialize' from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/ plugins/thinking-sphinx/lib/thinking_sphinx/search_methods.rb:404:in new'
from /Users/manish/AcceleWeb/srcror/allvineyards.com/vendor/
plugins/thinking-sphinx/lib/thinking_sphinx/search_methods.rb:404:in
`facets'
from (irb):4>>

Disable loading specifed plugins from vendor directory

I used the patched version of after_commit plugin and they crash when I try load the thinking-sphinx gem.
So, сan be to list plug-ins which it is not necessary to load in the THINKING_SPHINX_DISABLED_PLUGIN constant ?

:sql_order broken on 1.2.x

Investigation and brief discussion is on google groups: http://groups.google.com/group/thinking-sphinx/browse_thread/thread/a12f16a3d32dcd68/cd19866066b9d43f

Searching by Fields not working for 1.2.x

With freelancing-god-thinking-sphinx (1.1.x) I was able to search by field as

User.search :conditions => {:name => "pat"}
or
User.search "@name pat", :match_mode => :extended

but for 1.2.x is not working anymore.

Am I missing something?

Some subclasses not being instantiated (crc can't be found in hashmap)

So I have a superclass with 4 subclasses (STI) and a define_index block in the superclass.

Sometimes, when searching (in development mode), it seems that all the subclasses aren't loaded. The TS::Collection.class_from_crc method seems to be failing because of this.

NoMethodError: You have a nil object when you didn't expect it!
The error occurred while evaluating nil.constantize
    from /usr/local/ruby18/lib/ruby/gems/1.8/gems/freelancing-god-thinking-sphinx-1.1.19/lib/thinking_sphinx/collection.rb:101:in `class_from_crc'

I've tracked this down (via a ruby-debug session) to the fact that the subclasses call on L96 of collection.rb isn't actually pulling in all the subclasses; therefore, the @@models_by_crc hash doesn't actually include all the relevant models & crcs.

I'm guessing this has something to do with the class auto-loading in development mode.... Let me know if I can provide any other helpful info.

Facets of associations don't work correctly

My class Project has_many locations, so in project.rb I wrote:

indexes locations(:name), :as=>:location_name, :facet=>true 

All good and true, but facet support doesn't give me quite the results that I'd expect: A project P1 has locations L1, L2, another project P2 has locations L2 and L3. I'd like to see facets like these:

• L1 (1)
• L2 (2)
• L3 (1)

Instead I get

• [L1, L2] (1)
• [L2, L3] (1)

This has been discussed on http://groups.google.com/group/thinking-sphinx/browse_thread/thread/19c8d749f9dcd578

Escape excerpts by default

Excerpts don't seem to be escaped by default.
Using the excerpt feature and not escaping its output can get very nasty if the excerpted content is text with HTML tags.

I was using:

result.excerpts.text

Which was rendering:
<span class="match">Hi</span>!<br> [...]Create a profile for each of your <a href="http:// &#8230;

The <a href> tag from my results was left open because thinking sphinx was excerpting it.

To fix this I now use a less elegant version:
@results.excerpt_for(h result.text)

Maybe it would make sense to escape the text in the excerpt method?

Scope usage doesn't match with documentation

Scopes currently return a ThinkingSphinx::Search instance, rather than a scoped instance of the model or whatnot. This means that scope chaining and calling ".search" on a scope both fail.

Additionally, if using a scope that takes a parameter, Ruby will fail to parse a statement like:

Model.my_scope("foo") :title => "Foobar"

For example:

class Node < ActiveRecord::Base
  define_index do
    indexes title, :sortable => :true
    has deleted
  end

  sphinx_scope(:is_active) { { :with => {:deleted => 0} } }
  sphinx_scope(:with_title) {|title| {:conditions => {:title => title} }}
end

The lack of a search method tends to make this break rather horribly.

>> Node.search("Funny People")
=> [#<Nodes::Movie id: 177180, title: "Funny People", ...>]

>> Node.is_active.search("Funny People")
NoMethodError: undefined method `search' for #<ThinkingSphinx::Search:0xca4f658>

>> Node.with_title "Funny People"
=> [#<Nodes::Movie id: 177180, title: "Funny People", ...>]

>> Node.with_title("Funny People").is_active
NoMethodError: undefined method `is_active' for #<ThinkingSphinx::Search:0xcaf2f9c>
 from vendor/plugins/thinking-sphinx/lib/thinking_sphinx/search.rb:76:in `method_missing'
 from (irb):9

>> Node.with_title("Funny People") :with => {:deleted => 0}
SyntaxError: compile error
(irb):12: syntax error, unexpected ':', expecting $end
Node.with_title("Funny People") :with => {:deleted => 0}
                                 ^
 from (irb):12
 from :0

array-valued attribute filters cannot include nil?

Having some issues with queries that have array-valued attribute filters that may include nil. It returns an empty array when it shouldn't. i.e., the following works fine:

Job.extended_search 'foo', :with => { :education_level_id => [ 1, 2, 3, 4 ] } # => non-empty array

the following also:

Job.extended_search 'foo', :with => { :education_level_id => nil } # => non-empty array


but the following doesn't:

Job.extended_search 'foo', :with => { :education_level_id => [ nil, 1, 2, 3, 4 ] } # => empty array

It just returns an empty array.

Installed libraries should be preferred over those bundled with TS

I'd prefer that TS use my installed version of delayed_job instead of the bundled one if it is available.

http://github.com/brandon/thinking-sphinx/commit/cc697a463351388ba593955f1fe556f5fc8675a5

Autoload capistrano tasks

Most capfiles (I think) have a line like:

Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

Maybe if there was a recipes/thinking_sphinx.rb that just loaded thinking_sphinx/deploy , world peace might come about. Or at least, autoloading of the recipes.

Large CRC codes causing errors in PostgreSQL

In some versions of Postgres, CRCs larger than 2^31 would cause casting errors. Patch below from Luis Avila fixes this.

line 43 --- lib/thinking_sphinx/adapters/postgresql_adapter.rb
"COALESCE(#{clause}, #{default})"
to
"COALESCE(CAST(#{clause} as bigint), #{default})"

The 0.9.9 branch doesn't pass through new sphinx settings

The one in particular I as trying to use was index_exact_words, but there are probably others, so some kind of audit/reconciliation needs to be performed

define_index before a association is defined causes invalid sql but no error

If you have a define_index that runs before an indexed association is defined, TS quietly sets the index as if the association didn't exist and it was instead set on the model itself.

class Comment < AR:B
  define_index do
    indexes post.name
  end

  belongs_to :post
end

That will result in comment_core sql that loads "comment.name" instead of "post.name". It would be good if this failed immediately.

I found a reference to this (I think) at this link: http://www.mail-archive.com/[email protected]/msg02013.html

Indexation problem

Hello,
You said to me to post a bug report in google groups:
I've isolated the problem in a little github project:
http://github.com/tibastral/ts-bug

Txs for Ur help

first migration error

I have the same problem specified at http://github.com/freelancing-god/thinking-sphinx/issues/closed/#issue/1.

Here is the trace:

$ rake db:migrate --trace
(in /Users/massi/Progetti/Lavoro/w/working_copy)
** Invoke db:migrate (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
Mysql::Error: Table 'wasa3_development.users' doesn't exist: SHOW FIELDS FROM `users`
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/connection_adapters/abstract_adapter.rb:212:in `log'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/connection_adapters/mysql_adapter.rb:320:in `execute'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/connection_adapters/mysql_adapter.rb:466:in `columns'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1276:in `columns'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/base.rb:1289:in `column_names'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/index.rb:239:in `crc_column'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/index.rb:251:in `add_internal_attributes_and_facets'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/index.rb:41:in `initialize'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/active_record.rb:69:in `new'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/vendor/plugins/thinking-sphinx/lib/thinking_sphinx/active_record.rb:69:in `define_index'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/app/models/user.rb:72
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:380:in `load_without_new_constant_marking'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:380:in `load_file'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:521:in `new_constants_in'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:379:in `load_file'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:259:in `require_or_load'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:425:in `load_missing_constant'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:80:in `const_missing_not_from_s3_library'
/Library/Ruby/Gems/1.8/gems/aws-s3-0.5.1/lib/aws/s3/extensions.rb:174:in `const_missing'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:92:in `const_missing'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/inflector.rb:361:in `constantize'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/inflector.rb:360:in `each'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/inflector.rb:360:in `constantize'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/core_ext/string/inflections.rb:162:in `constantize'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:157:in `observed_class'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:183:in `observed_classes'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:166:in `initialize'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/singleton.rb:94:in `new'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/singleton.rb:94:in `instance'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:38:in `instantiate_observers'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:36:in `each'
/Library/Ruby/Gems/1.8/gems/activerecord-2.3.2/lib/active_record/observer.rb:36:in `instantiate_observers'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.2/lib/action_controller/dispatcher.rb:16:in `define_dispatcher_callbacks'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:182:in `call'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:182:in `evaluate_method'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:166:in `call'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:90:in `run'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:90:in `each'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:90:in `send'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:90:in `run'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/callbacks.rb:276:in `run_callbacks'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.2/lib/action_controller/dispatcher.rb:52:in `send'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.2/lib/action_controller/dispatcher.rb:52:in `run_prepare_callbacks'
/Library/Ruby/Gems/1.8/gems/rails-2.3.2/lib/initializer.rb:609:in `prepare_dispatcher'
/Library/Ruby/Gems/1.8/gems/rails-2.3.2/lib/initializer.rb:183:in `process'
/Library/Ruby/Gems/1.8/gems/rails-2.3.2/lib/initializer.rb:113:in `send'
/Library/Ruby/Gems/1.8/gems/rails-2.3.2/lib/initializer.rb:113:in `run'
/Users/massi/Progetti/Lavoro/wasa3/working_copy/config/environment.rb:9
/Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
/Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in `require'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:521:in `new_constants_in'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.2/lib/active_support/dependencies.rb:156:in `require'
/Library/Ruby/Gems/1.8/gems/rails-2.3.2/lib/tasks/misc.rake:4
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:617:in `call'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:617:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:612:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:612:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:578:in `invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:571:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:588:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:585:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:585:in `invoke_prerequisites'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:577:in `invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:571:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:564:in `invoke'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2027:in `invoke_task'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2005:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2044:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:1999:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:1977:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:2044:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/lib/rake.rb:1974:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.4/bin/rake:31
/usr/bin/rake:19:in `load'
/usr/bin/rake:19

I have found this http://github.com/sml/thinking-sphinx/blob/3d7eda29ec2186d6c21747f9a60e40d5455c8629/lib/thinking_sphinx/index.rb.

I think that the line 180 could be the solution, but in #issue/1 you say that this problem is already fixed.

:sortable option does not work with :as option

setup in Model

define_index do
  indexes :name, :as => 'name', :sortable => true
end

config file snippet

sql_query = SELECT `model`.`id` * 2 + 0 AS `id` , CAST(`model`.`name` AS CHAR) AS `name_sort`, ...

expected config file snippet

sql_query = SELECT `model`.`id` * 2 + 0 AS `id` , CAST(`model`.`name` AS CHAR) AS `name`, ... 

originally reported in the google group at http://groups.google.com/group/thinking-sphinx/browse_thread/thread/c39d916425e68414

newest gem 1.2.12 needs a gemspec update

Hi Pat,

I grabbed the latest gem off of gemcutter, 1.2.12 and it looks like the gem.files in the gemspec is missing a bunch of file, specifically the "recipes/*" for the capistrano deployement and "features" folder.

Or is this not supposed to be available for the gem version, only the plugin?

Thanks,
~ Tom

Don't bundle Delayed Job

Thinking Sphinx shouldn't bundle Delayed Job. Installation should be left up to the user.

Floating-point / decimal facets have incorrect values.

Adding a facet for a floating-point or decimal facet gives crazy values, not what actually exists in the db.

See https://gist.github.com/b20078db3dbf7c8a9bd0/9e032e5e786ad3431e4ae9c757f5190eab9bd07b

rake ts:config doesn't output any sources or indexes with config.threadsafe!

Rails 2.3.2

I've confirmed on my EngineYard slice and my local dev box (OS X).

Without config.threadsafe!, everything works just fine. http://gist.github.com/114935

With config.threadsafe!, rake ts:config doesn't output any sources or indexes into *.sphinx.conf. Note that I do get indexer and searchd. http://gist.github.com/114936

undefined method sphinx_scope

I am getting an undefined method error for sphinx_scope
I am using ruby enterprise version 1.8.6-20090421

this is what is in my environment.rb

config.gem(
'freelancing-god-thinking-sphinx',
:lib => 'thinking_sphinx',
:version => '1.2.9', :source => 'http://gems.github.com'
)

config.gem 'mislav-will_paginate', :version => '~> 2.3.11', :lib => 'will_paginate',

:source => 'http://gems.github.com'

here's my gem list
actionmailer (2.3.3, 2.3.2)
actionpack (2.3.3, 2.3.2)
activerecord (2.3.3, 2.3.2)
activeresource (2.3.3, 2.3.2)
activesupport (2.3.3, 2.3.2)
capistrano (2.5.8)
cassandra (0.5.6.2)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
fastthread (1.0.7)
freelancing-god-thinking-sphinx (1.2.9)
gem_plugin (0.2.3)
haml (2.2.3)
highline (1.5.1)
hpricot (0.8.1)
htmldoc (0.2.1)
httpclient (2.1.5.2)
image_science (1.2.1)
json (1.1.9)
memcached (0.16.3)
mislav-will_paginate (2.3.11)
mms2r (2.3.0)
mongrel (1.1.5)
mysql (2.7)
net-scp (1.0.2)
net-sftp (2.0.2)
net-ssh (2.0.15)
net-ssh-gateway (1.0.1)
passenger (2.2.5, 2.2.1)
postgres (0.7.9.2008.01.28)
rack (1.0.0, 0.9.1)
rails (2.3.3, 2.3.2)
rake (0.8.4)
RubyInline (3.8.3)
soap4r (1.5.8)
sqlite3-ruby (1.2.4)
thrift (0.0.810255.1)
tmail (1.2.3.1)

ZenTest (4.1.4)

and here's the stack trace

=> Booting Mongrel
=> Rails 2.3.3 application starting on http://0.0.0.0:3000
/opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activerecord-2.3.3/lib/active_record/base.rb:1959:in method_missing_without_paginate': undefined methodsphinx_scope' for #Class:0x3913590 (NoMethodError)
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/mislav-will_paginate-2.3.11/lib/will_paginate/finder.rb:168:in method_missing' from /media/disk/var/www/xoonoo.com/current/app/models/user.rb:98 from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:380:inload_without_new_constant_marking'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:380:in load_file' from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:521:innew_constants_in'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:379:in load_file' from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:259:inrequire_or_load'
from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/activesupport-2.3.3/lib/active_support/dependencies.rb:425:in load_missing_constant' ... 35 levels... from /opt/ruby-enterprise/lib/ruby/gems/1.8/gems/rails-2.3.3/lib/commands/server.rb:84 from /opt/ruby-enterprise/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:ingem_original_require'
from /opt/ruby-enterprise/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from script/server:3

I had this working before but I have just moved to a new workstation and I am trying to setup my environment and now I get this error. Any ideas ?

indexer doesn't like configuration generated when you have no user set in database.yml

A coworker was running into this issue:

using config file '/Users/Armageddon/Projects/Business/VersusTech/fanvsfan/config/development.sphinx.conf'...
indexing index 'claim_core'...
ERROR: index 'claim_core': sql_connect: Access denied for user ''@'localhost' to database 'fan_development' (DSN=mysql://:***@localhost:3306/fan_development).
total 0 docs, 0 bytes
total 0.122 sec, 0.00 bytes/sec, 0.00 docs/sec
distributed index 'claim' can not be directly indexed; skipping.

It turned out his database.yml was like this:

development:
  adapter: mysql
  database: fan_development
  host: localhost
  encoding: utf8

Most notably, it's missing a user. activerecord seems to handle this gracefully, but the indexer, not so much. They were able to fix this by adding user: root, which MySQL must have been assuming.

After digging around, I found a line in lib/thinking_sphinx/source.rb that sets the default username to "". Perhaps this could be set to "root" instead?

Remote MySQL support

I was pretty sad when I found out that, on my production server, MySQL lives on another server - and, from what I can figure out, Thinking Sphinx doesn't have support for remote MySQL - even though Sphinx itself does. I'm tempted to go in and edit it up myself, but as someone who knows nothing of the inner workings of the plugin... yeah.

Thanks for readin', and here's hoping this sort of change would be something simple!

Conflict when app contains Search class

You can recreate this with a new Rails app with the following (and appropriate DB):

config.gem "freelancing-god-thinking-sphinx", :lib => "thinking_sphinx"

class User < AR::Base; has_many :searches; end
class Search < AR::Base; belongs_to :user; end

Now, you get errors when you try to access @user.searches:

>> User.new.searches
NoMethodError: undefined method `quoted_table_name' for ThinkingSphinx::ActiveRecord::Search:Module

Obviously, this should be the Search model, not the TS:AR:Search module. I think the problem really comes down to this:

>> User.send :compute_type, "Search"
=> ThinkingSphinx::ActiveRecord::Search

...which if you look in the Rails source, is in the end doing something like this (base.rb L2198):

>> User.class_eval 'Search'
=> ThinkingSphinx::ActiveRecord::Search

I don't really know how to fix this though... :/

MVA with NULLs don't store them as zeros

For MySQL, GROUP_CONCAT doesn't keep NULLs, so would need to convert NULLs to 0's before the concat. Wouldn't hurt to do the same for PostgreSQL as well.

String facets through multiple association levels are unreliable

Not all results are getting returned properly in some cases - see closed issue #31.

Thinking Sphinx clobbers named_scope?

I'm new to Sphinx and Thinking Sphinx but I have it up and running locally and recently stumbled across something disturbing. When I combined a Thinking Sphinx search with an existing named_scope, I get unexpected results.

>> Product.available(true)
=> [#<Product id: 1, available: true, name: "iPod">]
>> Product.available(true).search("iPod")
=> [#<Product id: 1, available: true, name: "iPod">]
>> Product.available(true).search("Zune")
=> []
>> Product.available(false)
=> []
>> Product.available(false).search("iPod")
=> [#<Product id: 1, available: true, name: "iPod">]
>> Product.available(false).search("Zune")
=> []

These all generate the expected output, except for the fifth entry. I've read about problems with Thinking Sphinx generating unwanted nil values in the returned records when used with named_scope, but this seems to be a different issue altogether. Is this a known bug? Not a bug at all? What am I missing?

<model>.search returns nils in array of results if records are hidden by the default scope

I'm using soft-deletion via is_paranoid, which adds a default scope w/ e.g. :conditions => 'deleted_at IS NOT NULL' to your models.

I've added where 'deleted_at IS NOT NULL' to my index, but if results are returned in-between indexes there are nils in place of the destroyed records (which my views don't like).

This seems to me to be a bug - excuse me if this is a double post, but a cursory look didn't turn up any duplicates. I'll take a quick look and see if I can contribute a patch.

Does not work on Rails 3

It drops out with:
`require': no such file to load -- action_controller/dispatcher (MissingSourceFile)

The problem appears in init.rb where you require it.

Calling Model.facets results in numerous SQL database queries

Calling Model.facets results in SQL database queries even though it returns a hash of facets. Thinking Sphinx should be able to return the facets as stored in the Sphinx database.

excerpts and single table inheritance

I have a Feature model and an Event model which inherits from Feature. My indexes are defined in Feature. If I do a search on Feature and try to get excerpts from the results, I get this error because the model returned is actually Event, not Feature :

Riddle::ResponseError: searchd error (status: 1): unknown local index 'event_core' in search request

It should actually use feature_core, not event_core.

field_weight property not honored in model in production environment

It seems that the field_weight property when defined in model is not being recognized when in production environment. It does work however when running in development environment.

The field weights are of course still recognized when put in the individual searches.