silverstripe/silverstripe-fulltextsearch

Adds external full text search engine support to Silverstripe

FullTextSearch module

Adds support for fulltext search engines like Sphinx and Solr to Silverstripe CMS. Compatible with PHP 7.2

Important notes when upgrading to fulltextsearch 3.7.0+

There are some significant changes from previous versions:

Draft content will no longer be automatically added to the search index. This new behaviour was previously an opt-in behaviour that was enabled by adding the following line to a search index:

$this->excludeVariantState([SearchVariantVersioned::class => Versioned::DRAFT]);

A new canView() check against an anonymous user (i.e. someone not logged in) and a ShowInSearch check is now performed by default against all records (DataObjects) before being added to the search index, and also before being shown in search results. This may mean that some records that were previously being indexed and shown in search results will no longer appear due to these additional checks.

These additional checks have been added with data security in mind, and it's assumed that records failing these checks probably should not be indexed in the first place.

Enable indexing of draft content:

You can index draft content with the following yml configuration:

SilverStripe\FullTextSearch\Search\Services\SearchableService:
  variant_state_draft_excluded: false

However, when set to false, it will still only index draft content when a DataObject is in a published state, not a draft-only or modified state. This is because it will still fail the new anonymous user canView() check in SearchableService::isSearchable() and be automatically deleted from the index.

If you wish to also index draft content when a DataObject is in a draft-only or a modified state, then you'll need to also configure SearchableService::indexing_canview_exclude_classes. See below for instructions on how to do this.

Disabling the anonymous user canView() pre-index check

You can apply configuration to remove the new pre-index canView() check from your DataObjects if it is not necessary, or if it impedes expected functionality (e.g. for sites where users must authenticate to view any content). This will also disable the check for descendants of the specified DataObjects. Ensure that your implementation of fulltextsearch is correctly performing a canView() check at query time before disabling the pre-index check, as this may result in leakage of private data.

SilverStripe\FullTextSearch\Search\Services\SearchableService:
  indexing_canview_exclude_classes:
    - Some\Org\MyDataObject
    # This will disable the check for all pagetypes:
    - SilverStripe\CMS\Model\SiteTree

You can also use the updateIsSearchable extension point on SearchableService to modify the result of the method after the ShowInSearch and canView() checks have run.

It is highly recommend you run a solr_reindex on your production site after upgrading from 3.6 or earlier to purge any old data that should no longer be in the search index.

These additional check can have an impact on the reindex performance due to additional queries for permission checks. If your site also indexes content in files, such as pdf's or docx's, using the text-extraction module which is fairly time-intensive, then the relative performance impact of the canView() checks won't be as noticeable.

Details on filtering before adding content to the solr index

• SearchableService::isIndexable() check in SolrReindexBase. Used when indexing all records during Solr reindex.
• SearchableService::isIndexable() check in SearchUpdateProcessor. Used when indexing single records during DataObject->write().

Details on filtering when extracting results from the solr index

• SearchableService::isViewable() check in SolrIndex. This will often be used in CWP implementations that use the CwpSearchEngine class, as well as most custom implementations that call MySearchIndex->search()
• SearchableService::isViewable() check in SearchForm. This will be used in solr implementations where a /SearchForm url is used to display search results.
• Some implementations will call SearchableService::isViewable() twice. If this happens then the first call will be cached in memory so there is virtually no performance penalty calling it a second time.
• If your implementation is very custom and does not subclass nor make use of either SolrIndex or SearchForm, then it's recommended you update your implementation to call SearchableService::isViewable().

Requirements

• Silverstripe 4.0+

Note: For Silverstripe 3.x, please use the 2.x release line.

Documentation

For pure Solr docs, check out the Solr 4.10.4 guide.

See the docs for configuration and setup, or for the quick version see the quick start guide.

For details of updates, bugfixes, and features, please see the changelog.

TODO

• Get rid of includeSubclasses - isn't actually used in practice, makes the codebase uglier, and ClassHierarchy can be used at query time for most of the same use cases

• Fix field referencing in queries. Should be able to do $query->search('Text', 'Content'), not $query->search('Text', SiteTree::class . '_Content') like you have to do now

  • Make sure that when field exists in multiple classes, searching against bare fields searches all of them

  • Allow searching against specific instances too

• Make fields restrictable by class in an index - 'SiteTree#Content' to limit fields to a particular class, maybe 'Content->Summary' to allow calling a specific method on the field object to get the text

• Allow following user relationships (Children.Foo for example)

• Be clearer about what happens with relationships to stateful objects (e.g. Parent.Foo where Parent is versioned)

• Improvements to SearchUpdater

  • Make it work properly when in-between objects (the A in A.B.Foo) update

  • Allow user logic to cause triggering reindex of documents when field is user generated

• Add generic APIs for spell correction, file text extraction and snippet generation

silverstripe-fulltextsearch's Issues

Update references to queuedjobs to symbiote/silverstripe-queuedjobs

This includes configuration YAML e.g.

- silverstripe/queuedjobs
+ symbiote/silverstripe-queuedjobs

And class/namespace references:

- SilverStripe\QueuedJobs\Service\Blah
+ Symbiote\QueuedJobs\Service\Blah

See comments:

• #141 (comment)
• #141 (comment)

Add .upgrade.yml file mapping

To allow users upgrading to the SS4 version of this module to have their class references updated automatically by the upgrader tool.

[Warning] Unknown class passed as parameter

Using PHP < 5.3.7 is_subclass_of() causes an error when indexing a class with an Enum object. The enum passes parameters into the function causing the warning.

A fix would be to modify addAllFulltextFields(...) in SearchIndex.php to strip the parentheses and check if the function exists first:

    public function addAllFulltextFields($includeSubclasses = true) {
        foreach ($this->getClasses() as $class => $options) {
            foreach (SearchIntrospection::hierarchy($class, $includeSubclasses, true) as $dataclass) {
                $fields = DataObject::database_fields($dataclass);

                foreach ($fields as $field => $type) {
                    if (preg_match('/^(\w+)\(/', $type, $match)) $type = $match[1];
                    $type = preg_replace('/\s*\([^)]*\)/', '', $type);
                    if (class_exists($type) && is_subclass_of($type, 'StringField')) $this->addFulltextField($field);
                }
            }
        }
    }

Apache Solr PHP client sending deprecated waitFlush param

Using one of the more recent builds of Solr, I've found that I can run the Solr_Configure task, but not the Solr_Reindex task.

If I run the Solr_Reindex, the error I get is:

ERROR [User Error]: Uncaught Apache_Solr_HttpTransportException: '400' Status: Bad Request
IN GET /dev/tasks/Solr_Reindex
Line 364 in /path/to/webroot/fulltextsearch/thirdparty/solr-php-client/Apache/Solr/Service.php

The stack trace is:

Apache_Solr_Service->_sendRawPost(http://localhost:8983/solr/WebsiteSolrIndex/update?wt=json,<commit expungeDeletes="false" waitFlush="true" waitSearcher="true" />,3600)
Service.php:829

Apache_Solr_Service->commit()
Solr.php:209

Solr_Reindex->run(SS_HTTPRequest)
TaskRunner.php:84

TaskRunner->runTask(SS_HTTPRequest)
RequestHandler.php:288

RequestHandler->handleAction(SS_HTTPRequest,runTask)
Controller.php:194

Controller->handleAction(SS_HTTPRequest,runTask)
RequestHandler.php:200

RequestHandler->handleRequest(SS_HTTPRequest,DataModel)
Controller.php:153

Controller->handleRequest(SS_HTTPRequest,DataModel)
RequestHandler.php:222

RequestHandler->handleRequest(SS_HTTPRequest,DataModel)
Controller.php:153

Controller->handleRequest(SS_HTTPRequest,DataModel)
Director.php:325

Director::handleRequest(SS_HTTPRequest,Session,DataModel)
Director.php:143

Director::direct(dev/tasks/Solr_Reindex,DataModel)
cli-script.php:120

The issue is that the XML being sent includes the waitFlush="true|false" param, which apparently hasn't been used since Solr 1.4 (source: https://bugs.php.net/bug.php?id=62332), and in the latest version of Solr has been removed altogether. The Solr logs show the following error:

1612541 [qtp672845127-19] ERROR org.apache.solr.core.SolrCore  ? org.apache.solr.common.SolrException: Unknown commit parameter 'waitFlush'
    at org.apache.solr.handler.RequestHandlerUtils.validateCommitParams(RequestHandlerUtils.java:85)
    at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:263)
    at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
    at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
    at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:368)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
    at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:695)

I haven't yet checked if there's an updated version of the thirdparty module - but just wanted to log this issue so if others find the same issue, they at least know what needs to be changed. I was able to fix the issue by just changing line 827 of fulltextsearch/thirdparty/solr-php-client/Apache/Solr/Service.php and removing the waitFlush param entirely, but that's obviously only a temp fix until we can update the library :)

Make SearchVariant::call(...) arguments variadic

With PHP 5.6 as the minimum for SS4 we can use variadic arguments for things like this so simplify the method signature.

See comment: #141 (comment)

UNLOAD a core once its renamed

Solr_Configure indirectly writes solr.xml through an API request, adding new cores and reloading as required.
If the class name of an index changes, the old core entry isn't removed from solr.xml.
SilverStripe can't detect renames as such, but can diff existing cores vs. new ones.
See http://wiki.apache.org/solr/CoreAdmin

That's assuming SilverStripe is the only client using Solr cores, which is highly likely.
In the unlikely case there's another client creating its own cores, we could pass in a merge=true parameter
to the Solr_Configure task?

Update namespaces for Subsites and check the integration is working

See comments:

• #141 (comment)
• #141 (comment)
• #141 (comment)

Update Postgres module class namespaces, add to Travis matrix

See comment: #141 (comment)

If we support Postgres in this module it would be good to add it to the Travis build matrix once we know it's working. It's not part of the SS3 version's configuration at this point.

Use namespace imports rather than fully qualified class names in strings

See example: #141 (comment)

Sold won't search Live web pages

It seems the SearchVariantVersioned is causing some weird issues returning only non published pages. I noticed while debugging in SolrIndex.php on line 662 the $fq is returning one parameter with the value:

"+(_versionedstage:"Live" (*:* -_versionedstage:[* TO *]))",

This causes Live pages to not be shown. I can fix this by overriding getFiltersComponent() function by adding the following value to the array:

    public function getFiltersComponent(SearchQuery $searchQuery) {
        return array_merge(
            parent::getFiltersComponent($searchQuery),
            array(' -(ClassName:ProgrammeInformationMirrorPage)', ' -(ClassName:RedirectorPage)', "+(_versionedstage:\"Live\" )")
        );
    }

It's by luck Solr is picking up on the last array value, and not the first.

Wrong inclusion of Phockito in SolrReindexTest and SolrIndexVersionedTest

The inclusion of Phockito::include_hamcrest() should be limited to the test execution as per:
https://github.com/hafriedlander/silverstripe-phockito/

This is not the case in SolrReindexTest and SolrIndexVersionedTest which makes running the tests impossible because of the PHP Fatal error: Cannot redeclare any() (previously declared in [...]/vendor/hafriedlander/phockito/hamcrest-php/hamcrest/Hamcrest.php:384) in [...]/vendor/phpunit/phpunit/PHPUnit/Framework/Assert/Functions.php on line 59

Ensure module is compatible with environment variables

There are some remaining configurable points which look for constants having been defined.

We should update these now to look for environment variables instead, since _ss_environment.php is a thing of the past. It's still possible for users to define constants in their mysite/_config.php files, but we should encourage them to use env vars instead.

See comment: #141 (comment)

Subsites index cleared

There seems to be an issue when running the Solr_Reindex task on a site with subsites clears the search index for all but the main subsite.

v2.3.1: $dirty_indexes incorrect default property value

Fix:
#148

Currently in SearchUpdateCommitJobProcessor:

/**
 * List of dirty indexes to be committed
 *
 * @var array
 */
public static $dirty_indexes = true;

This results in many Warning being thrown as $dirty_indexes is used (from what I could see) exclusively as type array, not boolean.

EG:

#0 in_array() expects parameter 2 to be array, boolean given:: called at [/sites/statsunleashed/releases/20170811014625/fulltextsearch/code/search/processors/SearchUpdateCommitJobProcessor.php:217]

Replace RequestFilter with HTTPMiddleware

The RequestFilter mechanism is still supported but is deprecated in 4.0. We should replace it with an HTTPMiddleware instead.

See comment: #141 (comment)

Solr 6 supported?

Hello everyone,

has anyone used this in combination with Solr 6?

Cheers,
Peter

Global Hamcrest matchers unit tests cause Behat conflicts

The issue is more completely described here: https://gitlab.cwp.govt.nz/cwp/cwp-installer/issues/3 but is basically that when Hamcrest is setup as global (default) then global method name conflicts arise when other code needs to use same-named methods. In the present case, you cannot run a Behat test suite with these set as global.

The fix is to simply pass false to Phockito::include_hamcrest() in the test classes that call it.

Applying this fix and running the module's tests, results in most still passing, except for SolrIndexTest which would be fixed by declaring all methods as public statics in Hamcrest.php within a class, and then calling Hamcrest::anything() or similar. (This after ~11s consideration)

Remove code that has been deprecated in version 2.0

With the 3.0 release we can remove code that was deprecated in version 2.0.

See comment: #141 (comment)

Travis failure when running with subsites module

These tests pass locally with fulltextsearch + subsites, but travis seems to be failing.

I'm debugging this on my fork on travis at https://travis-ci.org/tractorcow/silverstripe-fulltextsearch/builds/68890811

Solr_Reindex task with Translatable content - Cannot index multiple languages' content

I have a website running SS 3.6.x with Translatable module installed. By default, the Solr_Reindex task will only index SiteTree content within the current locale of the user's session.

I've hacked in an extension hook at the beginning and end of the run() method within the FullTextSearch's Solr_Reindex task (/fulltextsearch/code/solr/Solr.php) in order to toggle the translatable enable_locale_filter() method for each iteration of the reindex task. That works, but I'd like to find a better way to approach the problem that doesn't involve hacking the FTS module.

Since the reindex task is broken up by the recordsPerRequest value in the config, I haven't been able to figure out a way to toggle the translatable locale filter for all iterations of the task. It works for the first task, but gets reset for all subsequent iterations.

Could extend hooks be added to the beginning and end of the tasks to toggle and reinstate translatable filters? Or is there another way to provide filter settings that won't be replaced between multiple reindex tasks?

Archived and Unpublished pages do not remove 'Live' variant from index

To replicate:

Create a new page and save it as draft.
Search for the page (in incognito mode) - 0 results.

Publish the page.
Search for the page - 1 result.

Unpublish the page.
Search for the page.
Expected - 0 results.
Actual - PaginatedList reports total size of 1, but is empty.

Publish the page, then Archive it.
Search for the page.
Expected - 0 results.
Actual - PaginatedList reports total size of 1, but is empty.

Versioned alters this query so that we only get the item if it has a live version; the item is being returned from the index still, but is not being added to the list.
https://github.com/silverstripe/silverstripe-fulltextsearch/blob/master/code/solr/SolrIndex.php#L749

However, the total including the ones filtered out is being set as the PaginatedList total:
https://github.com/silverstripe/silverstripe-fulltextsearch/blob/master/code/solr/SolrIndex.php#L786

As a workaround I add these lines to affected DataObjects.

`
public function onBeforeDelete()
{
$this->LastEdited = time();
$this->write();
}

public function onAfterUnpublish()
{
    singleton('nameofyourindex')->delete(
        ClassInfo::baseDataClass($this->ClassName),
        $this->ID,
        array (
          'SearchVariantVersioned' => 'Live',
          'SearchVariantSubsites' => '0',
        )
    );
}

`

Will not reconnect to database if connection times out during long indexes

During a re-index that takes a significant amount of time (imagine an hour) for a single class, there will be no other SQL queries running. As such, the connection will be closed.

When the next class is started, the query to get $total will fail.

We should display a warning/notice and reconnect if this happens.

Investigate todo comment re: failure if you addClass a non-table class

See comments #141 (comment) and #141 (comment)

Fix PSR-4 autoloader structure

The directory names need to be case sensitive to match the class namespaces.

Simplify class type check

See comment: #141 (comment)

Solr results returns contents of files and pages

The solr extension is configured to return all indexed fields. This is redundant as only class name, ID, and a few others are needed. returning a large file will cause memory exceeded errors on some large files.

Delete 2.4 branch - it's super confusing

Yo - the 2.4 branch is the first ever version of fulltextsearch (I assume for SS2.4) and as we are on release 2.2 at the moment, it's quite confusing to have this branch and will probably cause problems in times soon to come

singleton($step['class'])->extend('augmentSQL', $sql); in SearchIndex::getDirtyIDs() causes throwing Fatal Errors

singleton($step['class'])->extend('augmentSQL', $sql); in SearchIndex::getDirtyIDs() function breaks some DataExtension objects when calling to their function augmentSQL(), eg. Translatable, or SiteTreeSubsites, since Translatable::augumentSQL(SQLQuery &$query, DataQuery &$dataQuery = null) is expecting the second parameter is a DataQuery object and directly uses it to call its function getQueryParam() in its implementation without checking if $dataQuery is a non-object or not

So this issue could be either fixed in Translatable class or SearchIndex class. Giving that if we fix here in SearchIndex, non of those DataExtension classes need to do anything, I hope it could fixed here.

large number of errors coming through QueuedJobService

Every time the queued job runs I get bombarded with a large number of error emails. There's several uncatched errors that constantly occur during the queued jobs run. I've tried deleting the job, but because it's baked into the job module there's no way of turning it off. Here's an example of the error that comes out

[Warning] Invalid argument supplied for foreach()
/sites/op/releases/20160331010614/framework/dev/Backtrace.php:93 
• SS_Backtrace::filter_backtrace(,) 
Backtrace.php:170
• SS_Backtrace::get_rendered_backtrace() 
LogErrorEmailFormatter.php:48
• SS_LogErrorEmailFormatter->format(Array) 
LogEmailWriter.php:60
• SS_LogEmailWriter->_write(Array) 
Abstract.php:85
• Zend_Log_Writer_Abstract->write(Array) 
Log.php:428
• Zend_Log->log(Array,3,) 
Log.php:178
• SS_Log::log(Array,3) 
QueuedJobService.php:310
• QueuedJobService->checkJobHealth() 
QueuedJobService.php:778
• QueuedJobService->runQueue(2) 
ProcessJobQueueTask.php:42
• ProcessJobQueueTask->run(SS_HTTPRequest) 
TaskRunner.php:84
• TaskRunner->runTask(SS_HTTPRequest) 
RequestHandler.php:288
• RequestHandler->handleAction(SS_HTTPRequest,runTask) 
Controller.php:200
• Controller->handleAction(SS_HTTPRequest,runTask) 
RequestHandler.php:200
• RequestHandler->handleRequest(SS_HTTPRequest,DataModel) 
Controller.php:157
• Controller->handleRequest(SS_HTTPRequest,DataModel) 
RequestHandler.php:222
• RequestHandler->handleRequest(SS_HTTPRequest,DataModel) 
Controller.php:157
• Controller->handleRequest(SS_HTTPRequest,DataModel) 
Director.php:385
• Director::handleRequest(SS_HTTPRequest,Session,DataModel) 
Director.php:149
• Director::direct(dev/tasks/ProcessJobQueueTask,DataModel) 
cli-script.php:121

There's also this error:

[Error] Broken jobs were found in the job queue
/sites/op/releases/20160331010614/queuedjobs/code/services/QueuedJobService.php:308 

user feedback for Solr_Configure

In CWP 1.1.1 and up there is no feedback for a successful Solr_Configure run through the browser, it would be nice to get a message similar to earlier versions so the user can be sure the Configure was successful.

Remove Phockito test dependency from test suite

We have an opportunity right now to make subsites more extensible/flexible if needed, which should allow us to remove Phockito from the test suite - I assume it's in place to help mocking the global state in SS3.

Worth noting that Subsites is now transitioning to using an injectable SubsiteState which contains this data, rather than public global state modifications.

See comment: #141 (comment)

SearchUpdateImmediateProcessor doesn't implement QueuedJob

When the indexer gets triggered via a queued job I'm getting the following error

$ framework/sake dev/tasks/ProcessJobQueueTask
Running Task ProcessJobQueueTask

[2016-09-17 00:01:57] Running FullTextSearch Update Job and others from 2.

Fatal error: Call to undefined method SearchUpdateImmediateProcessor::setJobData() in /var/www/html/queuedjobs/code/services/QueuedJobService.php on line 214

SolrReindexImmediateHandler calls cli-script.php from webserver which is not allowed.

SolrReindeximmediateHandler calles cli-script.php on line 62. This throws an error because cli-script.php is not allowed to be called via the web server. When running dev/tasks/Solr_Reindex the script is run as the web server user. This will never work properly because of

if(isset($_SERVER['HTTP_HOST'])) {
    echo "cli-script.php can't be run from a web request, you have to run it on the command-line.";
    var_dump($_SERVER);
    die();
}

which is called first thing in cli-script.php.

Solr searching fails when not searching Versioned content

If you create a sub-class of SolrIndex to define indexes on DataObjects that don't use the Versioned extension, the index will be built and loaded into Solr, but no searches performed against that index will work.

For example:

// Story.php
class Story extends DataObject {
    private static $db = array(
        'Title' => 'Varchar(255)'
    );
}

// StorySolrIndex.php
class StorySolrIndex extends SolrIndex {
    public function init() {
        $this->addClass('Story');
        $this->addFulltextField('Title');
    }
}

Running Solr_Configure and Solr_Reindex will work fine (Solr_Reindex will say class Story, total: 5 in state [])

However, creating an instance of that SolrIndex and running a query against it will fail:

$index = new StorySolrIndex();
$query = new SearchQuery();
$query->search('Test', null, array(
    'Story_Title' => 2
));

$results = $index->search($query);

Nothing will be returned, but Solr will log an Exception with the message:
SEVERE: org.apache.solr.common.SolrException: undefined field _versionedstage

If you add something to the StorySolrIndex that is versioned (e.g. SiteTree), then the search will work as usual, but with extra stuff in the index.

// StorySolrIndex.php
class StorySolrIndex extends SolrIndex {
    public function init() {
        $this->addClass('Story');
        $this->addClass('SiteTree'); // This works around the bug
        $this->addFulltextField('Title');
    }
}

BUG no support for obsolete indexes

If SearchUpdateProcessor::process is called with a dirty record marked against an index that no longer exists, then it will fail on line 58 of SearchUpdateProcessor.

if (!$indexes[$index]->variantStateExcluded($state)) { // since $indexes[$index] is null

This only comes up when running a queued job that has been queued against an obsolete codebase.

I suggest that Solr_Reindex flushes all queued search update tasks, since it doesn't make sense to continue running those tasks against an instance if this work will be repeated.

Solr_Configure needs more verbosity!

As part of some user feedback from CWP regarding solr we have found that agencies feel there is a lack of error reporting with regards to a Solr_Configure. I believe one can make this a bit more verbose with a ?Verbose=1 currently, but I may be wrong.

Anyway, it would be good to see any errors that occur on a Solr_Configure by default and maybe a nice success message if there are none.
(see HD16473 for more info)

Remove manual path require calls for thirdparty Solr PHP client library

See comment: #141 (comment)

This code is in the thirdparty directory. If we can't include this as a composer dependency, we should provide a composer compatible autoloader path to it so we can remove some of the old code like this:

    public static function include_client_api()
    {
        static $included = false;

        if (!$included) {
            $solr_php_path = __DIR__. '/../..' . '/thirdparty/solr-php-client/';
            set_include_path(get_include_path() . PATH_SEPARATOR . $solr_php_path);
            require_once($solr_php_path . 'Apache/Solr/Service.php');
            require_once($solr_php_path . 'Apache/Solr/Document.php');

            $included = true;
        }
    }

Can the Webdav port be different from the Solr port?

At the moment I can not find a way to use a different port for webdav than used for Solr (defaulting to 8983).
In some configuration cases that port is different.
I can write my own SolrConfigStore class as a workaround from the looks of it.

Wonder if there can be an optional config value inside indexstore config for the port for more flexibility?

Issue with custom types/definitions being ignored

Hello,

Attempting to use a custom type and definition when searching but it doesn't seem to be used at all.

SearchPeopleIndex.php

class SearchPeopleIndex extends SolrIndex
{
    public function init()
    {
        $this->addClass('SiteTree');
        $this->addClass('Person');
        $this->addFulltextField('AuthorName');
        $this->addCopyField('Person_AuthorName', 'AuthorsPre');
    }

    /**
     * @return mixed
     */
    public function getTypes()
    {
        return $this->renderWith(Director::baseFolder() . '/people-finder/core/templates/types.ss');
    }

    /**
     * @return string
     */
    function getFieldDefinitions()
    {
        $parent = parent::getFieldDefinitions();
        $parent .= $this->renderWith(Director::baseFolder() . '/people-finder/core/templates/definitions.ss');
        return $parent;
    }
}

Types.ss

[...]
<fieldType name="AuthorsPrefix" class="solr.TextField"  positionIncrementGap="100">
    <analyzer type="index">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="200" side="front"/>
    </analyzer>
    <analyzer type="query">
        <tokenizer class="solr.StandardTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory" />
    </analyzer>
</fieldType>

Definitions.ss

<field name="AuthorsPre" type="AuthorsPrefix" indexed="true" multiValued="true"/>

In theory I should be able to search for John S and get a result for John Snow, however I get zero results. When I search for John I get John Snow, and when I search for Snow I also get John Snow, just not when searching for John S...

The query going to Solr looks like

[SearchPeopleIndex] webapp=/solr path=/select params={q=%2Bjohn+%2Bs&json.nl=map&fq=%2B(ClassName:Person)&rows=20&wt=json} hits=0 status=0 QTime=0

Any tips or pointers for how to get this working?

Reindex task removes all indexed content but doesn't replace it with new content.

I find that when I have Solr setup and run the Reindex it removes all content from the Solr core but does not replace it with new content from my Pages/DataObjects.

If I save/publish within the CMS the content is however indexed.

Is this how it is intended or is there a problem with the Reindex task?

cannot install on php 7

Tried to install on

• Centos 7
• PHP 7
• Silverstripe 4
• dnadesign/silverstripe-elemental 2.x-dev

Installation has failed.

Attached a compressed composer.json.

Follows output of composer command.

[vagrant@dev html]$ composer require silverstripe/fulltextsearch
Using version ^2.4 for silverstripe/fulltextsearch
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - silverstripe/framework 3.4.6-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.6-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.6 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.5-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.5 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.4-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.4 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.3-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.1-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.1-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.0-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.4.0 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.4 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.3-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.3-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.2-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.1-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.1-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.0-rc3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.0-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.0-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.0-beta1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.3.0 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.6 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.5-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.5-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.5 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.4-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.4 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.3-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.3-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.2-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.2-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.1-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.1-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.0-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.0-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.2.0 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.21 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.20-rc2 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.20-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.20 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.19-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.19 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.18-rc2 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.18-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.18 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.17-rc2 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.17-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.17 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.16-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.16 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.15 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.14-rc1 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.1.14 requires php >=5.3.2,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.x-dev requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.5-beta1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.5 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.4-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.4 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.3-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.2-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.1-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.1-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.0-rc3 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.0-rc2 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.0-rc1 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - silverstripe/framework 3.5.0 requires php >=5.3.3,<7 -> your PHP version (7.0.25) does not satisfy that requirement.
    - Conclusion: don't install silverstripe/fulltextsearch 2.4.0
    - don't install silverstripe/fulltextsearch 2.5.x-dev|install silverstripe/fulltextsearch 2.x-dev
    - Conclusion: don't install silverstripe/fulltextsearch 2.x-dev
    - Conclusion: remove silverstripe/framework 4.0.0
    - Installation request for silverstripe/fulltextsearch ^2.4 -> satisfiable by silverstripe/fulltextsearch[2.4.0, 2.4.x-dev, 2.x-dev, 2.5.x-dev].
    - Conclusion: don't install silverstripe/framework 4.0.0
    - silverstripe/fulltextsearch 2.4.x-dev requires silverstripe/framework ~3.1 -> satisfiable by silverstripe/framework[3.1.0, 3.1.0-beta1, 3.1.0-beta2, 3.1.0-beta3, 3.1.0-rc1, 3.1.0-rc2, 3.1.0-rc3, 3.1.1, 3.1.10, 3.1.10-rc1, 3.1.10-rc2, 3.1.11, 3.1.11-rc1, 3.1.12, 3.1.13, 3.1.13-rc1, 3.1.14, 3.1.14-rc1, 3.1.15, 3.1.16, 3.1.16-rc1, 3.1.17, 3.1.17-rc1, 3.1.17-rc2, 3.1.18, 3.1.18-rc1, 3.1.18-rc2, 3.1.19, 3.1.19-rc1, 3.1.2, 3.1.2-rc1, 3.1.20, 3.1.20-rc1, 3.1.20-rc2, 3.1.21, 3.1.3, 3.1.3-rc1, 3.1.3-rc2, 3.1.4, 3.1.4-rc1, 3.1.5, 3.1.5-rc1, 3.1.6, 3.1.6-rc1, 3.1.6-rc2, 3.1.6-rc3, 3.1.7, 3.1.7-rc1, 3.1.8, 3.1.9, 3.1.9-rc1, 3.2.0, 3.2.0-beta1, 3.2.0-beta2, 3.2.0-rc1, 3.2.0-rc2, 3.2.1, 3.2.1-rc1, 3.2.1-rc2, 3.2.2, 3.2.2-rc1, 3.2.2-rc2, 3.2.3, 3.2.3-rc1, 3.2.3-rc2, 3.2.4, 3.2.4-rc1, 3.2.5, 3.2.5-rc1, 3.2.5-rc2, 3.2.6, 3.3.0, 3.3.0-beta1, 3.3.0-rc1, 3.3.0-rc2, 3.3.0-rc3, 3.3.1, 3.3.1-rc1, 3.3.1-rc2, 3.3.2, 3.3.2-rc1, 3.3.3, 3.3.3-rc1, 3.3.3-rc2, 3.3.4, 3.4.0, 3.4.0-rc1, 3.4.1, 3.4.1-rc1, 3.4.1-rc2, 3.4.2, 3.4.3, 3.4.3-rc1, 3.4.4, 3.4.4-rc1, 3.4.5, 3.4.5-rc1, 3.4.6, 3.4.6-rc1, 3.4.6-rc2, 3.5.0, 3.5.0-rc1, 3.5.0-rc2, 3.5.0-rc3, 3.5.1, 3.5.1-rc1, 3.5.1-rc2, 3.5.2, 3.5.2-rc1, 3.5.3, 3.5.3-rc1, 3.5.4, 3.5.4-rc1, 3.5.5, 3.5.5-beta1, 3.5.x-dev, 3.6.0, 3.6.0-beta1, 3.6.0-beta2, 3.6.0-rc1, 3.6.1, 3.6.1-alpha2, 3.6.2, 3.6.2-beta1, 3.6.x-dev, 3.x-dev, 3.7.x-dev].
    - Can only install one of: silverstripe/framework[3.1.0, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-beta1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-beta2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-beta3, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-rc2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.0-rc3, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.10, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.10-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.10-rc2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.11, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.11-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.12, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.13, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.13-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.2-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.3, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.3-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.3-rc2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.4, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.4-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.5, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.5-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.6, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.6-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.6-rc2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.6-rc3, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.7, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.7-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.8, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.9, 4.0.0].
    - Can only install one of: silverstripe/framework[3.1.9-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.2.0-beta1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.2.0-beta2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.0, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.0-beta1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.0-beta2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.0-rc1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.1-alpha2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.2, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.2-beta1, 4.0.0].
    - Can only install one of: silverstripe/framework[3.6.x-dev, 4.0.0].
    - Can only install one of: silverstripe/framework[3.x-dev, 4.0.0].
    - Can only install one of: silverstripe/framework[3.7.x-dev, 4.0.0].
    - Installation request for silverstripe/framework (locked at 4.0.0) -> satisfiable by silverstripe/framework[4.0.0].


Installation failed, reverting ./composer.json to its original content.

Update SQLite module class namespaces, add to Travis matrix

See comment: #141 (comment)

If we support SQLite in this module it would be good to add it to the Travis build matrix once we know it's working. It's not part of the SS3 version's configuration at this point.

See related: #160

Solr Synonyms Don't Work

Scenario
I create three different pages.
On each of them I have put one of: gdp, jel, knp (unique words so I know these aren't anywhere else on the site)
I run a Solr_Reindex
I search for "gdp" and just that one page is returned in the results.
Searching any of the unique words returns the correct result.

Now, I'd like to make these words synonyms.
I go into the Fulltext Search tab in Settings of the CMS (which uses CWP's SynonymsSiteConfig)
This config modifies the synonyms.txt file.
I enter gdp, jel, knpas my Solr Synonyms. I run a Solr_Configure.

Result
Now when I search any of the unique words, I get no results.
I do a Reindex just to see if that has an effect. Still no results.

Expected Result
What I would expect, is that if I searched for any of those words that all 3 pages would be returned.

Am I misunderstanding the concept of Solr Synonyms?
Or do they not function correctly in the module?
An issue with version of Solr used in fulltextsearch-local?
Or something else?

Just thought I would get the convo started and see if anyone else is having this issue. 😄

Escape query chars

I get a 400 error in Solr if any of this characters are around

+ - && || ! ( ) { } [ ] ^ " ~ * ? : \ /

Sometimes it behaves well if the character is surrounded by letters instead of spaces, but should be better to escape them anyway. For example, searching for "Wellington/Petone" won't give an error, but "Wellington / Petone" will.
I've added a preg_replace as a workaround, might be worth a look to include it as a fix in the module.

SolrIndex.php (line 376, function search() )

foreach ($query->search as $search) {
    $text = $search['text'];
    $text = preg_replace('/([\+\-!\(\)\{\}\[\]\^"~\*\?:\\/\|]|&&|\|\|)/', '\\\${1}', $text);   <---
    preg_match_all('/"[^"]*"|\S+/', $text, $parts);

Test and ensure various points are working as per Elliot's notes from the SS4 upgrade PR

See comment: #141 (comment)

Example is incorrect

while setting up this module, some issue with the Page class and Title column.

when I set weighting for Title and Content, Solr returned error saying column not found.

In MyIndex.php, Class should be SiteTree not Page ( Title and Content are in SiteTree table.)

    $this->addClass('SiteTree');
    $this->addFulltextField('Title');
    $this->addFulltextField('Content');

After changing to SiteTree, weighting is working.

Document should be updated

Solr_Configure exits with 0 on failure

Solr_Configure failed as it didn't have permission to create the .solr directory but its exit code was still 0. Its not easy to detect failure in deployments which rely on it or in CI.

sudo -u www-data php framework/cli-script.php dev/tasks/Solr_Configure
Running Task Solr_Configure

[2017-06-22 10:55:47] solr_configure.ERROR: Failure: Failed creating target directory /home/ubuntu/{project}/.solr/SolrSearchIndex/conf, please check permissions [] []
[2017-06-22 10:55:47] solr_configure.ERROR: Failure: Failed creating target directory /home/ubuntu/{project}/.solr/FAQSearchIndex/conf, please check permissions [] []

Removing Queued Jobs module reduces functionality to almost nothing

Removing the QueuedJobs module from your site means that you can only do a small-sized index ie <10 pages. This is almost the same as not functioning for a majority of sites.

Two courses of action could help this. One, we highly recommend (and make obvious that we recommend) that QueuedJobs be installed for best performance of the module. Two, we remove the part of the module that allows it to function without QueuedJobs. and upgrade the major version of the module.

Remove unused SapphireTest::is_running_test check and any other references to it

The test suite at the moment is passing without throwing this exception: #141 (comment)

The fact that it's not being executed at the moment is not having an adverse effect on the test suite, so we should just remove this instead.

Also check for other references to is_running_test in the code and remove as necessary.

SearchVariantSubsites alterDefinition issue

In the case where the silverstripe/subsites module is installed and there are multiple classes being indexed, e.g. both SiteTree and File, the alterDefinition method (and then ultimately the SolrIndex->_addAs() method) needs to be able to handle having the "_subsite" field added to multiple classes (in the case example mentioned above, both SiteTree and File).

The issue here is when Solr_Configure is called, it calls the SearchVariantSubsites->alterDefinition method for each of the classes, but each time it just overwrites the "base" and "origin" key to the class called, so only the last class added to the SolrIndex gets the "_subsite" field applied to it, e.g. If we add the "SiteTree" class to the SolrIndex and then the "File" class to the SolrIndex, only the "File" class ends up with the "_subsite" field (because of SolrIndex->_addAs() line 519 - if ($field['base'] == $base) {)

A couple of initial thoughts would be to either allow the $field['base'] value to be an array or it needs to set class specific fields for subsites (rather than the global "_subsite" field), e.g. go through the classes in the index and add {$Class}_SubsiteID to the respective Solr documents.