Issues with search re-indexing

[brynwhyman]

Issues with search re-indexing

[sminnee]

This seems to be the error:

[2019-04-01 12:10:03][INFO] Job caused exception mapper [downloads] of different type, current_type [long], merged_type [text] in /var/www/mysite/releases/f62b7d0443bd65b2e0e71cb04651b00d73f80b41/vendor/ruflin/elastica/lib/Elastica/Transport/Guzzle.php at line 101
[2019-04-01 12:10:03][INFO] Defining the mappings
ERROR [Emergency]: Uncaught Elastica\Exception\ResponseException: mapper [downloads] of different type, current_type [long], merged_type [text]
IN GET dev/tasks/ProcessJobQueueTask
Line 101 in /var/www/mysite/releases/f62b7d0443bd65b2e0e71cb04651b00d73f80b41/vendor/ruflin/elastica/lib/Elastica/Transport/Guzzle.php

Source
======
92: 
93: $response->setTransferInfo(
94: [
95: 'request_header' => $request->getMethod(),
96: 'http_code' => $res->getStatusCode(),
97: ]
98: );
99: 
100: if ($response->hasError()) {
* 101: throw new ResponseException($request, $response);
102: }
103: 
104: if ($response->hasFailedShards()) {
105: throw new PartialShardFailureException($request, $response);
106: }
107: 

Trace
=====
Elastica\Transport\Guzzle->exec({"path":"ss_mysite\/Addon\/_mapping","method":"PUT","data":{"Addon":{"properties":{"name":{"type":"string"},"description":{"type":"string"},"type":{"type":"string"},"compatibility":{"type":"string"},"vendor":{"type":"string"},"tags":{"type":"string"},"released":{"type":"date"},"downloads":{"type":"string"},"readme":{"type":"string"}}}},"query":[],"contentType":"application\/json","connection":{"config":{"url":null,"bigintConversion":false},"host":"search-cloudy-ssaddon1-prod2xes-toi7xpgnfuyoa3x7zkipva7j5q.ap-southeast-2.es.amazonaws.com","port":80,"path":null,"proxy":null,"transport":"AwsAuthV4","persistent":true,"timeout":5,"connections":[],"roundRobin":false,"log":false,"retryOnConflict":0,"username":null,"password":null,"aws_region":"ap-southeast-2","enabled":true}}, Array)
Request.php:193

Elastica\Request->send()
Client.php:674

Elastica\Client->request(ss_mysite/Addon/_mapping, PUT, Array, Array)
Client.php:706

Elastica\Client->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Index.php:562

Elastica\Index->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Type.php:523

Elastica\Type->requestEndpoint(Elasticsearch\Endpoints\Indices\Mapping\Put)
Mapping.php:250

Elastica\Type\Mapping->send()
ElasticaService.php:218

Heyday\Elastica\ElasticaService->define()
ReindexTask.php:41

Heyday\Elastica\ReindexTask->run(SilverStripe\Control\HTTPRequest)
RunBuildTaskJob.php:80

Symbiote\QueuedJobs\Jobs\RunBuildTaskJob->process()
QueuedJobService.php:723

Symbiote\QueuedJobs\Services\QueuedJobService->runJob(568)
QueuedJobService.php:1117

Symbiote\QueuedJobs\Services\QueuedJobService->processJobQueue(2)
QueueRunner.php:23

Symbiote\QueuedJobs\Tasks\Engines\QueueRunner->runQueue(2)
QueuedJobService.php:1094

Symbiote\QueuedJobs\Services\QueuedJobService->runQueue(2)
ProcessJobQueueTask.php:59

Symbiote\QueuedJobs\Tasks\ProcessJobQueueTask->run(SilverStripe\Control\HTTPRequest)
TaskRunner.php:104

SilverStripe\Dev\TaskRunner->runTask(SilverStripe\Control\HTTPRequest)
RequestHandler.php:323

SilverStripe\Control\RequestHandler->handleAction(SilverStripe\Control\HTTPRequest, runTask)
Controller.php:284

SilverStripe\Control\Controller->handleAction(SilverStripe\Control\HTTPRequest, runTask)
RequestHandler.php:202

SilverStripe\Control\RequestHandler->handleRequest(SilverStripe\Control\HTTPRequest)
Controller.php:212

SilverStripe\Control\Controller->handleRequest(SilverStripe\Control\HTTPRequest)
RequestHandler.php:226

SilverStripe\Control\RequestHandler->handleRequest(SilverStripe\Control\HTTPRequest)
Controller.php:212

SilverStripe\Control\Controller->handleRequest(SilverStripe\Control\HTTPRequest)
Director.php:361

SilverStripe\Control\Director->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)
VersionedHTTPMiddleware.php:41

SilverStripe\Versioned\VersionedHTTPMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
BasicAuthMiddleware.php:68

SilverStripe\Security\BasicAuthMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
AuthenticationMiddleware.php:61

SilverStripe\Security\AuthenticationMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
CanonicalURLMiddleware.php:188

SilverStripe\Control\Middleware\CanonicalURLMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
HTTPCacheControlMiddleware.php:42

SilverStripe\Control\Middleware\HTTPCacheControlMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
ChangeDetectionMiddleware.php:27

SilverStripe\Control\Middleware\ChangeDetectionMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
FlushMiddleware.php:29

SilverStripe\Control\Middleware\FlushMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
RequestProcessor.php:66

SilverStripe\Control\RequestProcessor->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
SessionMiddleware.php:20

SilverStripe\Control\Middleware\SessionMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
AllowedHostsMiddleware.php:60

SilverStripe\Control\Middleware\AllowedHostsMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
TrustedProxyMiddleware.php:176

SilverStripe\Control\Middleware\TrustedProxyMiddleware->process(SilverStripe\Control\HTTPRequest, Closure)
HTTPMiddlewareAware.php:62

SilverStripe\Control\Director->SilverStripe\Control\Middleware\{closure}(SilverStripe\Control\HTTPRequest)
HTTPMiddlewareAware.php:65

SilverStripe\Control\Director->callMiddleware(SilverStripe\Control\HTTPRequest, Closure)
Director.php:370

SilverStripe\Control\Director->handleRequest(SilverStripe\Control\HTTPRequest)
HTTPApplication.php:48

SilverStripe\Control\HTTPApplication->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)

call_user_func(Closure, SilverStripe\Control\HTTPRequest)
HTTPApplication.php:66

SilverStripe\Control\HTTPApplication->SilverStripe\Control\{closure}(SilverStripe\Control\HTTPRequest)
HTTPMiddlewareAware.php:65

SilverStripe\Control\HTTPApplication->callMiddleware(SilverStripe\Control\HTTPRequest, Closure)
HTTPApplication.php:67

SilverStripe\Control\HTTPApplication->execute(SilverStripe\Control\HTTPRequest, Closure, )
HTTPApplication.php:47

SilverStripe\Control\HTTPApplication->handle(SilverStripe\Control\HTTPRequest)
cli-script.php:22

[2019-04-01 12:10:03][INFO]
[2019-04-01 12:10:03][INFO] Job paused at 2019-04-01 12:10:03

[sminnee]

In theory this could be fixed by changing this to "long" or something, but it's been a string since the SS3 version too, which is a big confusing.

https://github.com/silverstripe/addons.silverstripe.org/blob/master/app/src/dataobjects/Addon.php#L140

Perhaps nuke the index and reindex?

[robbieaverill]

I'm running 5.6.16 locally and not having this issue. Will try and find out exactly which version SSP is running to see if there's a discrepancy.

[robbieaverill]

I've tried recreating a new index. I can't see a way to clear the current index. That didn't help anyway, so I'll look at adjusting the type instead.

[sminnee]

"long" feels like a more correct type for an Int than "string", in any case.

[robbieaverill]

I've had the index reset, it's still evaluating to "long", and ES is trying to save it as a string. My local environment's index looks the same as the UAT environment, but the production environment looks different.

I don't know enough about this, for now I'm going to try setting the downloads field to be indexed as a string via config.

Addon::getElasticaMapping() has explicit casting for each field, but it looks like this isn't being used on production. The elastica module is probably being allowed to introspect the DB types and make its own decisions, which I assume the presence of the getElasticaMapping() method should prevent. I don't know why this happens on production and not on UAT (and local) which would explain the difference in the indexes.

[robbieaverill]

Here's a summary of where I'm at:

• Local and UAT works fine
• Production has downloads defined as long
• The code mapping is for downloads to be a string
• The error happens when trying to recreate the index with the index mapping we define, because a previous index spec already exists which it conflicts with

Failing call to set new index mapping:

Elastica\Transport\Guzzle->exec({"path":"ss_mysite\/Addon\/_mapping","method":"PUT","data":{"Addon":{"properties":{"name":{"type":"string"},"description":{"type":"string"},"type":{"type":"string"},"compatibility":{"type":"string"},"vendor":{"type":"string"},"tags":{"type":"string"},"released":{"type":"date"},"downloads":{"type":"string"},"readme":{"type":"string"}}}},"query":[],"contentType":"application\/json","connection":{"config":{"url":null,"bigintConversion":false},"host":"MYELASTICHOSTHERE","port":80,"path":null,"proxy":null,"transport":"AwsAuthV4","persistent":true,"timeout":5,"connections":[],"roundRobin":false,"log":false,"retryOnConflict":0,"username":null,"password":null,"aws_region":"ap-southeast-2","enabled":true}}, Array)
Request.php:193

Note "downloads":{"type":"string"} above.

Production ES index spec:

{
  "ss_mysite": {
    "aliases": {},
    "mappings": {
      "Addon": {
        "properties": {
          "SS_Published": {
            "type": "boolean"
          },
          "compatibility": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "description": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "downloads": {
            "type": "long"
          },
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "readme": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "released": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "tags": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "vendor": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1554338601480",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "qMeCpAhgR_SyptSOimjWZA",
        "version": {
          "created": "5060899"
        },
        "provided_name": "ss_mysite"
      }
    }
  }
}

UAT/local index spec:

{
  "ss_mysite": {
    "aliases": {},
    "mappings": {
      "Addon": {
        "properties": {
          "SS_Published": {
            "type": "boolean"
          },
          "compatibility": {
            "type": "text"
          },
          "description": {
            "type": "text"
          },
          "downloads": {
            "type": "text"
          },
          "name": {
            "type": "text"
          },
          "readme": {
            "type": "text"
          },
          "released": {
            "type": "date"
          },
          "tags": {
            "type": "text"
          },
          "type": {
            "type": "text"
          },
          "vendor": {
            "type": "text"
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1551313022147",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "wXdfr8TySb-R6KiVxVLUJw",
        "version": {
          "created": "5060899"
        },
        "provided_name": "ss_mysite"
      }
    }
  }
}

@stogj manually cleared the index for me and said that when he checked it again it still had an index definition. It should have responded with something like this:

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index",
        "resource.type": "index_or_alias",
        "resource.id": "addons",
        "index_uuid": "_na_",
        "index": "addons"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index",
    "resource.type": "index_or_alias",
    "resource.id": "addons",
    "index_uuid": "_na_",
    "index": "addons"
  },
  "status": 404
}

Nevertheless, he did something and I tried again. The reindex task then worked successfully, and when I refreshed to run it again (via browser) it reverted back to failing again.

It feels like something is interjecting and changing the index spec outside of the application layer (to me).

I'm going to try switching the mapping to a long explicitly instead.

[sminnee]

Cool. If this fixes it, it might be worth filing some upstream issues relating to changing index types.

[robbieaverill]

Alright well now I'm getting the inverse error (string vs long) on UAT from this change. At least I can reproduce the issue locally now (addons is my index name):

• curl -XDELETE http://localhost:9200/addons
• Run dev/tasks/Heyday-Elastica-ReindexTask to set initial index spec
• $ curl -s http://localhost:9200/addons/Addon/_mapping | jq .addons.mappings.Addon.properties.downloads (type: string)
• Edit Addon::getElasticaMapping() and change downloads from string to long
• Run dev/tasks/Heyday-Elastica-ReindexTask to update index spec (see error)

For whatever reason the ES infrastructure in production isn't clearing correctly, but when reindexing ES from the app layer it should delete the entire index including mappings. I'll see if I can adjust the module to do this.

[robbieaverill]

Ok I've raised the issue at WPP-Public/akqa-nz-silverstripe-elastica#19 and made a patch PR at #243, and have fixed this in our project with #243 so it deletes the existing index every time it goes to reindex it.

Tested on UAT and deployed to production, working. Happy to close this now.