Elasticsearch not receiving data from scrapy

[jayzeng]

Elasticsearch not receiving data from scrapy

[ravikkr18]

Hi harakiri406,

I Got the exact same issue, i just changed the ELASTICSEARCH_TYPE "items" to something else and now it works perfectly.

[harakiri406]

Already tried many ELASTICSEARCH_TYPE (data, link, url) entries, nothing works. I could suspect ES not accepting certain data types, but then my tcpdump would at least show some package

[ravikkr18]

in my case it first collect few 100-150 items than send them all to ES one by one , you can see ES returned output on terminal.

[harakiri406]

I found the cause of my problem: my data. It was hard to find in the logs because they were so big. Issue can be closed ;

2017-08-03 09:03:42 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-03 09:03:42 [scrapy.core.engine] ERROR: Scraper close failure
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/scrapyelasticsearch/scrapyelasticsearch.py", line 99, in close_spider
    self.send_items()
  File "/usr/local/lib/python2.7/dist-packages/scrapyelasticsearch/scrapyelasticsearch.py", line 86, in send_items
    helpers.bulk(self.es, self.items_buffer)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 195, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 162, in streaming_bulk
    for bulk_actions in _chunk_actions(actions, chunk_size, max_chunk_bytes, client.transport.serializer):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 61, in _chunk_actions
    data = serializer.dumps(data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/serializer.py", line 50, in dumps
    raise SerializationError(data, e)
SerializationError: