roycehaynes/scrapy-rabbitmq

A RabbitMQ Scheduler for Scrapy

scrapy-rabbitmq's Issues

Connecting to remote server

How Can i Connect to the remote RabbitMQ
when i use following in the setting file it gives me error

RABBITMQ_CONNECTION_PARAMETERS = {'host': 'http://test:[email protected]', 'port': 5672}

host format :http://username:[email protected]/

Connection parameters not working

I have given the following in my scrapy settings.py file
RABBITMQ_CONNECTION_PARAMETERS = {'host': 'amqp://username:password@rabbitmqserver', 'port':5672}
But I am getting the following error:
raise exceptions.AMQPConnectionError(error)
pika.exceptions.AMQPConnectionError: [Errno 11003] getaddrinfo failed

How can I use rabbit MQ server with my credentials?

Scheduler close failure occurs while spider closes

Below is the traceback:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 288, in <lambda>
    dfd.addBoth(lambda _: slot.scheduler.close(reason))
TypeError: close() takes exactly 1 argument (2 given)

Wrong constant name in scheduler.py

Hi, I had en error when try to start my script with scrapy and scrapy-rabbitmq
`2016-04-12 14:34:06 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 73, in crawl
yield self.engine.open_spider(self.spider, start_requests)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- ---
File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 238, in open_spider
yield scheduler.open(spider)
File "/app/project/external_apps/scrapy_rabbitmq/scheduler.py", line 56, in open
if len(self.queue):
File "/app/project/external_apps/scrapy_rabbitmq/queue.py", line 54, in len
response = self.server.queue_declare(self.key, passive=True)
File "/usr/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 2329, in queue_declare
self._flush_output(declare_ok_result.is_ready)
File "/usr/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1181, in _flush_output
raise exceptions.ChannelClosed(method.reply_code, method.reply_text)
pika.exceptions.ChannelClosed: (404, "NOT_FOUND - no queue 'my_spider:requests' in vhost 'docker'")
2016-04-12 14:34:06 [twisted] CRITICAL: `

I set settings as in example. Queue with name RABBITMQ_QUEUE_NAME from settings in my rabbitMQ server will be created.
Queue with name "my_spider:request" wasn't created.

Issue while pushing request

Below is the traceback:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\scrapy\commands\runspider.py", line 89, in run
    self.crawler_process.start()
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 251, in start
    reactor.run(installSignalHandlers=False)  # blocking call
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1194, in run
    self.mainLoop()
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 1203, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "C:\Python27\lib\site-packages\twisted\internet\base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "C:\Python27\lib\site-packages\scrapy\utils\reactor.py", line 41, in __call__
    return self._func(*self._a, **self._kw)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 118, in _next_request
    self.crawl(request, spider)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 182, in crawl
    self.schedule(request, spider)
  File "C:\Python27\lib\site-packages\scrapy\core\engine.py", line 188, in schedule
    if not self.slot.scheduler.enqueue_request(request):
  File "build\bdist.win-amd64\egg\scrapy_rabbitmq\scheduler.py", line 69, in enqueue_request

  File "build\bdist.win-amd64\egg\scrapy_rabbitmq\queue.py", line 62, in push

  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 572, in basic_publish
    (properties, body), False)
  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 1159, in _send_method
    self.connection.send_method(self.channel_number, method_frame, content)
  File "C:\Python27\lib\site-packages\pika\adapters\blocking_connection.py", line 274, in send_method
    self._send_method(channel_number, method_frame, content)
  File "C:\Python27\lib\site-packages\pika\connection.py", line 1509, in _send_method
    length = len(content[1])
exceptions.TypeError: object of type 'Request' has no len()

Usage documentation

it would be great to understand how this works