Timeout error trying scrapy shell against dockerized web site

[scalingexcellence]

Timeout error trying scrapy shell against dockerized web site

[find4u]

I just added the following to the setting.py and hey presto
# Disable S3
AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""

Hope this helps someone else

Newbie to Py and Scrapy

Richard

[juananpe]

Answering to myself, it seems that the boto library is showing this error when it can't connect to an S3/AWS host. Just set the variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to an empty string and your are good to go:

$ export AWS_ACCESS_KEY_ID="" && export AWS_SECRET_ACCESS_KEY="" && scrapy shell http://web:9312/properties/property_000000.html

(I suppose that there is a more elegant way to bypass this error, but that line works for me :)

[lookfwd]

Thanks a lot @juananpe for the clarification. This refers to page 113 of the e-book, or page 92 of the printed book.

You are right on exporting AWS_* credentials. This is exactly what I do as well in settings.py for Chapter 5 and any other chapter. As you say - it's just annoying boto detail and a non-elegant workaround is fine.

Hopefully this won't be a very common problem. If you run your scrapy shell command from within ch05/properties directory, it should work fine because scrapy shell includes automatically settings.py. My guess is you ran scrapy shell from a top-level directory.

It's great clarification. Thanks a million!