llonchj / scrapy-sentry Goto Github PK

Sentry component for Scrapy

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

scrapy-sentry's Introduction

scrapy-sentry

Logs Scrapy exceptions into Sentry

A python library that glues Sentry with Scrapy. Any spider errors will get pushed to Sentry. Please note that currently, exceptions anywhere else (e.g. the Scrapy pipeline) are not being reported to Sentry.

Requisites:

Sentry server

Installation

pip install scrapy-sentry

Setup

Add SENTRY_DSN and scrapy_sentry.extensions.Errors extension to your Scrapy Project settings.py.

Example:

# sentry dsn
SENTRY_DSN = 'http://public:[email protected]/1'
EXTENSIONS = {
    "scrapy_sentry.extensions.Errors":10,
}

Supported versions

This package works with Python 2.7, 3.4 and 3.5. It has been tested with Scrapy up to version 1.2.1.

scrapy-sentry's People

Contributors

Stargazers

Watchers

scrapy-sentry's Issues

How set log level

I have set scrapy log level = info in the settings.py, but spider still send debug to sentry server

Deprecation warning

python3.6/site-packages/scrapy_sentry/utils.py:7: ScrapyDeprecationWarning: Module `scrapy.conf` is deprecated, use `crawler.settings` attribute instead
  from scrapy.conf import settings

Capture exceptions outside of spiders

scrapy-sentry currently does not capture exceptions that happen outside the spider's on_error signal. This limits the usefulness of the extension. Luckily, It is pretty easy to fix as well. I currently have this in my Scrapy settings.py instead of configuring the extension:

from twisted.python import log
from raven import Client

client = Client(dsn=SENTRY_DSN)

def log_sentry(dictionary):
    if dictionary.get('isError') and 'failure' in dictionary:
        try:
            # Raise the failure
            dictionary['failure'].raiseException()
        except:
            # so we can capture it here.
            client.captureException()


log.addObserver(log_sentry)

It should be pretty easy to do the same in the extension, and possibly remove the on_error handling. @llonchj, if you agree, I'm happy to prepare a PR for that.

Above snippet is not my work, it was discovered by @elmkarami in http://stackoverflow.com/questions/25262765/handle-all-exception-in-scrapy-with-sentry and scrapy/scrapy#852.

WebService error

2013-04-13 03:34:50+0800 [scrapy.webservice.WebService] ERROR: ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted ERROR twisted INFO twisted (TCP Port 6080 Closed)

What I did:

# in settings.py
try:
    from settings_prod import *
except ImportError:
    pass

SENTRY_DSN = 'https://public_key:[email protected]/1'

import scrapy_sentry
scrapy_sentry.init(SENTRY_DSN)

then I ran scrapy crawl my_spider

Importing settings from scrapy.conf is deprecated

Should use crawler settings instead.

Deprecation warning

When running with Scrapy 1.2.1 the following warning is displayed

WARNING: .pyenv/versions/scrapy-2712/lib/python2.7/site-packages/scrapy_sentry/extensions.py:10: ScrapyDeprecationWarning: Module scrapy.log has been deprecated, Scrapy now relies on the builtin Python library for logging. Read the updated logging entry in the documentation to learn more. from scrapy import signals, log

Definitely not critical but wanted to pass the information along.