realslimshanky/spider-sense

A browser extension to monitor your spiders deployed on Scrapy Cloud.

Spider-Sense

A browser extension to monitor your spiders deployed on Zyte Scrapy Cloud.

How to install this extension from file?

• Firefox

How to install this extension from store?

• Firefox
• Chrome

(Latest updates) How to install this extension from source?

• Clone the repository
• Open about:debugging on Firefox, click on This Firefox and select Load Temporary Add-on... or open chrome://extensions/ on Chrome, enable Developer mode and select Load unpacked
• Select manifest.json file present in this repository
• You'll see a black 'Bolt' icon on the Extension/Add-on bar of respective browser
• Click on the icon to enable the popup and follow instructions

Please create issue if you find some, hope this extension helps your workflow :) - Shashank

spider-sense's Issues

Upgrade Chrome extension to use v0.2

Currently chrome extension uses v0.1, please update to use v0.2

A refactor is needed to resolve fickering of right-click menu

Currently, an entire div is replaced when data about a job is updated in the background. This result is flickering of the right-click menu something like this

Possible solution: Instead of updating entire div, update only the innerHTML of the specific row/field which needs updation.

Check feasibility for sending slack message from extension

Adding comments to jobs/spiders

A text/comment can be associated with the jobs in addition to fetched data about it's performance. This text/comment will be fedarated.

Status "finished" when the status in the Scrapy Cloud is "banned"

I've been using this extension some days and I really like it. Congrats!

However I found this behaviour a little annoying, as the appearing "status" in the extension differ from the "status" in the Scrapy Cloud. I didn't go through the code and I don't know if it's an API limitation, but it could be a good to have feature if it's feasible.

Reducing load delay of new jobs

There's a delay of atleast 2 seconds before the logs of new test jobs shows on the popup. This can be optimized.

Add requests to detailed stats

We need to add requests made by the dash job to detailed stats as well.

Extension not working on chrome

Upon entering API Key, it's showing Please enter valid key. The same works in firefox.

Need new designs for the entire extension

We need to redesign the extension so that it looks good and can improve the user experience.

Monitoring Project/Spider for updates

Monitoring dash for the Project/Spider for new test jobs and reporting on the extension for quick add.

Notifications for failed jobs or errors in outcome

A new feature request to show notification if a project has any project with errors or failed outcome. One per day.

Impement popup notification

Popup notification can be triggered upon reaching a certain item/error/log count or job done.

Fixing styles

Adding bootstrap or some other framework to fix styles.

Horizontal layout out of the view

When the spider names are too long, the horizontal row goes out of view, and the option to remove the job or get detail of the job through icons couldn't be used.

Refer to the image below. I have hidden the project IDs & spider names for obvious reasons.

We need something like horizontal scrolling for such cases when spider names are too long & they don't fit in the view.

PS: I am using the development version of the extension i.e. loading directly through the source code.

Alternative: Currently in such a case, I have to remove & re-install the extension to reset the view.

Updating Manifest file as v2 will not be accepted from next year onwards (i.e. from January 17, 2022)

Chromium support doc: https://blog.chromium.org/2020/12/manifest-v3-now-available-on-m88-beta.html

Chrome transition plan: https://developer.chrome.com/blog/mv2-transition/

Auto clean old jobs

Old jobs can be auto cleaned after X days. Autoclean can be enabled for individual jobs.