scrapinghub/varanus

A command line spider monitoring tool

Scrapinghub command line client

shub is the Scrapinghub command line client. It allows you to deploy projects or dependencies, schedule spiders, and retrieve scraped data or logs without leaving the command line.

Requirements

• Python >= 3.6

Installation

If you have pip installed on your system, you can install shub from the Python Package Index:

pip install shub

Please note that if you are using Python < 3.6, you should pin shub to 2.13.0 or lower.

We also supply stand-alone binaries. You can find them in our latest GitHub release.

Documentation

Documentation is available online via Read the Docs: https://shub.readthedocs.io/, or in the docs directory.

varanus's Issues

requests log is always presence

Request log is useful if we want to inspect some details, but it's better to disable it by default, and only show it if debug mode is on.

Suggestions for the graph layout

In the image below, we typically have this type of graph:

In order for the graphs to be more easily understandable, here are two (2) suggestions:

1. Make the graphs left-to-right by default where the newer ones are on the right. Another way to look at this would be that the job keys would be in ascending order.
2. Instead of a vertically-stacked graph, could there be perhaps an option to have a horizontally-stacked like the one below? This will make eye-ball comparing a lot easier since
   the bars all start with with the same base zero. In other words, it's very easy to see which bars is higher than the other in this layout.
   

Images

`items` and `pages` are always 0 for the `job` command

The items, pages, items_per_min, pages_per_min (and also maybe some other fields) are not populated properly.

The root cause is we populate MetaJob from the job's metadata, but those fields are not available in metadata.