Feature Request | 功能需求

[my8100]

Feature Request | 功能需求

[JerryChenn07]

This project is pretty good. If you can join Timed task, it will be a more complete project.

Come on!

[wymen2018]

1. You can select multiple crawlers when you want to set a timed task. You can select multiple crawlers to start at each time period after selecting the time point.
   Imagine that I have 100 crawlers in a project. This feature does provide convenience and management.
   2, as above, because there are a hundred crawlers, so can you have the function of custom label, that is, I can use the label screen and view the running status of the specified category of crawlers, and timed tasks
   Thanks!

[my8100]

Or modify the code below to vm.versions = ['default: the latest version'].concat(obj.versions);

scrapydweb/scrapydweb/templates/scrapydweb/schedule.html

Line 826 in 560e998

vm.versions = obj.versions;

[lidonghe]

Or modify the code below to vm.versions = ['default: the latest version'].concat(obj.versions);
scrapydweb/scrapydweb/templates/scrapydweb/schedule.html

Line 826 in 560e998

                         vm.versions = obj.versions;

It works, thanks

[my8100]

add a image of docker, please

@seozed
Check out the docker image created by @luzihang123.

my8100/logparser#15 (comment)

[my8100]

@tuchief

2、Online packaging and deployment

Have you tried the Projects > Deploy page?

[tuchief]

I know what you said, but this method should be manually packaged into egg, and then uploaded, and can not directly package the source project into egg, automatically upload it?

[my8100]

I know what you said, but this method should be manually packaged into egg, and then uploaded, and can not directly package the source project into egg, automatically upload it?

Ok, I would try to figure out a better way to eggify local projects.

[tuchief]

You can refer to https://github.com/Gerapy/Gerapy

[my8100]

v0.9.9: Add auto eggifying

[tuchief]

Wow, the response is fast! Expecting the same timing tasks

[my8100]

You can select multiple crawlers when you want to set a timed task. You can select multiple crawlers to start at each time period after selecting the time point.
Imagine that I have 100 crawlers in a project. This feature does provide convenience and management.

You mean there are 100 spiders in a project and you want to schedule some of them to run periodically?

as above, because there are a hundred crawlers, so can you have the function of custom label, that is, I can use the label screen and view the running status of the specified category of crawlers, and timed tasks

What about labeling some related jobs with a specific jobid?

[wymen2018]

You mean there are 100 spiders in a project and you want to schedule some of them to run periodically?

No, Simultaneously select multiple spiders to set one timed task.

What about labeling some related jobs with a specific jobid?

Labels will be better because they can be more visualized according to their own classification

[my8100]

OK, I would take it into account when implementing this feature. Thanks for your advice!

[my8100]

v1.0.0rc1: Add Email Notice, with multi-triggers provided, including:

• ON_JOB_RUNNING_INTERVAL

• ON_JOB_FINISHED

• When reaching the threshold of specific kind of log: ['CRITICAL', 'ERROR', 'WARNING', 'REDIRECT', 'RETRY', 'IGNORE'], in the meanwhile, you can ask ScrapydWeb to stop/forcestop current job automatically.

Get it via the pip install scrapydweb==1.0.0rc1 command.

Email sample:

[LWsmile]

ERROR in utils: !!!!! ConnectionError HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jobs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6111cae9b0>: Failed to establish a new connection: [Errno 111] Connection refused',))

[my8100]

ERROR in utils: !!!!! ConnectionError HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jobs (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6111cae9b0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Please new an issue with details.

[lidonghe]

timed task +1

[my8100]

v1.2.0: Support Timer Tasks to schedule a spider run periodically

[lidonghe]

While adding a timer task, I have to choose a version, but actually I only want to use the latest version so that if the project is updated, I don't need to update my task accordingly.

[my8100]

OK, it would be fixed in the future release.
For the time being, go to the Jobs page and click either multinode or Start button to work around.

[my8100]

The modification has been commited.

[heave-Rother]

For Logs categorization, can the same spiders distributed on different scrapyd be aggregated? @my8100

[my8100]

@heave-Rother
For the time being, you can switch to a specific page of the neighboring node
with the help of Node Scroller and Node Skipping.

If that cannot satisfy your need, could you draw a picture to show me your idea?

[heave-Rother]

@my8100 ok,

[my8100]

@heave-Rother

1. What would you use the checkboxes in the drop-down list for?
   Did you notice the checkboxes for nodes in the Servers page?
2. Stats aggregation would be followed up in PR #72.

[heave-Rother]

@my8100
yes, I want to choose the server I want to join the statistics.

[my8100]

@heave-Rother
I see. Thanks for your suggestion.

[seozed]

add a image of docker, please

[devxiaosong]

感谢作者，这是我找到的最好的爬虫集群操作平台。提几个需求：
1，给每个node加描述，方便自己看。
2，通过手机短信发送报警信息。
3，如何支持基于scrapy-redis的分布式爬虫的配置、启动？

[my8100]

@devxiaosong
Replied in #107.

[Tobeyforce]

Please add a short tutorial on how to switch from the flask development server to a production server with https enabled using letsencrypt. It would be much appreciated.

[my8100]

Please add a short tutorial on how to switch from the flask development server to a production server with https enabled using letsencrypt. It would be much appreciated.

Please try it out and share your result.

scrapydweb/scrapydweb/run.py

Line 116 in 6b9663b

logger.info("For running Flask in production, check out http://flask.pocoo.org/docs/1.0/deploying/")

scrapydweb/scrapydweb/default_settings.py

Lines 82 to 92 in 6b9663b

	############################## ScrapydWeb #####################################
	# The default is False, set it to True and add both CERTIFICATE_FILEPATH and PRIVATEKEY_FILEPATH
	# to run ScrapydWeb in HTTPS mode.
	# Note that this feature is not fully tested, please leave your comment here if ScrapydWeb
	# raises any excepion at startup: https://github.com/my8100/scrapydweb/issues/18
	ENABLE_HTTPS = False
	# e.g. '/home/username/cert.pem'
	CERTIFICATE_FILEPATH = ''
	# e.g. '/home/username/cert.key'
	PRIVATEKEY_FILEPATH = ''

[Jasonjk3]

可以在页面上动态添加/删除 scrapyd 节点吗

[my8100]

可以在页面上动态添加/删除 scrapyd 节点吗

Editing Scrapyd servers via GUI is not supported.