Comments (9)
Just wanted to point out to anyone having issues with SSL that scrapydweb uses assert os.path.isfile()
to check the certificate and key files which only returns true
if the path exists and is a file and you have (at least) read permissions.
Popular certificate aggregators like LetsEncrypt chmod the certificate as root only read/write.
That means you will need to run scrapydweb as root in order for the certificate to parse.
from scrapydweb.
The below assert line could be commented out.
scrapydweb/scrapydweb/utils/check_app_config.py
Lines 84 to 91 in 7a3b81d
from scrapydweb.
Maybe adding config paths for the certificate files and optionally adding them here:
https://github.com/my8100/scrapydweb/blob/master/scrapydweb/run.py#L92
Would be enough?
You might also need to require pyopenssl
(pip install pyopenssl
)
app.run with ssl_context:
app.run(ssl_context=('cert.pem', 'key.pem'))
from scrapydweb.
This works for me, and no need to install pyopenssl. Could you try and tell me your result?
context = ('/path/to/the/cert.pem', '/path/to/the/cert.key')
app.run(host=app.config['SCRAPYDWEB_BIND'], port=app.config['SCRAPYDWEB_PORT'], ssl_context=context)
from scrapydweb.
It seems to work :)
from scrapydweb.
What's the file type of your certificate? (Mine is '.pem')
from scrapydweb.
Fixed in v1.1.0: Now you can set up ENABLE_HTTPS, CERTIFICATE_FILEPATH, and PRIVATEKEY_FILEPATH to enable HTTPS mode.
from scrapydweb.
Thanks @my8100 👍
I can't think of a clean way to fix this issue other than acknowledging the file exists manually and commenting out that line as you say, would you agree?
- It should rightly fail without permission to read the cert/key, the
assert os.path.isfile()
is the correct method. - Non-root users shouldn't change the permission of the cert/key as it is set to root as a layer of security
- Installing scrapyd, logparser and scrapydweb shouldn't be installed as root as a layer of security
- Running scrapydweb as a root user should also be avoided as a layer of security
Kind of leaves you in no man's land.
Thanks for a great tool by the way @my8100 and hello from Australia!
from scrapydweb.
@jwjhdev
As the below comment said, HTTPS mode was not fully tested.
Thank you so much for the feedback.
scrapydweb/scrapydweb/default_settings.py
Lines 82 to 92 in 7a3b81d
from scrapydweb.
Related Issues (20)
- Not able to see stats section of the job HOT 1
- scrapydweb failed to run on python 3.8 HOT 5
- 启动报错:sqlite3.OperationalError: no such table: metadata HOT 13
- Is it possible to run multiple spider at the same time in a tmux machine with scrapydweb automatically
- items Oops! Something went wrong. HOT 1
- scrapydweb fresh install won't run HOT 8
- APScheduler 3.10 causing 500 errors HOT 2
- How to Change Timezone of scrapydweb? HOT 3
- Which scrapyd image you use? HOT 2
- Clean install on clean Ubuntu VM. Whatever I do it is not working. HOT 2
- Docker compose scrapdweb with scrapyd the log url use docker name
- Processes dont stop after finishing HOT 1
- v1.4.1 submit cron job can't run HOT 1
- ('Connection aborted.', timeout('timed out',))
- ERROR: Package 'scrapydweb' requires a different Python: HOT 4
- Error while installing scrapydweb HOT 2
- spiders are closed but showing as running/warning in the tasks page
- web界面可以使用中文吗? HOT 1
- DATABASE_URL配置连接域名:端口的mysql失败 HOT 2
- Jobs are killed without a clear reason
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapydweb.