Comments (7)
The .gz files are shown correctly if I open them in scrapyd's own web interface, while they fail in scrapydweb. To reproduce, you can just compress a log file by hand in scrapyd's directory.
Ok, I got the following result, would fix the problem ASAP.
from scrapydweb.
@waldner
Add Items page, and support extensions ['.log', '.log.gz', '.gz', '.txt', ''] for locating scrapy log in v0.9.5
from scrapydweb.
I suppose it's perfectly possible to use a custom logger that writes directly to a compressed file (or a custom feed exporter to do the same with items), although in my case they get compressed by a custom process after scrapy has run.
from scrapydweb.
The barebones web interface of scrapyd can display compressed (eg, .gz) log and item files, while scrapydweb seems not to.
Hi, @waldner
The items page would be added in future updates.
But I couldn't find the "compressed (eg, .gz) log" you mentioned, could you please upload a screenshoot?
Currently, the utf8 page is compressed in gzip, and you can get the original log (text/plain) via the Source link.
Also, the logs page is available at Logs/Directory.
from scrapydweb.
Sure, for example see this:
The .gz files are shown correctly if I open them in scrapyd's own web interface, while they fail in scrapydweb. To reproduce, you can just compress a log file by hand in scrapyd's directory.
from scrapydweb.
Great! It even works when clicking on the log in the job list view, whereas scrapyd's doesn't (you have to navigate to the actual folders). Thanks!
from scrapydweb.
Great! It even works when clicking on the log in the job list view, whereas scrapyd's doesn't (you have to navigate to the actual folders). Thanks!
@waldner
How do you schedule a spider to run and log in .log.gz format? It seems that only -d setting=LOG_FILE=logs/demo/test/2018-10-12T05_44_53.txt
works, whereas .log.gz and .gz would cause "ContentDecodingError ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check',))".
Or do you just compress a log file by hand after the crawling finished? ("It even works when clicking on the log in the job list view" only refers to the Finished part?)
from scrapydweb.
Related Issues (20)
- scrapydweb failed to run on python 3.8 HOT 5
- 启动报错:sqlite3.OperationalError: no such table: metadata HOT 13
- Is it possible to run multiple spider at the same time in a tmux machine with scrapydweb automatically
- items Oops! Something went wrong. HOT 1
- scrapydweb fresh install won't run HOT 8
- APScheduler 3.10 causing 500 errors HOT 2
- How to Change Timezone of scrapydweb? HOT 3
- Which scrapyd image you use? HOT 2
- Clean install on clean Ubuntu VM. Whatever I do it is not working. HOT 2
- Docker compose scrapdweb with scrapyd the log url use docker name
- Processes dont stop after finishing HOT 1
- v1.4.1 submit cron job can't run HOT 1
- ('Connection aborted.', timeout('timed out',))
- ERROR: Package 'scrapydweb' requires a different Python: HOT 4
- Error while installing scrapydweb HOT 2
- spiders are closed but showing as running/warning in the tasks page
- web界面可以使用中文吗? HOT 1
- DATABASE_URL配置连接域名:端口的mysql失败 HOT 2
- Jobs are killed without a clear reason HOT 6
- When i visiting items page i got "Oops! Something went wrong.". HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapydweb.