Comments (3)
You should wrap wait_for
call into try/except
block and gracefully close the task.
https://github.com/aosabook/500lines/blob/master/crawler/code/crawling.py#L233 is a good place for catching at first glance.
from 500lines.
I am a bit confused. It seems as if you want to put a timeout there, while I'm talking about the crawling "get" (targeting the server).
It would seem the place where you suggest is that whenever somehow getting an item from the queue taking too long, then it would gracefully end.
Whereas I'm in the fetch
method (https://github.com/aosabook/500lines/blob/master/crawler/code/crawling.py#L175) ( trying to put the wait_for
). Is that still correct?
from 500lines.
I put it at both places, and that seems to solve some issues. But now whenever the queue is empty it will try to stop the worker, and it will throw a ERROR:asyncio:Task was destroyed but it is pending!
I catch it, but still I get 2 messages PER worker at the end of the script (not so nice for wanting to save the logging):
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<get() done, defined at /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/queues.py:160> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_release_waiter(<Future cancelled>)() at /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/tasks.py:333]>
from 500lines.
Related Issues (20)
- crawler can't work as expected HOT 2
- Is this repo still open for including new projects? HOT 2
- Simple Web Server -- web server vs web application
- Chapter 3.Cluster typo
- where can I find the html documents after running the crawl.py HOT 1
- What do these constraints mean in DBDB->physical->Storage ???
- Recommend my project HOT 2
- flow-shop initial seed
- Modeller: when a cube is scaling, the light is calculated eorror
- We are engaged in translating this book from English into zh-TW. HOT 3
- Why does this code crash HOT 1
- An unreasonable loop in 'crawler/code/crawling.py'
- Does the socket need to be created here?
- 教师信息管理系统 HOT 1
- Crawler stuck on async queue.get() HOT 1
- cluster-how to build and run the project
- Python interpreter frame scope creation bug?
- add what technology used to create the projects at the beginning of each chapters
- about pedometer
- 求中文版 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 500lines.