你好,您的项目虽然说每个视频用一个线程去抓取,但是每个视频,只抓取到一部分二进制文件后,便出现了异常,有什么好的办法可以将每个视频都完整的抓取下来吗。部分异常信息如下:
Exception` in thread Thread-47:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 520384 out of 47830612 bytes
Exception in thread Thread-1:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 532059 out of 13004076 bytes
Exception in thread Thread-18:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 585460 out of 6128527 bytes
当前下载进度:---------------->>>>>>>> 6.47%Exception in thread Thread-48:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 582540 out of 24403607 bytes
Exception in thread Thread-36:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 532065 out of 10005207 bytes
Exception in thread Thread-35:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 532058 out of 49727052 bytes
Exception in thread Thread-40:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 586084 out of 62159002 bytes
当前下载进度:---------------->>>>>>>> 6.50%Exception in thread Thread-7:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 532063 out of 20505701 bytes
Exception in thread Thread-6:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 532065 out of 61492854 bytes
Exception in thread Thread-46:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 527684 out of 14292045 bytes
当前下载进度:---------------->>>>>>>> 6.53%Exception in thread Thread-2:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 586084 out of 10502982 bytes
Exception in thread Thread-5:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Users/yangqihua/Documents/project/python_project/spider_smooc/filedeal/file_downloader.py", line 24, in run
urllib.urlretrieve(fileurl,filepath, self.Schedule)#下载文件
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 98, in urlretrieve
return opener.retrieve(url, filename, reporthook, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 289, in retrieve
"of %i bytes" % (read, size), result)
ContentTooShortError: retrieval incomplete: got only 586087 out of 9053251 bytes