maicius / qqzonemood Goto Github PK

QQZone mood spider and analysis. QQ空间多线程爬虫和数据挖掘。提供线上服务，扫码登陆即可自动爬取和分析数据，还有网易云年度报告风格的数据展示；使用docker-compose打包程序，方便部署；额外提供QQ空间抽奖小程序。

Home Page: http://app.xiaomaidong.com

License: MIT License

Python 79.09% HTML 11.19% CSS 0.49% JavaScript 9.14% Dockerfile 0.09%

qzone qqzone qq spider mood thread flask docker slides slider

qqzonemood's Introduction

QQZoneMood

体验地址：

QQ交流群

群号：707583419

如果在使用中遇见bug，或者关于项目有什么想法，欢迎在群中交流

简介

多线程抓取QQ空间说说内容并进行分析，提供基于Flask和avalon.js的web配置界面，以及配套的测试用例
基于React Slides的数据展示页面，风格模仿网易云年度报告。子项目地址:QQZoneSwipper
提供抽奖小程序，可以根据指定说说的点赞或评论信息进行随机抽奖（可单独打包为windows下的命令行程序）
反向计算用户在每个时间点的好友数量，并可动态展示用户好友的增加过程，参考视频

系统架构图

部署注意事项

在本地分别启动后端和前端，端口号分别默认为5000和3000
修改 QQZoneMood/src/web/static/constant.js中QQ_FRONT_LOCATION变量为前端地址,如

http://localhost:3000
修改 QQZoneSwipper/src/utils/constant.js中SERVER_HOST变量为后端地址，如：

http://localhost:5000

注意url末尾都不要加斜杠（/）

docker版简介

本项目将网页配置版本打包为docker（以下简称docker版），本版本用于部署到线上
此外，还有大量已完成的功能（爬取好友的动态、爬取图片等）未进行整合
docker版主要包括以下功能：

1.配置用户名、QQ号、校验码

2.根据1中配置获取QQ空间的动态数量和好友基本信息

3.数据可视化
运行方式(请确保已经安装了docker和docker-compose)：

1.git clone https://github.com/Maicius/QQZoneMood.git

2.将 QQZoneMood/src/web/static/constant.js中QQ_FRONT_LACTION 修改为自己的前端项目地址（本地默认地址为localhost:3000）

3.cd QQZoneMood & docker-compose up -d

4.下载前端子项目

git clone https://github.com/Maicius/QQZoneSwipper

5.将 QQZoneSwipper/src/utils/constant.js中的SERVER_HOST修改为服务器的地址（本地默认地址为localhost:80）

6.启动前端子项目，请确保已安装了npm

cd QQZoneSwipper & npm install --save & npm run start

7.浏览器里输入地址: http://localhost:80
关于校验码

相当于用户在该网站的密码，为了安全请不要与QQ密码相同。设置校验码的目的是为了保证用户在该网站上只能查看自己的数据。

开发版运行方式

0.下载

git clone https://github.com/Maicius/QQZoneMood.git

cd QQZoneMood
1.安装依赖

pip3 install -r requirements.txt

（不建议使用conda，因为sha1等库在conda的源中没有）
2.修改配置文件

修改userinfo.json.example为文件userinfo.json，并填好QQ号、QQ密码、保存数据用的文件名前缀；

[可选]修改需要爬取的好友的QQ号和保存数据用的文件名前缀

3.__init__函数参数说明，请根据需要修改（所有参数都有默认值，即使不修改任何参与也能运行）

  def __init__(self, use_redis=False, debug=False, mood_begin=0, mood_num=-1, stop_time='-1',
           download_small_image=False, download_big_image=False,
           download_mood_detail=True, download_like_detail=True, download_like_names=True, recover=False,
           cookie_text=None, from_web=False, username='', nickname='', no_delete=True, pool_flag='127.0.0.1'):

  :param use_redis: If true, use redis and json file to save data, if false, use json file only.
  :param debug: If true, print info in console
  :param mood_begin: 开始下载的动态序号，0表示从第0条动态开始下载
  :param mood_num: 下载的动态数量，最好设置为20的倍数
  :param stop_time: 停止下载的时间，-1表示全部数据；注意，这里是倒序，比如，stop_time="2016-01-01",表示爬取当前时间到2016年1月1日前的数据
  :param recover: 是否从redis或文件中恢复数据（主要用于爬虫意外中断之后的数据恢复）。注意!!!此功能在多线程中不可用！！！
  :param download_small_image: 是否下载缩略图，仅供预览用的小图，该步骤比较耗时，QQ空间提供了3中不同尺寸的图片，这里下载的是最小尺寸的图片
  :param download_big_image: 是否下载大图，QQ空间中保存的最大的图片，该步骤比较耗时
  :param download_mood_detail:是否下载动态详情
  :param download_like_detail:是否下载点赞的详情，包括点赞数量、评论数量、浏览量，该数据未被清除
  :param download_like_names:是否下载点赞的详情，主要包含点赞的人员列表，该数据有很多都被清空了
  :param from_web: 表示是否来自web接口，如果为True，将该请求来自web接口，则不会读取配置文件
  :param username: 在web模式中，传递过来的用户QQ号
  :param nickname: 在web模式中，传递过来的用户昵称
  :param no_delete: 是否在redis中缓存数据，如果为True,则不会删除，如果为False，则设置24小时的缓存时间
  :param pool_flag: redis的连接池host，因为docker中host与外部不同，所以在启动程序时会自动判断是不是处于docker中

4.运行flask服务器

python3 src/web/server.py
5.其它程序入口可以参考test中测试用例

已实现功能

1.基本功能

这部分主要是获取数据和进行基本的统计分析

QQ空间动态爬取，包括：
1. 所有说说信息
2. 每条说说的详细信息（比1中的信息更全面，1中数据只显示每条说说的前10个评论）
3. 每条说说的点赞人列表
4. 更加详细的点赞人列表（3中获取的数据有很多被清空了，这里能稳定获取到点赞的人数量、浏览量和评论量）
5. 所有说说的图片（可选择是下载大图、缩略图还是都下载）
QQ空间好友基本信息爬取
1. 好友基本信息
2. 共同好友数量
3. 共同群组
4. 添加好友时间(可计算出用户在每个时间点的好友数量)
数据分析
1. 数据清洗，将所有抓取的信息清洗为excel或csv
2. 包括各种点赞排行、评论排行、发送时间统计等
3. 历史说说一览表
数据可视化

包括各种词云图、关系图
Web配置界面

使用Flask + avalon.js + echarts.js 搭建的简易web界面，为普通用户提供一个快速获取数据的方法

可视化爬虫过程

可视化用户数据、历史说说记录

2.衍生功能（这部分功能的代码没有完全添加到项目里）

QQ空间动态情感检测

基于百度自然语言处理API,可免费使用
QQ空间照片评分

基于Google NIMA模型
QQ空间好友关系演变图

戳这里查看视频演示

项目结构

说明如下：

resource:存放数据文件（不包括web中的静态资源）

src-spider:包括四个爬虫类和一个入口

BaseSpider(object): 爬虫基类，初始化各种变量和提供基础接口，统一管理爬虫的headers、数据的加载和存储
QQZoneSpider(BaseSpider):爬取QQ空间动态的主要逻辑，包括各种url的构建
QQZoneFriendSpider(QQZoneSpider): 爬取用户的好友基本信息和共同群组，计算用户在各个时间段的好友数量
QQZoneFriendMoodSpider(QQZoneSpider):爬取用户指定好友的动态
main: 程序入口，为web程序提供爬虫API

src-analysis:

QQZoneAnalysis: 数据清洗，将爬虫得到的原始数据清洗为excel形式，并做简单的数据统计和分析
Average: 计算平均评论量、点赞量、浏览量等数据
SentimentClassify: 调用百度人工智能API进行情感分类
TrainMood：已废弃，以前计划用来对文本内容分类等等

src-visual:

CreateGexf: 将用户好友数据生成Gephi软件可以接受的数据格式以进行聚类

src-web：网站模块

src-web-entity: 实体类
static: 静态资源，外部引用的包主要使用cdn
templates：网页

其它说明

python版本：3.6
模拟登陆时若出现图形验证码，可以在点击登陆后设置5秒暂停，以手动完成验证
数据存储方式：
目前提供了两种存储方式的接口（通过Spider中use_redis参数进行配置）:
1. 存储到json文件中
2. 存储到redis数据库中
  如果安装了redis，建议存储到redis中
  关于redis的安装和配置，请自行搜索
  Redis使用中常见问题可以参考这篇博客:Redis 踩坑笔记
注意：

程序提供了三种登陆的接口，分别是扫码登录、cookie登录、账号密码模拟登录，默认使用扫码登录其中账号密码模拟登录依赖于selenium，需要手动下载chrome driver和chrome浏览器
请注意版本匹配，可以查看这篇博客：
selenium之 chromedriver与chrome版本映射表（更新至v2.32）
运行结果例图：

QQ动态关键字词云

QQ空间说说按点赞和评论数分类图

手机端运行截图

使用Gephi软件生成的好友关系图

qqzonemood's People

Contributors

Stargazers

Watchers

Forkers

candythinking feisuweb staranddream xiangzisaisai zhouxiaoxu yuzhoustar itachigiotto mrnestor hustersyy anthony-lu lonelymito chaomas daydaystudylab youkilee amaranth0203 kevinyzy starrynight1995 liminxufen linprophet csliangdu zhixialang xiangyuwei id88 johnwongchn luoyhcode constantine-harris binandyan easy0ne hhy5277 xiaodin1 zeng1793 duanxiaoer shimmerming wangjunhao999 dingmengxiang dtyhero lxl0928 beifenku danielislearning cdanlover starhacker kenshinji kingking888 imdexian liushiyi1027 liaokongvfx shanghaobo haohanlin 223222981 343695222 lemon2ml quan0191 megetzz 2605759123 bigcatdi cqulun123 zivivan gavingu0 alex66668888 flyher flottant xunzh2000 hedou vincent-jun smeow0119 hlyu368 pengshistc shijingyu lennyniu zhongyinhei joshuap233 barryzm xgdyp ethan5072 foursmallleaves dexfire wohenlei jakejie163 xiongjunhan 1wtctw xbyl1234 cwzsquare cutano baisheng winniekun shitianshiwa quart233 syalsyyh kankan1987 xieyuanhuata wutong-shuyu huang-kai233 zz412000428 moxiaoyuancl ishelo lxngoddess5321 baldmonster33 yaosi-ym dtimor mintcyk

qqzonemood's Issues

多线程会遗漏数据

使用默认的线程数(10)跑程序的时候，某些线程抓取失败，导致爬取的数据不全(QQ 空间说说)，请问楼主这个怎么解决呢

imread/lib/_jpeg.cpp(10): fatal error C1083: Cannot open include file: 'jpeglib.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe' failed with exit status 2

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/
Collecting imread
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/4e/7e/52603151a6746a229f80b3d22a08e468dce9bcf723728a3b99d050372e86/imread-0.7.1.tar.gz
Requirement already satisfied: numpy in d:\program files\python37\lib\site-packages (from imread) (1.17.2)
Installing collected packages: imread
Running setup.py install for imread: started
Running setup.py install for imread: finished with status 'error'
Complete output from command "D:\Program Files\Python37\python.exe" -u -c "import setuptools, tokenize;file='C:\Users\Administrator\AppData\Local\Temp\pycharm-packaging\imread\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\Administrator\AppData\Local\Temp\pip-record-jldkfk24\install-record.txt --single-version-externally-managed --compile:
'pkg-config' 不是内部或外部命令，也不是可运行的程序
或批处理文件。
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.7
creating build\lib.win-amd64-3.7\imread
copying imread\ijrois.py -> build\lib.win-amd64-3.7\imread
copying imread\imread.py -> build\lib.win-amd64-3.7\imread
copying imread\imread_version.py -> build\lib.win-amd64-3.7\imread
copying imread\special.py -> build\lib.win-amd64-3.7\imread
copying imread_init_.py -> build\lib.win-amd64-3.7\imread
creating build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_bmp.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_error.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_formats.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_imread.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_imread_from_blob.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_imsave.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_jpeg.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_png.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_rois.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_tiff.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_webp.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests\test_xcf.py -> build\lib.win-amd64-3.7\imread\tests
copying imread\tests_init_.py -> build\lib.win-amd64-3.7\imread\tests
creating build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\0186-0099.roi -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\1_webp_a.webp -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\arange512_16bit.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\bit1.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\diag.xcf -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\error.jpg -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\error.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\error.tif -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\error.unknown -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\error.webp -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\ghsystem_flame.jpg -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\good -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\good.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\mono.tif -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\palette_zero.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\py-installer-indexed.bmp -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\rgba.png -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\rois.zip -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\stack.tiff -> build\lib.win-amd64-3.7\imread\tests\data
copying imread\tests\data\star1.bmp -> build\lib.win-amd64-3.7\imread\tests\data
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building 'imread._imread' extension
creating build\temp.win-amd64-3.7
creating build\temp.win-amd64-3.7\Release
creating build\temp.win-amd64-3.7\Release\imread
creating build\temp.win-amd64-3.7\Release\imread\lib
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=MahotasImread_PyArray_API_Symbol -DIMREAD_EXCLUDE_WEBP=1 "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\lib\site-packages\numpy\core\include" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\winrt" /EHsc /Tpimread/_imread.cpp /Fobuild\temp.win-amd64-3.7\Release\imread/_imread.obj
_imread.cpp
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/base.h(95): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/base.h(212): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/base.h(221): warning C4146: unary minus operator applied to unsigned type, result still unsigned
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/base.h(221): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(20): warning C4996: 'close': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _close. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(457): note: see declaration of 'close'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(23): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(23): warning C4996: 'read': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _read. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(521): note: see declaration of 'read'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(26): warning C4267: 'argument': conversion from 'size_t' to 'long', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(26): warning C4996: 'lseek': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _lseek. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(501): note: see declaration of 'lseek'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(27): warning C4996: 'lseek': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _lseek. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(501): note: see declaration of 'lseek'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(28): warning C4996: 'lseek': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _lseek. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(501): note: see declaration of 'lseek'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(32): warning C4267: 'argument': conversion from 'size_t' to 'unsigned int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/file.h(32): warning C4996: 'write': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _write. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(556): note: see declaration of 'write'
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/numpy.h(75): warning C4244: 'return': conversion from 'npy_intp' to 'int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib/numpy.h(114): warning C4244: 'argument': conversion from 'npy_intp' to 'int', possible loss of data
imread/_imread.cpp(100): warning C4267: 'argument': conversion from 'size_t' to 'const int', possible loss of data
imread/_imread.cpp(104): warning C4996: 'open': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _open. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(514): note: see declaration of 'open'
imread/_imread.cpp(134): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
imread/_imread.cpp(163): warning C4800: 'ImageFormat *': forcing value to bool 'true' or 'false' (performance warning)
imread/_imread.cpp(297): warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'unsigned int', possible loss of data
imread/_imread.cpp(297): warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'const unsigned int', possible loss of data
imread/_imread.cpp(286): warning C4996: 'open': The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name: _open. See online help for details.
C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt\corecrt_io.h(514): note: see declaration of 'open'
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=MahotasImread_PyArray_API_Symbol -DIMREAD_EXCLUDE_WEBP=1 "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\lib\site-packages\numpy\core\include" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\winrt" /EHsc /Tpimread/lib/formats.cpp /Fobuild\temp.win-amd64-3.7\Release\imread/lib/formats.obj
formats.cpp
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(95): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(212): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4146: unary minus operator applied to unsigned type, result still unsigned
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=MahotasImread_PyArray_API_Symbol -DIMREAD_EXCLUDE_WEBP=1 "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\lib\site-packages\numpy\core\include" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\winrt" /EHsc /Tpimread/lib/numpy.cpp /Fobuild\temp.win-amd64-3.7\Release\imread/lib/numpy.obj
numpy.cpp
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(95): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(212): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4146: unary minus operator applied to unsigned type, result still unsigned
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\numpy.h(75): warning C4244: 'return': conversion from 'npy_intp' to 'int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\numpy.h(114): warning C4244: 'argument': conversion from 'npy_intp' to 'int', possible loss of data
imread/lib/numpy.cpp(8): warning C4244: 'initializing': conversion from 'npy_intp' to 'int', possible loss of data
imread/lib/numpy.cpp(8): warning C4244: 'initializing': conversion from 'npy_intp' to 'const int', possible loss of data
imread/lib/numpy.cpp(9): warning C4244: 'initializing': conversion from 'npy_intp' to 'int', possible loss of data
imread/lib/numpy.cpp(9): warning C4244: 'initializing': conversion from 'npy_intp' to 'const int', possible loss of data
imread/lib/numpy.cpp(14): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
imread/lib/numpy.cpp(17): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=MahotasImread_PyArray_API_Symbol -DIMREAD_EXCLUDE_WEBP=1 "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\lib\site-packages\numpy\core\include" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\winrt" /EHsc /Tpimread/lib/_bmp.cpp /Fobuild\temp.win-amd64-3.7\Release\imread/lib/_bmp.obj
_bmp.cpp
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(95): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(212): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4146: unary minus operator applied to unsigned type, result still unsigned
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\tools.h(23): warning C4267: 'initializing': conversion from 'size_t' to 'int', possible loss of data
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DPY_ARRAY_UNIQUE_SYMBOL=MahotasImread_PyArray_API_Symbol -DIMREAD_EXCLUDE_WEBP=1 "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\include" "-ID:\Program Files\Python37\lib\site-packages\numpy\core\include" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\winrt" /EHsc /Tpimread/lib/_jpeg.cpp /Fobuild\temp.win-amd64-3.7\Release\imread/lib/_jpeg.obj
_jpeg.cpp
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(95): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(212): warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning)
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4146: unary minus operator applied to unsigned type, result still unsigned
c:\users\administrator\appdata\local\temp\pycharm-packaging\imread\imread\lib\base.h(221): warning C4267: 'argument': conversion from 'size_t' to 'int', possible loss of data
imread/lib/_jpeg.cpp(10): fatal error C1083: Cannot open include file: 'jpeglib.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2

----------------------------------------

Command ""D:\Program Files\Python37\python.exe" -u -c "import setuptools, tokenize;file='C:\Users\Administrator\AppData\Local\Temp\pycharm-packaging\imread\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\Administrator\AppData\Local\Temp\pip-record-jldkfk24\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\Administrator\AppData\Local\Temp\pycharm-packaging\imread
You are using pip version 19.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

[安全问题]代码内可能包含敏感信息

貌似你QQzone里面的代码中包含了你的QQ和密码耶，这些敏感信息，你确定不删？我记得b站当年就是github里泄露了些信息，导致内部API手册等外泄了耶

(泄露位置InterestingCrawler/QQZone/QQZoneShuoshuo.py)

本地部署问题

服务器控制台报错，好像爬取数据失败了，显示“服务器比较忙，请稍后再试”

如何爬取好友空间

从知乎上看过来的，感谢大神分享，想问一下如果要爬某个好友 A 的空间(有 QQ 号和访问权限)，需要修改 src/spider/main.py 中的哪些参数呢？

请问当前加了子项目的版本如何导出excel数据

即原来的功能：将所有抓取的信息清洗为excel或csv

扫码登陆后提示由于网络等原因登陆失败，请重新扫描二维码或稍后再试

因为没用过docker，摸索了挺久才成功跑起来，不知道是不是操作有问题，扫码登陆的时候提示
由于网络等原因登陆失败，请重新扫描二维码或稍后再试
而手机端是提示登陆成功的
试了很多次，还是这样

我想问题应该是出在这里，现在正在寻找修改的办法
File "F:\桌面\QQZoneMood\src\spider\QQZoneSpider.py", line 911, in get_qzone_token
qzonetoken = re.findall(re.compile("g_qzonetoken = (function(){ try{return "(.*)?""), content)[0]
IndexError: list index out of range

把这一行上的content print出来以后用在线的正则表达式测试去匹配发现匹配不到东西
怀疑是这个页面和这个项目的之前的页面相比已经发生了改变
但是本人学识水平有限，暂时无法找出解决方法

json以及csv文件

新手一枚，只是希望爬一点数据。为什么代码不会自动创建maicius_friend_detail_list.csv文件，我在文件夹中创建了一个后，运行又出现File "pandas_libs\parsers.pyx", line 545, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file。

Please attach methods to run

We need know how to run...
May be the developer can add some step-by-step guide in readme.md file.
;)

数据分析

数据分析过程第一次挺快的，为啥之后就特别慢？特别长时间也没出来

No matching distribution found for numpy

执行命令:

pip3 install -r requirements.txt

遇到错误：
No matching distribution found for numpy

发现不能下载QQ相册的照片

redis无法连接：Failed to connect redis: Error -2 connecting to redis:6379. Name or service not known.

docker版本我可以正常运行
然后我按照readme
修改userinfo.json.example为文件userinfo.json，并填好QQ号、QQ密码、保存数据用的文件名前缀；
随后我打开了redis，然后
python3 src/web/server.py

但是：
`36d3b270ee04:python -u /opt/project/src/web/server.py
['/opt/project/src/web', '/opt/project', '/opt/.pycharm_helpers/pycharm_display', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/site-packages', '/opt/.pycharm_helpers/pycharm_matplotlib_backend', '/opt/project', '/opt/project', '/opt/project/src', '/opt/project']
Traceback (most recent call last):
Failed to connect redis: Error -2 connecting to redis:6379. Name or service not known.
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 439, in connect
sock = self._connect()
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 494, in _connect
raise err
File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 482, in _connect
sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused