lubyruffy/fofa

fofa website

fofa's Issues

mysql切换到mongodb？sphinx切换到elasticsearch？

ip的归一化

0x0079.0x000000000000000028.0x0083.00257

0x是16进制

00是8进制

1-9开头的是10进制

0b是2进制。

支持https

用户可以自定义生成报表

只能从发布的rule中选择多项，然后每天跑一次数据，自动生成报表

你好，出现 undefined method `[]' for nil:NilClass错误请教。

在rake db:schema:load出现下面错误，请教怎么解决？
rake db:schema:load --trace
** Invoke db:schema:load (first_time)
** Invoke environment (first_time)
** Execute environment
rake aborted!
NoMethodError: undefined method []' for nil:NilClass /usr/share/fofa-clone/config/initializers/elasticsearch.rb:6:in<top (required)>'
/var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:in load' /var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:inblock in load'
/var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:240:in load_dependency' /var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:268:inload'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/engine.rb:652:in block in load_config_initializer' /var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/notifications.rb:166:ininstrument'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/engine.rb:651:in load_config_initializer' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/engine.rb:616:inblock (2 levels) in class:Engine'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/engine.rb:615:in each' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/engine.rb:615:inblock in class:Engine'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:30:in instance_exec' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:30:inrun'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:55:in block in run_initializers' /usr/lib/ruby/2.1.0/tsort.rb:226:inblock in tsort_each'
/usr/lib/ruby/2.1.0/tsort.rb:348:in block (2 levels) in each_strongly_connected_component' /usr/lib/ruby/2.1.0/tsort.rb:418:inblock (2 levels) in each_strongly_connected_component_from'
/usr/lib/ruby/2.1.0/tsort.rb:427:in each_strongly_connected_component_from' /usr/lib/ruby/2.1.0/tsort.rb:417:inblock in each_strongly_connected_component_from'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:44:in each' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:44:intsort_each_child'
/usr/lib/ruby/2.1.0/tsort.rb:411:in call' /usr/lib/ruby/2.1.0/tsort.rb:411:ineach_strongly_connected_component_from'
/usr/lib/ruby/2.1.0/tsort.rb:347:in block in each_strongly_connected_component' /usr/lib/ruby/2.1.0/tsort.rb:345:ineach'
/usr/lib/ruby/2.1.0/tsort.rb:345:in call' /usr/lib/ruby/2.1.0/tsort.rb:345:ineach_strongly_connected_component'
/usr/lib/ruby/2.1.0/tsort.rb:224:in tsort_each' /usr/lib/ruby/2.1.0/tsort.rb:205:intsort_each'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/initializable.rb:54:in run_initializers' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/application.rb:352:ininitialize!'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/railtie.rb:194:in public_send' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/railtie.rb:194:inmethod_missing'
/usr/share/fofa-clone/config/environment.rb:5:in <top (required)>' /var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:inrequire'
/var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in block in require' /var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:240:inload_dependency'
/var/lib/gems/2.1.0/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:274:in require' /var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/application.rb:328:inrequire_environment!'
/var/lib/gems/2.1.0/gems/railties-4.2.3/lib/rails/application.rb:457:in block in run_tasks_blocks' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:incall'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in block in execute' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:ineach'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in execute' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:inblock in invoke_with_call_chain'
/usr/lib/ruby/2.1.0/monitor.rb:211:in mon_synchronize' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:ininvoke_with_call_chain'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:201:in block in invoke_prerequisites' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:199:ineach'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:199:in invoke_prerequisites' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:178:inblock in invoke_with_call_chain'
/usr/lib/ruby/2.1.0/monitor.rb:211:in mon_synchronize' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:ininvoke_with_call_chain'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:in invoke' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:ininvoke_task'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in block (2 levels) in top_level' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:ineach'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in block in top_level' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:inrun_with_threads'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:in top_level' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:inblock in run'
/var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:in standard_exception_handling' /var/lib/gems/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:inrun'
/var/lib/gems/2.1.0/gems/rake-10.4.2/bin/rake:33:in <top (required)>' /usr/local/bin/rake:23:inload'
/usr/local/bin/rake:23:in `

'
Tasks: TOP => db:schema:load => environment

乱码问题

好像只有一台有乱码，需要提供一个工具检测哪一台出的问题。

webdb2_class.rb,thinking_sphinx.yml 这两个文件 怎么找不到了

很多文件里都 require 了 ,可是并没有找到这个文件额

添加错误请求http的队列，达到阈值后加如黑民单

主要是gfw导致的问题，比如facebook怎么请求都不行，但是很多网站又链接过去了。

可以考虑通过dnsenum的方式导入新域名

可否提供之前官网的组件库

可否提供之前官网的组件库

添加根据url输出应用的功能

支持增量实时显示更新数据功能

-用gearman系统，动态添加任务
-调用sphinx增量索引

增加工具从制定页面提取所有的host

比如一些平台会放出有漏洞的host列表，可以批量提取

支持ip

目前只支持域名加端口，不支持ip，很多gov.cn的直接链接到ip和端口的网站

将识别规则分类

• 云服务：如cloudflare之类的
• 主机系统：如xampp之类的
• 建站系统：如wordpress之类的。应该下分子类比如综合，企业，电商之类的
• 前端组件：比如jquery，bootstrap之类的
• 第三方模块：应该下分子类比如统计、广告、评论之类的

下划线（_）连起来的在sphinx中不会检索到

HTTP/1.1 200 OK
Content-Type: text/html; charset=gbk
Content-Encoding: gzip
Vary: Accept-Encoding
Set-Cookie: KQ7w_2132_saltkey=xbQQQaL0; expires=Wed, 06-Aug-2014 10:50:25 GMT; path=/; httponly, KQ7w_2132_lastvisit=1404726625; expires=Wed, 06-Aug-2014 10:50:25 GMT; path=/, KQ7w_2132_sid=H4d16o; expires=Tue, 08-Jul-2014 10:50:25 GMT; path=/, KQ7w_2132_lastact=1404730225%09portal.php%09; expires=Tue, 08-Jul-2014 10:50:25 GMT; path=/, KQ7w_2132_stats_qc_reg=deleted; expires=Sun, 07-Jul-2013 10:50:24 GMT; path=/, KQ7w_2132_cloudstatpost=deleted; expires=Sun, 07-Jul-2013 10:50:24 GMT; path=/, KQ7w_2132_sid=H4d16o; expires=Tue, 08-Jul-2014 10:50:25 GMT; path=/
Server: IIS
X-Powered-By: WAF/2.0
Date: Mon, 07 Jul 2014 10:50:25 GMT
Connection: close
Content-Length: 22848

通过KQ7w_2132_cloudstatpost可以检索到，但是cloudstatpost这样的就不能检索到。需要测试一下如何进行配置。

rake fofa:restart_all 报错