virtadpt/exocortex

This is a rewrite and cleanup of the information gathering, indexing, and analysis system that I use to monitor information sources all over the Internet.

exocortex's Issues

Feature Set Ideas

i had forgotten i had started this but here it is:

WOOF (Web Of Omniscience Framework)

Purpose:

1. know everything ...

Feature set:

• inputs:
  • email (sms via email)
  • socket
  • webgui
  • mobile phone gui
  • TTS phone tree
  • dialup?
  • curiculum generation (list stuff you should know and get resources to help you learn them)
• outputs:
  • raw to disk/stream
  • email (sms via email)
  • TTS phone call
  • mobile phone gui
  • dialout?
• data sources:
  • search spider (google, bing, etc)
  • mailing list scapper/spider
  • emrgency service scanner (online listener count and increased amplitude spikes are probably the most reasonable metrics)
  • rss/atom spider
  • twitter scrapper/spider
• filters
  • blacklist
  • whitelist
  • algorithmic inclusion and exclusion
  • regex
• interfaces:
  • cli:
  • ncurse-like ui
  • oneliner (ala gpg)
  • screen/tty-based
  • gui:
  • html5+css3
  • swing? (java gui, much easier to use in scala)
  • gtk?
  • qt?
  • x?
  • API
  • cli (ala uci, for scripting)
  • socket (unix style so it can do web or local)
  • python
  • C/C++ (can be used via python, java, erlang, lua, etc)
  • Java (for android, also gives a foothold into many jvm based languages)

Structure

• module
  • core
  • the bit that the other bits are grafted onto
  • source of user input for the rest of the module
  • source
  • data source
  • filter
  • filter to handle data source or output of other filters
  • io rules
  • rules governing when and where things are sent to the user
• framework core
  • module that takes modules as sources

interesting random related crap

https://github.com/BirdAPI/Google-Search-API
https://gist.github.com/28405
http://bazaar.launchpad.net/~divmod-dev/divmod.org/trunk/files/head:/Reverend/
http://code.google.com/p/gdata-python-client/ (read available APIs and limitations carefully before getting too exceited)
https://bitbucket.org/fzzzy/python-actors/
http://docs.python.org/library/imaplib.html
http://docs.python.org/library/imghdr.html
http://docs.python.org/library/imp.html
http://pypi.python.org/pypi/galerts

http://docs.python.org/library/unittest.html

http://docs.python.org/library/cgihttpserver.html (see example code at the bottom)

http://orange.biolab.si/ (dunno if it is worth it but it's what i found)
http://www.elegantchaos.com/node/129 (looks more 'basement dweller' so it might be better)

Spider for standard extras

when poking at a site look for interesting things like .git, .svn, .hg blindly in directories found while spidering, inorder to incidentally discover interesting things :P