virtadpt/exocortex

This is a rewrite and cleanup of the information gathering, indexing, and analysis system that I use to monitor information sources all over the Internet.

NOTE: This project is effectively dead. If you're looking at this you should direct yourself to Exocortex Halo. All things considered it's far more advanced and way more useful (unless all you're looking for is sample code to learn how to write your own Python modules...)

This is a catch-all directory for a project I'm calling Exocortex - a system that spiders and indexes arbitrary data sources (Twitter timelines, RSS feeds, and other online storehouses of personally interesting and useful information) and makes it possible to search and extract information from them as needed. This project is likely to be in alpha for a long time because it scratches a couple of personal itches. It's also an excuse for me to learn about some stuff I don't know already, like the guts of XMPP and machine learning.

This is going to be a once in a while project for me, something that I work on when I have a little time and need to do something different. Please don't treat this like the next new thing or the second coming of Alan Turing, I set this Github project up so that there would be a remote repository that I could fall back on.

Written in Python2.

Dependencies:

• SleekXMPP
• Tweepy

exocortex's Issues

Feature Set Ideas

i had forgotten i had started this but here it is:

WOOF (Web Of Omniscience Framework)

Purpose:

1. know everything ...

Feature set:

• inputs:
  • email (sms via email)
  • socket
  • webgui
  • mobile phone gui
  • TTS phone tree
  • dialup?
  • curiculum generation (list stuff you should know and get resources to help you learn them)
• outputs:
  • raw to disk/stream
  • email (sms via email)
  • TTS phone call
  • mobile phone gui
  • dialout?
• data sources:
  • search spider (google, bing, etc)
  • mailing list scapper/spider
  • emrgency service scanner (online listener count and increased amplitude spikes are probably the most reasonable metrics)
  • rss/atom spider
  • twitter scrapper/spider
• filters
  • blacklist
  • whitelist
  • algorithmic inclusion and exclusion
  • regex
• interfaces:
  • cli:
  • ncurse-like ui
  • oneliner (ala gpg)
  • screen/tty-based
  • gui:
  • html5+css3
  • swing? (java gui, much easier to use in scala)
  • gtk?
  • qt?
  • x?
  • API
  • cli (ala uci, for scripting)
  • socket (unix style so it can do web or local)
  • python
  • C/C++ (can be used via python, java, erlang, lua, etc)
  • Java (for android, also gives a foothold into many jvm based languages)

Structure

• module
  • core
  • the bit that the other bits are grafted onto
  • source of user input for the rest of the module
  • source
  • data source
  • filter
  • filter to handle data source or output of other filters
  • io rules
  • rules governing when and where things are sent to the user
• framework core
  • module that takes modules as sources

interesting random related crap

https://github.com/BirdAPI/Google-Search-API
https://gist.github.com/28405
http://bazaar.launchpad.net/~divmod-dev/divmod.org/trunk/files/head:/Reverend/
http://code.google.com/p/gdata-python-client/ (read available APIs and limitations carefully before getting too exceited)
https://bitbucket.org/fzzzy/python-actors/
http://docs.python.org/library/imaplib.html
http://docs.python.org/library/imghdr.html
http://docs.python.org/library/imp.html
http://pypi.python.org/pypi/galerts

http://docs.python.org/library/unittest.html

http://docs.python.org/library/cgihttpserver.html (see example code at the bottom)

http://orange.biolab.si/ (dunno if it is worth it but it's what i found)
http://www.elegantchaos.com/node/129 (looks more 'basement dweller' so it might be better)

Spider for standard extras

when poking at a site look for interesting things like .git, .svn, .hg blindly in directories found while spidering, inorder to incidentally discover interesting things :P