jdwije/linkedin-data-engine

A LinkedIn Data Mining Application for Network Visualization

LinkedIn Data Engine

This is an application that efficiently scrapes LinkedIn participant data via the LinkedIn APIs.

Written by Jason Wijegooneratne for Philip Schneider's Master Thesis.

About

The LinkedIn Data Engine (LDE) was originally designed to gather linkedin network data, from willing participants, in order to visualze network graphs in gephi. Some people expressed interest in its capabilites so I decied to release its source. Feel free to use, hack, do what you want with this stuff, it is released under the MIT licesence

Design

LDE is developed with the CodeIgnighter (CI) PHP framework and is designed to run on a linux HTTP stack backed by a a MYSQL server. The way this app is structured is such that there is a client facing portion of the CI backed website, which handles authenticating participants, getting and storing their oauth tokens, and also presenting a nice show of face. There is also a restricted section of the CI website. This is intended to be called via a cronjob issuing a curl command with a post parameter passcode set in order to run the schedule. This way the app fetches data on whatever schedule you define in your crons. The passcode paramter is passed with the curl command to restrict the general public (including robots) from trigguring your app. With that lets move onto..

Requirements

A linux server with HTTP capabilities (I use Apache), PHP 5+, MYSQL 5.5, CURL 7.27.0, and GIT installed. It might work with other versions of the aforementioned software however this is untested. With a few tweaks and mods you can run this from any sort of web stack that is php enabled...

Installation (LINUX/GIT) and Usage

After you have setup your server...

• cd to your http docs dir
• clone https://github.com/jdwije/LinkedIn-Data-Engine.git
• tailor the application/config/ config.php & database.php CI files to your environment
• setup your mysql data base. you can import the lde.sql file to mirror the required table structures
• schdule a cronjob like the one in the linux.cron file
• share the link to your webserver and watch the data start flooding in! After you have some participants you can access php my admin via www.yoururl.com/lde to inspect your data (sorry no management interface yet). It is a smart idea to create a sepereate mysql database user with readonly privileges specifically for inspecting the database in order to avoid messing anything up whilst you're in there.

Licensing

This software is lisenced under the MIT lisence by Jason Wijegooneratne & Philip Schneider. See the lde_user_license.txt file for a full description.

linkedin-data-engine's Issues

Install script

Database initialization should be automated. Also this script should check for the required dependencies (curl, oauth, etc) and notify the user if stuff is missing.

401 HTTP response header

It seems that with some participants when we try to fetch there network details we get a 401 error from the LinkedIn server. Not sure what is causing this however it is seemingly unrelated to the hanging scheduler bug and might have something to do with a user's settings as a 401 error indicates a contract has not been made. something to keep an eye on. removing user from the schedule is advised for now.

Hanging Scheduler Bug

The default behavior of the scheduler is to continuously request participant data until the apps data limit is hit. Should this happen when the app is part of the way through fetching a participants contacts, on resume, the app just hangs and doesn't fetch any further contacts. In this case you must manually reset the schedule table to kickstart everything again.