really-lazy-bone/beautiful-data

Visualizing the Flickr photo data on the map

beautiful-data

beautiful-data (BD) is an Open Source project that focuses on a large-scale data processing and visualization. Its ultimate goal is to provide researchers, business analysts and students a software framework for working with "big data" in relatively short order.

The code organization structure is loosely based on (inspired by) Jean-Paul Calderone's python project directory. Undoubtedly at some point we will have to integrate java and other languages/technologies. At which point, a reasonable refactoring effort is required.

The project's official website is at http://bd.pcwerk.com/

directory structure

lazy-bones-beautiful-data/
   LICENSE
   README.md
   bd_analyze/
   bd_collect/
   bd_store/
   bd_visualize/
   bin/
   website/

The bd_ directories contain source code for the four modules of project; the website directory contains the static web-related documents (e.g., .html, .css, and .js files); the bin directory contains wrapper scripts that can be used to execute the project. In theory, code within the bd_ directories should never be directly executed.

latex setup

Texmaker 3.2 was used to create the project.

The following template was used as requested: http://www.latextemplates.com/template/journal-article

If errors are returned in the the compiling of the file, that is probably due to an uninstalled font error. Please make sure to install texlive-fonts-recommended.

References:

[1] Calderone, Jean-Paul. "Filesystem structure of a Python project" http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html

beautiful-data's Issues

Showing Source Data in Table View

Maybe using ngTable to show the data and let user compare the data and do the testing for us!

Loading Animation while Loading "Huge" Data

Since our data is quite large, for the User Experience, we should provide this animation to indicate that user is waiting for the data and to show website is actually responding to user's action.

Tweak the View of the HighCharts and GoogleMap

Maybe tweak both library's graph to make it more uniform throughout the website

For example, the scrolling bar can be changed to fit into the minimal design.

Configuration File and Integration with Big Data Project

Jonthan, can you push your code with the configuration file to the master branch.

I think it would be best if we can put our collectFlickr.py into the Collect folder and run the bd.py to run our project as well. If you can do this, then we can do a pull request to the big data so that we can get some feedback on the project integration instead of us guessing out.

Interaction with Graph

Some simple interaction with graph would be really cool to be included.

For instance, search box, sort button and so on. This may require a web server end before adding these features.

Beautify the User Interface

The website can look better...

Add this to the background image.

Soring Flickr Data to Hadoop

Storing the data that Jonathan has collected from Flickr to Hadoop.

Note: have a note on the output format from Hadoop so that I know how to use the output data into the visualization part.

Analyze - ML or Statistics or any fashion to analyze the data

Lets use this issue to document what we have for the analyze part. @jkroening @surhorse

If you have any thought, please comment at this issue so that we can all document what we have in one place.

The data we have so far contains the following attributes

Attribute Name	Description/Note
photo_id	Unique Identifier
tags	List of tags that author puts
gelocation	longitude and latitude indicate where this photo is being taken, main used to show on the google map right now
date_taken	when photo is being taken, meta data related to photo
date_posted	when the photo is being uploaded to the Flickr Service
views	how many views this photo has when we downloaded the data
locale	where the photo is in String format
county	county
region	region
url	an url indicate where this image is on the Flickr server, mainly used to display the image on the visualization

Map Multiple Marker

If multiple markers show up on the same latitude and longitude. You cannot expand the markers to show multiple information windows at same time.

Eventually, this leads up to the marker information loss.

Time Series Chart

There is issue with time chart. Must fix before final presentation.

Host the Project onto Google App Engine

Instead of doing all the processing from the javascript. Host the project on the google app engine and use the python backend to do the simple analyzing and processing so that the performance issue on the website can be resolved.