deercoder/hadoop-practice

Hadoop demo, my final project for advanced database

README

Hadoop project, MapReduce, Data Analysis, Java

This project is to use Java to manuplate multiple text files with selecting, projection, natural joint and aggregation. All these operations are done with MapReduce process instead of database.

The target of the project is to let us get familar with the MapReduce machnism, how to design reasonable Map and Reduce function, and how to dispatch/deploy the tasks in Hadoop.

For details, check the ProjectDescription.pdf

##GUIDE

This folder stores the necessary data source, code as well as the output for the Advanced Database Project, the structure is as follows:

data: the source data file that stores the manuplated data(city.txt, country.txt, countrylanguage.txt)

output: the output result of each experiments, there is a flag file indicating the status, and the other part-r-00000 is the generated result.

src: the source code of the project written in Java, for each seperate task there is a package named ex1, ex2,ex3, ex4.

INSTALL_GUIDE_2nd.pd: the guide that I referred when configuring the Hadoop environment

##Contact

Email: Chang Liu([email protected])

hadoop-practice's Issues

How to save in unicode in Hadoop

In ex1, I output the city name, but it seems that the format of hadoop don't support unicode, so my solution will display abnormally, this is a bug.

And I think hadoop should be good at dealing this issue, since many data contains regional code and characters in different languages.

Leave a mark here and will solve it soon.

How to deal with the zero number in ex1

For hadoop's reduce(), here I just use empty(which means number zero) to output one collection. For example, if there is a city that satisfy the requirement, I just output (city, 0), as reduce() or map() both needs to operate as a tuple. But as a solution it's actually not so good.

But I think it's not elegant, there must be method to eliminate the zero so that there is only the city list for ex1.

Problem Set 1: output the cities where the populations are more than 300,000.

Problem: I will output the city name as well as an 0 behind it.

code refactoring for ex1, ex2

These two projects are not elegant, many of codes are poorly written and contains some format error, I need to rewrite them in good way...

bug for ex3

ex3 doesn't actually output all the countries that is English, it outputs ALL the countries. We have to add other judge conditions.

cannot call map() function, but there is no error information and still outputs something

It's quite strange that I can execute the map() function, no error is there, and there is also something output there, but it seems that our map() function is actually not called, since at the very first beginning, log is not output.

Then why this bug happens? And why there is no error?

ex3 is not accurate, the project should output countries, not cities.

as title shows, wrong coding for the problem set, don't meet the requirements.