Comments (12)
What is the size of the cluster?
from code.
This is local env. Pseudo-distributed mode.
8GB Memory. 4 CPU's
from code.
It is very tough to see the expected behavior while running in local or pseudo distributed mode. You need to try it out in true cluster. By reducing number of mappers to 30 instead of 100 need not result in reduction in execution time. But it will use fewer resources on large cluster which means more jobs can run simultaneously.
from code.
I was just wondering why don't I see performance improvement referring this post that uses CombineFileInputFormat.
Since all other performance metrics discussed on series have shown noticeable performance improvement even on the local machine.
from code.
Can you paste counters information of both runs?
from code.
Hi dgadiraju,
Below are the details of both runs:
small_files_git_before_log.txt
Thanks.
from code.
Before one is map only job, there are no reducers involved in it. These runs cannot be comparable.
These below lines are in after, but not in before. So they are not functionally same to compare.
Reduce input groups=1
Reduce shuffle bytes=3271558374
Reduce input records=272629852
Reduce output records=1
from code.
I haven't made any changes in code. But I see following logs in both of the runs:
before: 16/03/06 21:28:24 INFO mapred.LocalJobRunner: reduce task executor complete
after: 16/03/06 22:10:03 INFO mapred.LocalJobRunner: reduce task executor complete.
Also, job RowCount.java (before) has NoKeyRecordCountReducer reducer. These job looks no different than using combineTextInputFormat in later case.
Thanks for the help!
from code.
I am not sure. As mentioned earlier, running local mode gives inconsistent results. At least you need to consider VM and set up full cluster on single node. Best is to use Cloudera or Hortonworks.
from code.
The issue is with the execution time since second run took 3 times as compared to first run. Also there were plenty of resources available on local machine while running the job.
There are also chances that the time difference ration would be same as the before job(100 mapper) possibly take less than 10 minutes. If executed on cluster.
from code.
I am not able to figure out with the information provided. Counters is not showing the true picture of what you are saying. Clearly "before" is not using reducers as per the counter information.
These does not mean that there is reduce task. If you go through the progress, "before" reduce progress have reached 100% directly where as "after" it became 100% incrementally.
before: 16/03/06 21:28:24 INFO mapred.LocalJobRunner: reduce task executor complete
after: 16/03/06 22:10:03 INFO mapred.LocalJobRunner: reduce task executor complete.
I am not 100% sure about pseudo distributed mode and I always prefer VMs with full cluster on it. Hence I cannot give proper explanation.
I am 100% sure that your comparison is not right. "before" runs as map only and "after" runs using reducers. You have to troubleshoot from that angle and see why "before" is not using reducers while "after" is using reducers.
from code.
Hi dgadiraju,
I tried rerunning both of the jobs, the before and after. Now I see the progress of reduce phase incrementally. I have attached the output of runs. So this becomes a valid comparision.
However, the difference in execution time still present(41 minutes & 31 minutes).
Any thoughts on this?
Thanks!
small_files_before_git.txt
small_files_after_git.txt
from code.
Related Issues (10)
- scala content on spark_demo_pyspark.txt
- Tt
- Hi durga, How can I resolve this error ?
- Need help in configuring ambari.repo
- sqoop import for orders table HOT 2
- Not able to do git clone setup please help me out in this.
- Setup Spark1.2.1 on quickstartVM HOT 1
- EDW PIG Issue
- How we can give the path of our conf.properties file that have our AWS credentials
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from code.