I am trying to perform an export into mongo using the Hive-mongo driver but a very frustrating error is popping up exactly at the very end of the map-reduce process. This error has to do with the serde dependency which I believe is the hive-exec.jar. Is this a bug? Can you please help me solve this issue..?
Stacktrace:
[root@localhost ~]# hive
Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties
Hive history file=/tmp/root/hive_job_log_root_201305281708_1510602612.txt
hive> insert into table mongo_popular_routes
> select r.route_short_name as id,
> r.route_long_name as name,
> cast(count(*) / 31 as INT) as daily_movements
> from train_movements tm
> join routes r
> on tm.route_id = r.route_id
> where tm.event_type = 1
> group by r.route_short_name, r.route_long_name;
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_1369775178800_0001, Tracking URL = http://localhost.localdomain:8088/proxy/application_1369775178800_0001/
Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=127.0.0.1:8021 -kill job_1369775178800_0001
Hadoop job information for Stage-0: number of mappers: 2; number of reducers: 1
2013-05-28 17:09:05,141 Stage-0 map = 0%, reduce = 0%
2013-05-28 17:09:21,597 Stage-0 map = 50%, reduce = 0%, Cumulative CPU 1.16 sec
2013-05-28 17:09:22,850 Stage-0 map = 50%, reduce = 0%, Cumulative CPU 1.16 sec
2013-05-28 17:09:23,991 Stage-0 map = 50%, reduce = 0%, Cumulative CPU 3.6 sec
2013-05-28 17:09:25,087 Stage-0 map = 50%, reduce = 0%, Cumulative CPU 3.6 sec
2013-05-28 17:09:26,460 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 4.09 sec
2013-05-28 17:09:27,665 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 6.11 sec
2013-05-28 17:09:28,849 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 6.11 sec
2013-05-28 17:09:30,031 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 6.14 sec
2013-05-28 17:09:31,185 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 7.09 sec
2013-05-28 17:09:32,395 Stage-0 map = 50%, reduce = 17%, Cumulative CPU 7.13 sec
2013-05-28 17:09:33,567 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 8.02 sec
2013-05-28 17:09:34,718 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 8.02 sec
2013-05-28 17:09:36,034 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 8.05 sec
2013-05-28 17:09:37,230 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 8.97 sec
2013-05-28 17:09:38,435 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 8.97 sec
2013-05-28 17:09:39,691 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 9.83 sec
2013-05-28 17:09:40,803 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 9.83 sec
2013-05-28 17:09:41,992 Stage-0 map = 59%, reduce = 17%, Cumulative CPU 9.86 sec
2013-05-28 17:09:43,117 Stage-0 map = 68%, reduce = 17%, Cumulative CPU 11.04 sec
2013-05-28 17:09:44,227 Stage-0 map = 68%, reduce = 17%, Cumulative CPU 11.04 sec
2013-05-28 17:09:45,405 Stage-0 map = 68%, reduce = 17%, Cumulative CPU 11.05 sec
2013-05-28 17:09:46,539 Stage-0 map = 76%, reduce = 17%, Cumulative CPU 12.23 sec
2013-05-28 17:09:47,696 Stage-0 map = 76%, reduce = 17%, Cumulative CPU 12.23 sec
2013-05-28 17:09:48,832 Stage-0 map = 76%, reduce = 17%, Cumulative CPU 13.52 sec
2013-05-28 17:09:49,934 Stage-0 map = 76%, reduce = 17%, Cumulative CPU 13.52 sec
2013-05-28 17:09:51,061 Stage-0 map = 76%, reduce = 17%, Cumulative CPU 13.54 sec
2013-05-28 17:09:52,166 Stage-0 map = 83%, reduce = 17%, Cumulative CPU 14.95 sec
2013-05-28 17:09:53,299 Stage-0 map = 100%, reduce = 17%, Cumulative CPU 15.57 sec
2013-05-28 17:09:54,578 Stage-0 map = 100%, reduce = 17%, Cumulative CPU 15.79 sec
2013-05-28 17:09:55,653 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:09:56,726 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:09:57,783 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:09:58,863 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:09:59,985 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:01,117 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:02,234 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:03,364 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:04,438 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:05,516 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:06,713 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:07,850 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:09,011 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:10,109 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:11,240 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:12,379 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:13,443 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:14,499 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:15,628 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:16,743 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:17,862 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:18,935 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:20,091 Stage-0 map = 100%, reduce = 0%, Cumulative CPU 14.86 sec
2013-05-28 17:10:21,294 Stage-0 map = 100%, reduce = 100%, Cumulative CPU 14.86 sec
MapReduce Total cumulative CPU time: 14 seconds 860 msec
Ended Job = job_1369775178800_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1369775178800_0001_m_000001 (and more) from job job_1369775178800_0001
Task with the most failures(4):
Task ID:
task_1369775178800_0001_r_000000
URL:
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.SerDeUtils.hasAnyNullObject(Ljava/util/List;Lorg/apache/hadoop/hive/serde2/objectinspector/StandardStructObjectInspector;[Z)Z
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:448)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.SerDeUtils.hasAnyNullObject(Ljava/util/List;Lorg/apache/hadoop/hive/serde2/objectinspector/StandardStructObjectInspector;[Z)Z
at org.apache.hadoop.hive.ql.exec.JoinOperator.processOp(JoinOperator.java:126)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 7 more