Comments (10)
Did you have your HDFS working, and have right privileges?
from hibench.
I don't have a standard HDFS. I am using Lustre. I am able to run the TestDFSIO benchmark that comes along with Hadoop distribution.
from hibench.
What do you mean by myHadoop and Hadoop?
from hibench.
myHadoop : myHadoop provides a framework for launching Hadoop clusters within traditional high-performance compute clusters and supercomputers. It allows users to provision and deploy Hadoop clusters within the batch scheduling environment of such systems with minimal expertise required.
Hadoop : the opensource Apache Hadoop
I am able to run the simple wordcount, pi and TestDFSIO too. Are there any parameters that I can pass using "-D" to DFSIOE to force it using the base directory? For some reason, the "-Dtest.build.data= .... ", doesn't seem to work when I run the example. I use this option with "TestDFSIO", and it works just fine. I even tried using explicit path in the run script that is in HiBench distribution.
I have similar problems with "Nutchindexing" and "Bayesian" where the default file to read from is "/usr/share/dict/linux.word" and I don;t have that file but I do have "/usr/share/dict/american". Is there a way to change the files to read from?
from hibench.
TestDFSIOEnh may not support that argument.
for the dictionary file, you could make a soft link as a workaround.
from hibench.
Got the same issue. I'm using HDP 2.1 and Ambari 2.0.0.
DFSIOEnh fail to run this command to create the data input for test:
${HADOOP_EXECUTABLE} jar ${DATATOOLS} org.apache.hadoop.fs.dfsioe.TestDFSIOEnh \
-Dmapreduce.map.java.opts="-Dtest.build.data=${INPUT_HDFS} $MAP_JAVA_OPTS" \
-Dmapreduce.reduce.java.opts="-Dtest.build.data=${INPUT_HDFS} $RED_JAVA_OPTS" \
-write -skipAnalyze -nrFiles ${RD_NUM_OF_FILES} -fileSize ${RD_FILE_SIZE} -bufferSize 4096
The job was received by Yarn but then job failed and containers were killed.
15/04/22 08:18:54 INFO mapreduce.Job: Task Id : attempt_1429536760347_0016_m_000004_1, Status : FAILED
Error: java.io.IOException: Mkdirs failed to create /HiBench/benchmarks/TestDFSIO-Enh/io_data (exists=false, cwd=file:/hadoop/yarn/local/usercache/centos/appcache/application_1429536760347_0016/container_e02_1429536760347_0016_01_000017)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:449)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:851)
at org.apache.hadoop.fs.dfsioe.TestDFSIOEnh$WriteMapperEnh.doIO(TestDFSIOEnh.java:209)
at org.apache.hadoop.fs.dfsioe.IOMapperBase.map(IOMapperBase.java:123)
at org.apache.hadoop.fs.dfsioe.IOMapperBase.map(IOMapperBase.java:41)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
from hibench.
I have created a soft link for linux.words file, however I do see the error
"ERROR: number of words should be greater than 0"
The linux.words file is not empty.
from hibench.
We have been seeing the same issue as @hasonhai with Hortonworks HDP 2. I found the issue after debugging and tested a fix. I can send a Pull request, if you are interested.
from hibench.
Sure. Thanks.
On Sat, Jul 4, 2015 at 12:42 AM, viplav [email protected] wrote:
We have been seeing the same issue as @hasonhai
https://github.com/hasonhai with Hortonworks HDP 2. I found the issue
after debugging and tested a fix. I can send a Pull request, if you are
interested.—
Reply to this email directly or view it on GitHub
#80 (comment)
.
from hibench.
I have sent a pull request ( #110 ) to fix the issue with using wrong FS resulting in errors like "Mkdirs failed to create" etc. Please take a look and merge so that people who run into this issue (e.g. using Hortonworks HDP 2) will benefit.
from hibench.
Related Issues (20)
- can you add me pls
- I am facing the below mentioned issue while trying to execute the pagerank algorithm in hadoop from hibench. Please give a solution. HOT 1
- upgrade kafka version
- Hi
- HiveError:HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient HOT 1
- Does Hibench Suite work with spark version 3.2.1 ? HOT 1
- Hi
- Error when run sql test
- Does Hibench Suite support Hadoop version 3.x ? HOT 1
- i don't no what i do -_- HOT 2
- Could tasks submitted to Spark by Hibench be viewed in Spark's History Server?
- Issues with run-sparkbench on Google Cloud Platform (GCP)
- spark config value modification does not apply.
- for single node hadoop configuration
- Hi
- The
- ERROR [26/26] RUN cd /root/HiBench && mvn clean package -Dspark=1.6 -Dscala=2.10
- Attempt to run HiBench on JDK 17 or higher versions HOT 1
- Hi
- Runstreaming job failed with error in bench.log like "org.apache.spark.SparkException: Error getting partition metadata for 'identity'. Does the topic exist?"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hibench.