Comments (4)
Thanks for your feedback!
For hive concurrency mode, you need to config something like hive.metastore.uris
in hive-site.xml, before hive metastore service is up. You could check hive doc for more details at [Hive document about setting up remote metastore server].(https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreServer)
Also, could you file your fixes for HiBench as pull requests? it would be great to see more contributors and a better HiBench.
We will investigate your other issues a little bit later, since the Chinese new year is on the way. We are going to have a holiday.
Happy New Year and Thank you again!
from hibench.
Hello again!
Happy New (or maybe Goat) year!
I am coming back now since I didn 't have any answer of the other issues and to let you know about how I fixed the issue with hivebench running in parallel.
Actually, making hivebench run in parallel was a little bit more difficult than I expected, so, I am going to copy the links I found and helped me fix it. Fyi I have a remote metastore database and a local metastore server.
- Follow the instructions for remote metastore database -> https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin (ensure that you have the right version of mysql)
- Follow the instructions in the begining in order to set properly your new mysql database -> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/v4-2-0/CDH4-Installation-Guide/cdh4ig_topic_18_4.html#topic_18_4_3_unique_1__p_522_unique_1
- Make your database accessible from the metastore server host -> http://dev.mysql.com/doc/refman/5.6/en/adding-users.html
You may have to give more permissions to your new user e.g. ALTER, CREATE - Download and set the CLASSAPATH for the java-connector -> http://dev.mysql.com/doc/connector-j/en/connector-j-installing-classpath.html
Problems I encountered:
- In the hive-site.xml, replace ${system:java.io.tmpdir}/${system:user.name} by /tmp/mydir as what has been told in https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration (source: http://stackoverflow.com/questions/27099898/java-net-urisyntaxexception-when-starting-hive)
- If you have this error "ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)" then here is your answer -> http://stackoverflow.com/questions/1673530/error-2003-hy000-cant-connect-to-mysql-server-on-127-0-0-1-111
I hope this will help other guys too!
I am waiting for your answer about running in parallel the nutchindexing benchmark.
from hibench.
Sorry for leaving it for so long... was working on something else these days.
For Mahout versions, as far as I am concerned we are using the same version. You can even ignore the mahout hibench provided, but set you own MAHOUT_HOME to benchmark any compatible mahout, unless it doesn't support arguments we are using(we didn't test all mahout versions, but I think most of them would work).
For the nutchindexing problem, it may results from a not clean config. We are switch between different configurations according to your hadoop deployment and this could cause some problem.
For the dfsioe, it is a good catch. Maybe we need to handle the case when user gives us an empty configuration.
If I still miss anything, feel free to let me know. You are really helping us a lot and we do appreciate everything you did.
Again, we'd like you to file your fixes as pull request, so that we can review it in detail and hopefully merge them into trunk. And it would be great to see more contributors and a better HiBench.
I'll file some bugs as we discussed here separately. Thanks a lot!
from hibench.
Oh and for the nutchindexing temp file, can you specify which temp file we are using?
from hibench.
Related Issues (20)
- can you add me pls
- I am facing the below mentioned issue while trying to execute the pagerank algorithm in hadoop from hibench. Please give a solution. HOT 1
- upgrade kafka version
- Hi
- HiveError:HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient HOT 1
- Does Hibench Suite work with spark version 3.2.1 ? HOT 1
- Hi
- Error when run sql test
- Does Hibench Suite support Hadoop version 3.x ? HOT 1
- i don't no what i do -_- HOT 2
- Could tasks submitted to Spark by Hibench be viewed in Spark's History Server?
- Issues with run-sparkbench on Google Cloud Platform (GCP)
- spark config value modification does not apply.
- for single node hadoop configuration
- Hi
- The
- ERROR [26/26] RUN cd /root/HiBench && mvn clean package -Dspark=1.6 -Dscala=2.10
- Attempt to run HiBench on JDK 17 or higher versions HOT 1
- Hi
- Runstreaming job failed with error in bench.log like "org.apache.spark.SparkException: Error getting partition metadata for 'identity'. Does the topic exist?"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hibench.