Giter Site home page Giter Site logo

spoddutur / cloud-based-sql-engine-using-spark Goto Github PK

View Code? Open in Web Editor NEW
32.0 7.0 14.0 189 KB

Cloud-based SQL engine using SPARK where data is accessible as JDBC/ODBC data source via Spark ThriftServer.

Scala 36.68% Java 63.32%
apache-spark thrift-server spark-thrift-server sql-engine sparksql jdbc beeline hadoop-framework

cloud-based-sql-engine-using-spark's People

Contributors

spoddutur avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

witnesslq evinhas plthiyagu jiayuasu wasim24 myownr vamosraghava wyfunique samar40288 toannhu96 mjzfrank etra

cloud-based-sql-engine-using-spark's Issues

SessionHiveMetaStoreClient issue

Spark Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre/bin/java -cp /Users/ibm/atif/spark/conf/:/Users/ibm/atif/spark/jars/* -Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift JDBC/ODBC Server spark-internal

2019-04-04 17:29:50 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-04-04 17:29:55 INFO HiveThriftServer2:2566 - Started daemon with process name: [email protected]
2019-04-04 17:29:55 INFO SignalUtils:54 - Registered signal handler for TERM
2019-04-04 17:29:55 INFO SignalUtils:54 - Registered signal handler for HUP
2019-04-04 17:29:55 INFO SignalUtils:54 - Registered signal handler for INT
2019-04-04 17:29:55 INFO HiveThriftServer2:54 - Starting SparkContext
2019-04-04 17:29:55 INFO SparkContext:54 - Running Spark version 2.4.0
2019-04-04 17:29:55 INFO SparkContext:54 - Submitted application: Thrift JDBC/ODBC Server
2019-04-04 17:29:55 INFO SecurityManager:54 - Changing view acls to: ibm
2019-04-04 17:29:55 INFO SecurityManager:54 - Changing modify acls to: ibm
2019-04-04 17:29:55 INFO SecurityManager:54 - Changing view acls groups to:
2019-04-04 17:29:55 INFO SecurityManager:54 - Changing modify acls groups to:
2019-04-04 17:29:55 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ibm); groups with view permissions: Set(); users with modify permissions: Set(ibm); groups with modify permissions: Set()
2019-04-04 17:30:01 INFO Utils:54 - Successfully started service 'sparkDriver' on port 54993.
2019-04-04 17:30:01 INFO SparkEnv:54 - Registering MapOutputTracker
2019-04-04 17:30:01 INFO SparkEnv:54 - Registering BlockManagerMaster
2019-04-04 17:30:01 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-04-04 17:30:01 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2019-04-04 17:30:01 INFO DiskBlockManager:54 - Created local directory at /private/var/folders/x6/5rj4r3f94m96hr12q2srdz700000gn/T/blockmgr-88cba71c-88de-4080-83ed-cba14ce6b8d8
2019-04-04 17:30:01 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2019-04-04 17:30:01 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2019-04-04 17:30:01 INFO log:192 - Logging initialized @22420ms
2019-04-04 17:30:01 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-04-04 17:30:01 INFO Server:419 - Started @22524ms
2019-04-04 17:30:01 INFO AbstractConnector:278 - Started ServerConnector@bf71cec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-04-04 17:30:01 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1500e009{/jobs,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bdaf2ce{/jobs/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42d236fb{/jobs/job,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@19f21b6b{/jobs/job/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1532c619{/stages,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@46044faa{/stages/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1358b28e{/stages/stage,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7de4a01f{/stages/stage/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2bfeb1ef{/stages/pool,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@778ca8ef{/stages/pool/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@208e9ef6{/storage,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@78b236a0{/storage/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@261d8190{/storage/rdd,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@34448e6c{/storage/rdd/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@60e9df3c{/environment,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@907f2b7{/environment/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@435ce306{/executors,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@537b32ef{/executors/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dc51783{/executors/threadDump,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b61d0c6{/executors/threadDump/json,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6f815e7f{/static,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@110844f6{/,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6f89f665{/api,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3a43d133{/jobs/job/kill,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@39ce27f2{/stages/stage/kill,null,AVAILABLE,@spark}
2019-04-04 17:30:01 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://10.16.2.83:4040
2019-04-04 17:30:01 INFO Executor:54 - Starting executor ID driver on host localhost
2019-04-04 17:30:01 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54994.
2019-04-04 17:30:01 INFO NettyBlockTransferService:54 - Server created on 10.16.2.83:54994
2019-04-04 17:30:01 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-04-04 17:30:02 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 10.16.2.83, 54994, None)
2019-04-04 17:30:02 INFO BlockManagerMasterEndpoint:54 - Registering block manager 10.16.2.83:54994 with 366.3 MB RAM, BlockManagerId(driver, 10.16.2.83, 54994, None)
2019-04-04 17:30:02 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 10.16.2.83, 54994, None)
2019-04-04 17:30:02 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 10.16.2.83, 54994, None)
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54562ea6{/metrics/json,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO SharedState:54 - loading hive config file: file:/Users/ibm/atif/spark/conf/hive-site.xml
2019-04-04 17:30:02 INFO SharedState:54 - Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/ibm/atif/spark/spark-warehouse').
2019-04-04 17:30:02 INFO SharedState:54 - Warehouse path is 'file:/Users/ibm/atif/spark/spark-warehouse'.
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6a9950f1{/SQL,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ad54c55{/SQL/json,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a3e5f23{/SQL/execution,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6293e39e{/SQL/execution/json,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3daa82be{/static/sql,null,AVAILABLE,@spark}
2019-04-04 17:30:02 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
2019-04-04 17:30:03 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:03 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:03 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
2019-04-04 17:30:04 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:04 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:04 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
2019-04-04 17:30:05 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:05 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:05 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
2019-04-04 17:30:06 WARN Hive:168 - Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:183)
at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:117)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:272)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
... 36 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 42 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:183)
at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:117)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:272)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 50 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
... 47 more
2019-04-04 17:30:06 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:06 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:06 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
2019-04-04 17:30:07 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:07 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:07 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
2019-04-04 17:30:08 INFO metastore:376 - Trying to connect to metastore with URI thrift://localhost:9083
2019-04-04 17:30:08 WARN metastore:428 - Failed to connect to the MetaStore Server...
2019-04-04 17:30:08 INFO metastore:459 - Waiting 1 seconds before next connection attempt.
Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:183)
at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:117)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:272)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
... 18 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
... 33 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 39 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:86)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:183)
at org.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:117)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:272)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:384)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:286)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:215)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:79)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
... 47 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:236)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
... 44 more
2019-04-04 17:30:09 INFO SparkContext:54 - Invoking stop() from shutdown hook
2019-04-04 17:30:09 INFO AbstractConnector:318 - Stopped Spark@bf71cec{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-04-04 17:30:09 INFO SparkUI:54 - Stopped Spark web UI at http://10.16.2.83:4040
2019-04-04 17:30:09 INFO MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2019-04-04 17:30:09 INFO MemoryStore:54 - MemoryStore cleared
2019-04-04 17:30:09 INFO BlockManager:54 - BlockManager stopped
2019-04-04 17:30:09 INFO BlockManagerMaster:54 - BlockManagerMaster stopped
2019-04-04 17:30:09 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2019-04-04 17:30:09 INFO SparkContext:54 - Successfully stopped SparkContext
2019-04-04 17:30:09 INFO ShutdownHookManager:54 - Shutdown hook called
2019-04-04 17:30:09 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/x6/5rj4r3f94m96hr12q2srdz700000gn/T/spark-d554e64a-4cfd-46db-a63d-27da4372ed0c
2019-04-04 17:30:09 INFO ShutdownHookManager:54 - Deleting directory /private/var/folders/x6/5rj4r3f94m96hr12q2srdz700000gn/T/spark-70a9d853-cd5e-49d1-8526-2dcc293d2e4f

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.