Comments (12)
Getting the below error while connecting to remote metastore using kerberos
Need ur suggestion? Anyone please help to resolve. Thanks
from waggle-dance.
Hi @patduin / @abhimanyugupta07 ,
Does the WD supports DBTokenStore?
In my company all the existing Hive metastore is configured with DBTokenStore. For the initial connection it's connecting successfully using Kerberos. For the subsequent connection, it's trying with TOKENs and getting failed with DIGEST-MD5: IO error acquiring password.
from waggle-dance.
I know very little about Kerberos as we don't use it ourselves.
Kerberos has been a community contribution perhaps @zzzzming95 can help?
from waggle-dance.
I know very little about Kerberos as we don't use it ourselves.
Kerberos has been a community contribution perhaps @zzzzming95 can help?
Thanks for the reply. @zzzzming95 kindly help on this issue.
from waggle-dance.
The below config has to be configured both in primary & remote metastores?
The answer is yes. In waggle-dance , we need to use one token to access all metastore (include primary & remote metastores) .
DBTokenStore
mean using mysql to store token , it only share the token who using the same mysql. So it need to change the token sotre to ZKtokenstore.
It should be noted that you need to pay attention to the token storage capacity. By default, the storage limit of a single znode in zk is about 50,000 tokens.
from waggle-dance.
The below config has to be configured both in primary & remote metastores?
The answer is yes. In waggle-dance , we need to use one token to access all metastore (include primary & remote metastores) .
DBTokenStore
mean using mysql to store token , it only share the token who using the same mysql. So it need to change the token sotre to ZKtokenstore.It should be noted that you need to pay attention to the token storage capacity. By default, the storage limit of a single znode in zk is about 50,000 tokens.
Thanks for ur comment. PROD metastore already using DBTokenStore. Changing it to ZooKeeperTokenStore may not be the right choice.
Shall I use the same MySQL database where the PROD metastore is using?
Will tht help to resolve the issue?
from waggle-dance.
Shall I use the same MySQL database where the PROD metastore is using?
Using the same mysql mean you just use the same metastore beacause DBTokenStore
is link with other interfaces(like get_table , get_partition) .Then there is no need to use waggle-dance.
We also switched from DBTokenStore
to ZkTokenStore
, which is a feasible solution.
from waggle-dance.
Shall I use the same MySQL database where the PROD metastore is using?
Using the same mysql mean you just use the same metastore beacause
DBTokenStore
is link with other interfaces(like get_table , get_partition) .Then there is no need to use waggle-dance.We also switched from
DBTokenStore
toZkTokenStore
, which is a feasible solution.
Sorry I misunderstood. Yes purpose to access to all remote metastores. So we can't keep the token along with it. As u said ZooKeeperTokenStore is the only solution.
from waggle-dance.
Hi @patduin ,
In my use case, I have some metastores have the same database name. How to handle this in WD?
from waggle-dance.
Please open new discussion for such questions that has nothing todo with original issue. Please see the readme: https://github.com/ExpediaGroup/waggle-dance/blob/main/README.md#database-resolution
from waggle-dance.
Hello, @zzzzming95 . If our primary metastore and federation metastore use different KDCs, how should we configure them in wd?
Do you mean that these configurations are the same in both WD, primary ms, and federation ms.
That is to say, for example,
hive. cluster.delegation. token. store. zookeeper. connectString
, are they all the same?
Are their values are all zk1:2181, zk2:2181, and zk3:2181
.
<property> <name>hive.cluster.delegation.token.store.class</name> <value>org.apache.hadoop.hive.thrift.ZooKeeperTokenStore</value> </property> <property> <name>hive.cluster.delegation.token.store.zookeeper.connectString</name> <value>zk1:2181,zk2:2181,zk3:2181</value> </property> <property> <name>hive.cluster.delegation.token.store.zookeeper.znode</name> <value>/hive/token</value> </property>
Also, could you provide me with a social way so that I can consult with you, such as WeChat.
from waggle-dance.
#313 @sivaponting Perhaps my PR can meet your needs.
from waggle-dance.
Related Issues (20)
- Hive 3 support HOT 6
- Support Presto views
- add support for single char dot wildcard in like queries
- Delete database failed
- improve metrics by tagging the call and metastore.
- mvn clean package failed HOT 7
- Hiveserver2 uses the metastore url service of the waggle dance. Operating to modify the table partition in the same session will change the ugi permission of the session
- CloseableThriftHiveMetastoreIfaceClientFactory new instance get blocked HOT 1
- Hive default database fails to pass the verification HOT 2
- how to disable spring actuator HOT 5
- Waggle-dance does not support DIGEST protocol HOT 2
- HA Support HOT 3
- Connecting WD via Beeline HOT 4
- Tesing WD with CDH 3.x version HOT 2
- Hive3.x get_table_meta null pointer exception HOT 6
- Add configuration to enable exposing all monitoring data through Prometheus" HOT 3
- General Enquires on WD HOT 2
- The `waggle-dance-federation.yml` file is updated when Waggledance is stopped. HOT 1
- Use different tokens instead of forcing WD and all HMS to use the same delegatetoken in the kerberos environment HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from waggle-dance.