expediagroup/waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.

waggle-dance's Issues

Update parent version

The newest version of hotels-oss-parent includes the sonatype-oss-release profile with the maven-gpg-plugin, so it can be removed from this project.

Acceptance Criteria;

• Update to version 2.0.5 of hotels-oss-parent.
• Remove the sonatype-oss-release profile in pom.xml.

Persist config file during process restart

As a SysOps
I want WD to avoid overwriting local config during process restart
So that I do not lose local configs

Acceptance Criteria:

• WD to not update/re-write config file during restart

tunneling: Support multiple host key encryptions

As a user of Waggle Dance
I'd like to have a working tunnel where my host key is of any encryption type
So that I can seamlessly create tunnels on newer SSH installations

We've seen the JCSH library we use for tunneling fail with Caused by: com.jcraft.jsch.JSchException: UnknownHostKey on host checks when the .ssh/known_hosts file has keys encrypted with ecdsa-sha2-nistp256.

Acceptance Criteria:

• Investigate if we can make JCSH work with all encryptions

Work around

A command line workaround is to get the RSA key using ssh-keyscan host-ip and adding that to the known_hosts file:

ssh-keyscan <host-ip> >> .ssh/known_hosts

Refactor duplicated SSH code

As a developer
I want code that is duplicated by code present in hcommon-ssh removed from WaggleDance
So that I have less code to maintain, can fix bugs in one place etc.

Acceptance Criteria
The following are candidates for refactoring, further investigation might reveal that it's not feasible so for all of the below we should either make the change or document why we decided not to:

• Replace com.hotels.bdp.waggledance.api.model.MetastoreTunnel with com.hotels.hcommon.ssh.SshSettings
• If the above is successful create a similar ticket to this for CircusTrain to make the same change for com.hotels.bdp.circustrain.api.conf.MetastoreTunnel
• Replace com.hotels.bdp.waggledance.api.validation.constraint.TunnelRoute with com.hotels.hcommon.ssh.validation.constraint.TunnelRoute and
  com.hotels.bdp.waggledance.api.validation.validator.TunnelRouteValidator with com.hotels.hcommon.ssh.validation.validator.TunnelRouteValidator
• If the above is successful create a similar ticket to this for CircusTrain to make the same change for com.hotels.bdp.circustrain.api.validation.constraints.TunnelRoute and com.hotels.bdp.circustrain.api.validation.constraintvalidators.TunnelRouteValidator)
• Look at TunnelableFactorySupplier and how we're using WaggleDanceHiveConfVars, could we replace some of this with com.hotels.hcommon.ssh.SshSettings or com.hotels.hcommon.hive.metastore.client.tunnelling.TunnellingMetaStoreClientSupplierBuilder? See ExpediaGroup/circus-train@0272e36 for a similar change to CircusTrain that could possibly be applied here.
• Check whether is any scope for refactoring/removing any code in com.hotels.bdp.waggledance.client.TunnelingMetaStoreClientFactory
• Above code tested out in our development environment in AWS via a tunnel to ensure tunnels still work correctly.

Clarify what is meant by 'write' in the documentation

We make a distinction between 'write' and 'create', but understandably this may cast doubt on what behaviours 'write' encompasses. We should call out in a list something like:

• Read: anything that does not mutate metadata.
• Write: anything that might create or mutate metadata with the exception of the creation of databases.
• Create: Creation of databases.

Alternatively we could create a matrix of all metadata actions by support in each access control type.

Avoid reboot to refresh config

As a SysOps
I want to have a WD to update config from local FS without need of restart
So that it can be managed easily in production environment

Acceptance Criteria:

• Able to add/remove/update Whitelist DB without having to restart Waggle dance service.
• Waggle Dance to refresh in-memory config from local FS config file every 5 or 10sec (maybe configurable via property)

Enable custom ThriftHiveMetastore.IFace implementations

Enable configurable pluggable implementations of the ThriftHiveMetastore.IFace interface on a per (primary or federated) metastore basis.

Example of how the configuration could look like

primary-meta-store:
  access-control-type: READ_ONLY
  database-prefix: ''
  name: primary
  closeable-iface: com.company.foo.bar.MyImpl
  #com.company.foo.bar.MyImpl is a custom 
  #implementation of the CloseableThriftHiveMetastoreIface  that can forward
  #requests to some service in the background and adapt them to the Hive interface 
  #e.g. federate Glue onto hive, federate BigQuery onto hive etc by using this
  #interface as an adapter from your data store to the 
  #CloseableThriftHiveMetastoreIface  interface 

federated-meta-stores:
- access-control-type: READ_ONLY
  database-prefix: waggle_test_
  mapped-databases: []
  name: secondary
  remote-meta-store-uris: thrift://waggle-dance:48869
  #No closeable-iface implementation is specified so it defaults to Thrift 

  #Thrift Hive will be federated onto your custom implementation

Extract common Hive Metastore code into a library

Extract SSH logic into a separate library

There is some shared SSH-tunnel code here and in Circus Train which needs to be improved. We want to pull the code out to a separate library.

Sync configuration between all WD instances

As a SysOps
I want to have WD to sync configs between multiple WD instances behind the ELB
So that all WD instances have same configs

FAILED: ParseException line 1:7 'CREATE' 'REMOTE' 'TABLE' in ddl statement

USING Waggle Dance 2.3.7 rpm

Connected to waggle dance

hive --hiveconf hive.metastore.uris=thrift://localhost:48869.
Doesn't Like Create REMOTE table statement itself. Did anyone faced similar issue .

hive> CREATE REMOTE TABLE test
> CONNECTED TO third_db_hive.t3
> VIA 'org.apache.hadoop.hive.metastore.ThriftHiveMetastoreClientFactory'
> WITH TBLPROPERTIES (
> 'hive.metastore.uris' = 'thrift://xxxx.compute.amazonaws.com:9083'
> );
NoViableAltException(24@[846:1: ddlStatement : ( createDatabaseStatement | switchDatabaseStatement | dropDatabaseStatement | createTableStatement | dropTableStatement | truncateTableStatement | alterStatement | descStatement | showStatement | metastoreCheck | createViewStatement | createMaterializedViewStatement | dropViewStatement | dropMaterializedViewStatement | createFunctionStatement | createMacroStatement | createIndexStatement | dropIndexStatement | dropFunctionStatement | reloadFunctionStatement | dropMacroStatement | analyzeStatement | lockStatement | unlockStatement | lockDatabase | unlockDatabase | createRoleStatement | dropRoleStatement | ( grantPrivileges )=> grantPrivileges | ( revokePrivileges )=> revokePrivileges | showGrants | showRoleGrants | showRolePrincipals | showRoles | grantRole | revokeRole | setRole | showCurrentRole | abortTransactionStatement );])
at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
at org.antlr.runtime.DFA.predict(DFA.java:144)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3757)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:208)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
FAILED: ParseException line 1:7 cannot recognize input near 'CREATE' 'REMOTE' 'TABLE' in ddl statement

What I need to do to fix it.

Thanks

Waggle Dance fails to serve when destination bastion is down/unreachable

Issue:
Waggle Dance federates to primary Metasotre A and foreign Metastore B (over SSH Tunnel through bastion host BH).
If for any reason, BH is down/unreachable or disallow SSH connection then this cause whole WD to stops serving request even for Metastore A.

Excepted Behaviour:
WD to handle the above situation gracefully and continue to serve rest of the Metastores.

Current Behaviour:
WD fails to serve all requests.

Use configuration-properties in clients HiveConf

I'd like to use the configuration-properties that I can set in the waggle-dance-server.yml in HiveConf that is used to create Thrift Clients.

Currently we use the properties only in the Handler (the server), it would be good if we can set properties and override settings used in the clients.
For instance see ThriftMetastoreClient it contains this code:
retries = HiveConf.getIntVar(conf, HiveConf.ConfVars.METASTORETHRIFTCONNECTIONRETRIES);
But there is currently no way to override this cause we create the HiveConf ourselves in CloseableThriftHiveMetastoreIfaceClientFactory. Thus that values is hardcoded to the Hive default 3.

Acceptance Criteria:

• Use theconfiguration-properties(see WaggleDanceConfiguration class) when creating the HiveConf for the clients.
• Documents it's use in the README (it seems to be a hidden feature now).

Add integration test for federations over SSL

As a developer
I'd like to test that SSL connectivity
So I can make sure that federations over SSL works the same as federations over plain connectivity

Acceptance Criteria
New integration test that federates megastores over SSL

Note: this we'll depend on BeeJU ticket 8

Refactor FederationAdminController and its use of AbstractMetastore

As a developer
I want to ensure that code is logical and clean
so that I spend less time on maintenance

Acceptance Criteria
TBD based on extracting exactly what we want to from notes below

Notes
Below is a brain dump of discussions we've had on the issue that hopefully we can extract proper acceptance criteria out of:

Notes from handover discussion with @ddcprg:

FederationAdminController-> add() takes an Abstract class as param this should really be a concrete class or an interface? There is no wayit would know what the actual type is when it came to create one. We think currently only federations() method is being called by one of our end users.

Comments from @massdosage in Slack:

seeing as nobody is really using the REST interface I'm not so bothered about the above, it could probably do with an entire refactor as all the methods take the abstract meta store class which isn't ideal
but probably only worth doing if/when someone actually wants to do something more than a health check with the rest calls.

Comments from @patduin in Slack:

Yes, this is a general thing that we don’t have an interface over the AbstractMetastoreHierarchy
probably makes sense to add that. What do you say of introducing a rest model metastore class, that takes a API model metastore class in the constructor? it would decouple the API model class from being serialized in the yaml config and in the presentation layer (JSON). We’ve had issues with that before as we try to fix something for nice yml and that breaking the JSON for instance: private transient @JsonProperty @NotNull MetaStoreStatus status = MetaStoreStatus.UNKNOWN;
is a field in the AbstractMetastore, but it is only serialized for json not for the yml and hence transient which is a bit weird.

Fix Manifest metadata logging

From the logs:
2017-08-10T15:52:41,866 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:86 - Manifest location in JARs is null
2017-08-10T15:52:41,867 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:135 - Could not find manifest in location file:file:/opt/waggle-dance/service/waggle-dance-core-latest-exec.jar!/BOOT-INF/classes!//META-INF/MANIFEST.MF
2017-08-10T15:52:41,867 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:92 - Manifest location on disk is null
2017-08-10T15:52:41,867 DEBUG com.hotels.bdp.waggledance.manifest.ManifestAttributes:98 - Manifest location via getResource() is jar:file:/opt/waggle-dance/service/waggle-dance-core-latest-exec.jar!/META-INF/MANIFEST.MF

That's from an rpm install, not sure what happened but would be nice if we can see the Manifest attributes logged again.

Status health checks doesn't support tunnels

As a user of Waggle Dance
I'd like to see the status of a metastore that is connected to via a tunnel

Acceptance Criteria:

• Fix the STATUS check returned by the rest api to support a tunnelled connection.

From looking at the code I think it would be great (if possible) if we can reuse the connections already setup in the MetastoreMappingImpl. That way the status check is a more accurate representation of available connections.

Views error: java.lang.NoClassDefFoundError: com/sun/tools/javac/util/Pair

Issue in handling views that are generated via subqueries. WD will throw: java.lang.NoClassDefFoundError: com/sun/tools/javac/util/Pair

Related to #62

Move RPM GPG settings to a profile

The rpm-maven-plugin GPG passphrase has been configured directly in the plugin. This value is only required for releases but with the current settings snapshots and local compilations require this value or the compilation fails.

This configuration needs to be moved out to Sonatype profile.

Support Wild Cards in DB name

As a SysOps
I want WD to support regex in DB names
So that I can use regexp to whitelist multiple DBs

Acceptance Criteria:

• interpret the db name in the whitelist as a regular expression match accordingly
• Able to whitelist user_abc,user_def like user_.*
• Able to whitelist ALL databases with wildcard .*

Waggle Dance fail to respond if one of the metastore becomes unresponsive

If I have one Primary Metastore and n federated metastore and one of the metastore stop responding then client connecting to Waggle Dance thrift endpoint gets Thrift Exception and unable to query data/metadata of rest of the metastore.

Additional Info:

• WD is running in prefixed mode.

Internal error processing get_table_req

Here is the environment.

• waggle-dance version : 2.3.7 (also tested on 2.3.6)
• waggle-dance configuration
  • database-resolution: PREFIXED
  • primary-meta-store :
    • hive & metastore version : 2.3.0
    • thrift://localhost:9083
  • remote-meta-store
    • hive & metastore version : 2.1.0
    • thrift://remote-ip:9083

I can see the remote db and tables with 'prod_' prefix which I set. (I executed use prod_test;)
But when I execute a select query for a table(select * from test_1;), it makes error.

Here is the logs (and stack trace)

2018-07-25 23:25:29.500  INFO 28387 --- [           main] o.h.v.i.x.ValidationBootstrapParameters  : HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
2018-07-25 23:25:29.921  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@4f4a7090: startup date [Wed Jul 25 23:25:24 UTC 2018]; root of context hierarchy
2018-07-25 23:25:30.031  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations],methods=[GET]}" onto public java.util.List<com.hotels.bdp.waggledance.api.model.AbstractMetaStore> com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.federations()
2018-07-25 23:25:30.033  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations],methods=[POST]}" onto public void com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.add(com.hotels.bdp.waggledance.api.model.AbstractMetaStore)
2018-07-25 23:25:30.033  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations/{name}],methods=[DELETE]}" onto public void com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.remove(java.lang.String)
2018-07-25 23:25:30.033  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations/{name}],methods=[GET]}" onto public com.hotels.bdp.waggledance.api.model.AbstractMetaStore com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.read(java.lang.String)
2018-07-25 23:25:30.035  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2018-07-25 23:25:30.036  INFO 28387 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2018-07-25 23:25:30.111  INFO 28387 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:30.111  INFO 28387 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:30.200  INFO 28387 --- [           main] o.s.w.s.h.SimpleUrlHandlerMapping        : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:31.236  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/beans || /beans.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.238  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-07-25 23:25:31.239  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.240  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/configprops || /configprops.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.241  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.244  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.246  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.247  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-07-25 23:25:31.248  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/health || /health.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(javax.servlet.http.HttpServletRequest,java.security.Principal)
2018-07-25 23:25:31.250  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/info || /info.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.251  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/metrics/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.MetricsMvcEndpoint.value(java.lang.String)
2018-07-25 23:25:31.251  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/metrics || /metrics.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.252  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.254  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-07-25 23:25:31.259  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/loggers/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.LoggersMvcEndpoint.get(java.lang.String)
2018-07-25 23:25:31.259  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/loggers/{name:.*}],methods=[POST],consumes=[application/vnd.spring-boot.actuator.v1+json || application/json],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.LoggersMvcEndpoint.set(java.lang.String,java.util.Map<java.lang.String, java.lang.String>)
2018-07-25 23:25:31.260  INFO 28387 --- [           main] o.s.b.a.e.m.EndpointHandlerMapping       : Mapped "{[/loggers || /loggers.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:32.433  INFO 28387 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-07-25 23:25:32.454  INFO 28387 --- [           main] o.s.c.s.DefaultLifecycleProcessor        : Starting beans in phase 0
2018-07-25 23:25:32.808  INFO 28387 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 18000 (http)
2018-07-25 23:25:32.814  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : Starting WaggleDance on port 48869
Starting WaggleDance on port 48869
2018-07-25 23:25:32.839  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : Starting WaggleDance Server
2018-07-25 23:25:32.846  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : Started the new WaggleDance on port [48869]...
2018-07-25 23:25:32.847  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : Options.minWorkerThreads = 200
2018-07-25 23:25:32.847  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : Options.maxWorkerThreads = 1000
2018-07-25 23:25:32.847  INFO 28387 --- [           main] c.h.b.w.s.MetaStoreProxyServer           : TCP keepalive = true
2018-07-25 23:25:57.209  INFO 28387 --- [pool-4-thread-1] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:25:57.451  INFO 28387 --- [pool-4-thread-1] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:25:57.732  INFO 28387 --- [pool-4-thread-1] c.j.a.a.NamedThreads                     : jcabi-aspects 0.22.6/3f0a1f7 started new daemon thread jcabi-loggable for watching of @Loggable annotated methods
2018-07-25 23:25:57.736  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:25:57.766  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:25:57.766  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:25:57.766  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:25:57.769  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:25:57.769  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:25:58.077  INFO 28387 --- [pool-4-thread-1] c.h.b.w.s.FederatedHMSHandler            : Fetching database prod_test
2018-07-25 23:25:58.085  INFO 28387 --- [pool-4-thread-1] o.h.v.i.x.ValidationXmlParser            : HV000007: META-INF/validation.xml found. Parsing XML based configuration.
2018-07-25 23:25:58.096  INFO 28387 --- [pool-4-thread-1] o.h.v.i.x.ValidationBootstrapParameters  : HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
2018-07-25 23:25:58.114  INFO 28387 --- [pool-4-thread-1] c.h.b.w.s.FederatedHMSHandler            : Mapping is 'prod_'
2018-07-25 23:25:58.185  INFO 28387 --- [pool-4-thread-2] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:25:58.188  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Closed a connection to metastore, current connections: 1
2018-07-25 23:25:58.188  INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient          : Closed a connection to metastore, current connections: 0
2018-07-25 23:25:58.245  INFO 28387 --- [pool-4-thread-2] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:25:58.281  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:25:58.282  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:25:58.282  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:25:58.282  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:25:58.285  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:25:58.285  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:25:58.305  INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler            : Fetching database prod_test
2018-07-25 23:25:58.308  INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler            : Mapping is 'prod_'
2018-07-25 23:25:58.322  INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler            : Fetching database prod_test
2018-07-25 23:25:58.325  INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler            : Mapping is 'prod_'
2018-07-25 23:26:13.054 ERROR 28387 --- [pool-4-thread-2] o.a.h.h.m.RetryingHMSHandler             : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67)
        at com.sun.proxy.$Proxy134.get_table_req(Unknown Source)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
        at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
        at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

2018-07-25 23:26:13.055 ERROR 28387 --- [pool-4-thread-2] o.a.t.ProcessFunction                    : Internal error processing get_table_req

org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67) ~[classes!/:?]
        at com.sun.proxy.$Proxy134.get_table_req(Unknown Source) ~[?:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1) ~[classes!/:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57) ~[classes!/:?]
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1) ~[classes!/:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
        at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479) ~[classes!/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50) ~[classes!/:?]
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) [hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar!/:0.9.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

2018-07-25 23:26:14.066  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Closed a connection to metastore, current connections: 1
2018-07-25 23:26:14.066  INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient          : Closed a connection to metastore, current connections: 0
2018-07-25 23:26:14.067  INFO 28387 --- [pool-4-thread-3] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:26:14.105  INFO 28387 --- [pool-4-thread-3] c.h.b.w.m.m.MetaStoreMappingFactoryImpl  : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:26:14.136  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:26:14.138  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:26:14.138  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:26:14.138  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:26:14.141  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:26:14.141  INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient          : Connected to metastore.
2018-07-25 23:26:14.150 ERROR 28387 --- [pool-4-thread-3] o.a.h.h.m.RetryingHMSHandler             : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67)
        at com.sun.proxy.$Proxy134.get_table_req(Unknown Source)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
        at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
        at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

2018-07-25 23:26:14.150 ERROR 28387 --- [pool-4-thread-3] o.a.t.ProcessFunction                    : Internal error processing get_table_req

org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67) ~[classes!/:?]
        at com.sun.proxy.$Proxy134.get_table_req(Unknown Source) ~[?:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1) ~[classes!/:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57) ~[classes!/:?]
        at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479) ~[classes!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1) ~[classes!/:?]
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
        at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169) ~[jcabi-aspects-0.22.6.jar!/:?]
        at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479) ~[classes!/:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50) ~[classes!/:?]
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441) ~[hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar!/:0.9.3]
        at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) [hive-metastore-2.3.0.jar!/:2.3.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar!/:0.9.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Improve YAML config validation error messaging

As a user of Waggle Dance
I'd like to get some meaningful error when the yml config has errors
So that I can debug it myself

The yml config is sort of fiddly at times and when for instance the federation-server.yml is not indented correctly it fails to start WD and because logging isn't initialized the user gets no feedback.
Reproduce, try to load this config with (notice the extra spaces on the last two lines):

  database-prefix: ''
  name: local
  remote-meta-store-uris: thrift://localhost:9083
federated-meta-stores:
- remote-meta-store-uris: thrift://localhost:9083
    database-prefix: remote_
    name: remote

Acceptance Criteria:

• A bad yml config file is reject by WD with an error message (the more specific the better but an indication which file is wrong is already much better then current situation)

Configure Travis CI

We'd like to build and release via Travis-CI. Configure Travis for CI

Remove io.spring.platform.platform-bom

As a developer of Waggle Dance
I want to ensure that it doesn't use any end of life libraries
So that the code is easier to maintain

Acceptance Criteria

• io.spring.platform.platform-bom removed as a dependency from Waggle Dance and replaced with something more future proof.

Notes
Waggle Dance currently uses Spring's io.spring.platform.platform-bom to manage a number of Spring related depedencies. This is currently planned to be "end of life" by April 2019 (see https://spring.io/blog/2018/04/09/spring-io-platform-end-of-life-announcement). We should replace this with something else (the above blog post suggests using spring-boot-dependencies as a possible solution. We should also keep an eye on the project's official home page (https://platform.spring.io/platform/) to see if any other migrations are suggested. Hopefully a number of other people will have done this and we can learn from their experiences.

Move minimum supported Java version from 7 to 8

As a developer
I want to move Waggle Dance to use Java 8
So that I can benefit from the updated language features and not spend time maintaining backwards compatibility and also to allow certain libraries that no longer support Java 7 to be updated.

Acceptance Criteria

• Java 8 used as jdk.version when building WaggleDance.

Increase code coverage (2)

As a developer
I'd like to be confident my tests can detect refactoring issues
So I can easily make code changes

Acceptance criteria
Code coverage should be at least 80%

Below is a list of classes which could benefit from some extra testing and should be the focus of this ticket. They are arranged (more or less) in increasing test complexity:

ValidationError
FederatedMetaStore
PrimaryMetaStore
MetastoreUnavailableException
TunnelableFactorySupplier
TunnelingMetaStoreClientFactory
NotifyingFederationService
AdvancedPropertyUtils

PrefixBasedDatabaseMappingService
WaggleDanceConfiguration
YamlStorageConfiguration (trivial)
MonitoringConfiguration

FederatedHMSHandlerFactory
FederatedHMSHandler

handle view parse error silently

We are hitting an Hive bug in parsing the views https://issues.apache.org/jira/browse/HIVE-19896
This affects doing among others: desc table, show create table commands via Waggle Dance (only affects federated metastores, primary metastore is ok).

We should ignore the errors so the client is not affected

tunneling: Make strict host checking optional

As a user of Waggle Dance
I'd like to make strict host checking option
So that I can bypass any host checking issues

The JSCH library we use to setup the ssh tunnel can be configure to skip strict host checking. We ran into issue (see #32) where the encryption key wasn't supported and it would be great to have quick workarounds for this. Or at least leave it up to the user to enable or disable strict host checking.

Acceptance Criteria

• configuration flag in the yaml to disable/enable strict host checking per tunnel.

Upgrade to Hive 2.3.0

Upgrade Hive to version 2.3.0 to make it compatible with AWS EMR 5.8.0

DESCRIBE FORMATTED fails on Federated metastore tables

Hive command DESCRIBE FORMATTED failing:

2018-04-03T10:26:49,798 INFO  com.hotels.bdp.waggledance.client.ThriftMetastoreClient:158 - Opened a connection to metastore 'thrift://cloud9-lab-main.us-west-2.hcom-data-lab.aws.hcom:9083', total current connections to all metastores: 3
2018-04-03T10:26:49,798 INFO  com.hotels.bdp.waggledance.client.ThriftMetastoreClient:193 - Connected to metastore.
2018-04-03T10:26:49,802 DEBUG com.hotels.bdp.waggledance.mapping.service.impl.PrefixBasedDatabaseMappingService:173 - Database Name `bix_mobile` maps to metastore with EMPTY_PREFIX
2018-04-03T10:26:49,936 DEBUG com.hotels.bdp.waggledance.mapping.service.impl.PrefixBasedDatabaseMappingService:173 - Database Name `bix_mobile` maps to metastore with EMPTY_PREFIX
2018-04-03T10:26:50,014 DEBUG com.hotels.bdp.waggledance.server.FederatedHMSHandler:168 - #get_foreign_keys('ForeignKeysRequest(parent_db_name:null, parent_tb..54..l_apt_performancemanagement_endpoints_sort)'): thrown javax.validation.ConstraintViolationException(may not be null) out of com.jcabi.aspects.aj.MethodValidator#checkForViolations[189] in 45ms
2018-04-03T10:26:50,018 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler:204 - javax.validation.ConstraintViolationException: may not be null
	at com.jcabi.aspects.aj.MethodValidator.checkForViolations(MethodValidator.java:189)
	at com.jcabi.aspects.aj.MethodValidator.validateMethod(MethodValidator.java:154)
	at com.jcabi.aspects.aj.MethodValidator.beforeMethod(MethodValidator.java:87)
	at com.hotels.bdp.waggledance.mapping.service.impl.MonitoredDatabaseMappingService.databaseMapping(MonitoredDatabaseMappingService.java:48)
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys_aroundBody566(FederatedHMSHandler.java:1384)
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure567.run(FederatedHMSHandler.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
	at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
	at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys_aroundBody568(FederatedHMSHandler.java:1382)
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure569.run(FederatedHMSHandler.java:1)
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
	at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
	at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
	at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys(FederatedHMSHandler.java:1382)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
	at com.sun.proxy.$Proxy132.get_foreign_keys(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
	at com.sun.proxy.$Proxy132.get_foreign_keys(Unknown Source)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_foreign_keys.getResult(ThriftHiveMetastore.java:12933)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_foreign_keys.getResult(ThriftHiveMetastore.java:12917)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)

2018-04-03T10:26:50,018 ERROR org.apache.thrift.server.TThreadPoolServer:297 - Error occurred during processing of message.
javax.validation.ConstraintViolationException: may not be null
	at com.jcabi.aspects.aj.MethodValidator.checkForViolations(MethodValidator.java:189) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.jcabi.aspects.aj.MethodValidator.validateMethod(MethodValidator.java:154) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.jcabi.aspects.aj.MethodValidator.beforeMethod(MethodValidator.java:87) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.hotels.bdp.waggledance.mapping.service.impl.MonitoredDatabaseMappingService.databaseMapping(MonitoredDatabaseMappingService.java:48) ~[classes!/:?]
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys_aroundBody566(FederatedHMSHandler.java:1384) ~[classes!/:?]
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure567.run(FederatedHMSHandler.java:1) ~[classes!/:?]
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
	at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57) ~[classes!/:?]
	at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47) ~[classes!/:?]
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys_aroundBody568(FederatedHMSHandler.java:1382) ~[classes!/:?]
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure569.run(FederatedHMSHandler.java:1) ~[classes!/:?]
	at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
	at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169) ~[jcabi-aspects-0.22.6.jar!/:?]
	at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_foreign_keys(FederatedHMSHandler.java:1382) ~[classes!/:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_171]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_171]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_171]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_171]
	at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50) ~[classes!/:?]
	at com.sun.proxy.$Proxy132.get_foreign_keys(Unknown Source) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_171]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_171]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_171]
	at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_171]
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-metastore-2.3.0.jar!/:2.3.0]
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-metastore-2.3.0.jar!/:2.3.0]
	at com.sun.proxy.$Proxy132.get_foreign_keys(Unknown Source) ~[?:?]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_foreign_keys.getResult(ThriftHiveMetastore.java:12933) ~[hive-metastore-2.3.0.jar!/:2.3.0]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_foreign_keys.getResult(ThriftHiveMetastore.java:12917) ~[hive-metastore-2.3.0.jar!/:2.3.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar!/:0.9.3]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar!/:0.9.3]
	at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) ~[hive-metastore-2.3.0.jar!/:2.3.0]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar!/:0.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) [?:1.7.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) [?:1.7.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.7.0_171]
2018-04-03T10:28:44,323 DEBUG com.hotels.bdp.waggledance.server.TTransportMonitor:66 - Releasing disconnected sessions
2018-04-03T10:28:44,326 INFO  com.hotels.bdp.waggledance.client.ThriftMetastoreClient:224 - Closed a connection to metastore, current connections: 2
2018-04-03T10:28:44,326 INFO  com.hotels.bdp.waggledance.client.ThriftMetastoreClient:224 - Closed a connection to metastore, current connections: 1
2018-04-03T10:28:44,327 INFO  com.hotels.bdp.waggledance.client.ThriftMetastoreClient:224 - Closed a connection to metastore, current connections: 0








2018-04-03T10:33:44,323 DEBUG com.hotels.bdp.waggledance.server.TTransportMonitor:66 - Releasing disconnected sessions

Describe Formatted failure occurs on the primary and the federated metastore tables:
Config

primary-meta-store:
  access-control-type: READ_ONLY
  database-prefix: ''
  name: primary
  remote-meta-store-uris: thrift://localhost:9083 
federated-meta-stores:
- access-control-type: READ_ONLY
  remote-meta-store-uris: thrift://localhost:9083   
  mapped-databases: []
  database-prefix: waggle_test_
  name: secondary

Output

hive> show databases;
OK
bdp
default
waggle_test_bdp
waggle_test_default
Time taken: 1.307 seconds, Fetched: 4 row(s)
hive> describe formatted bdp.test;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException
hive> describe formatted waggle_test_bdp.test;
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.thrift.transport.TTransportException
hive> 

Documentation for installation is confusing

Issues

• Sets expectation that I can directly install from a TGZ download at https://github.com/HotelsDotCom/waggle-dance/releases but this is not true
• Does not describe how to build from TGZ, moves straight on to configuration making the assumption that the *-bin.tgz has been summoned from somewhere.
• RPM installation is not in an 'RPM' section, or even the Installation section, but is instead hidden away in the 'Running as a Service` section.
• Describes building an RPM, but does not describe where the RPM is generated, so makes it quite hard to locate: waggle-dance-rpm/target/rpm/waggle-dance-rpm/RPMS/noarch/
• Does not explicitly describe in which directory the service is installed to, forcing the user to context switch immediately after install: /opt/waggle-dance/
• Paths in README italicised but should be monospaced.

I'd suggest documenting RPM end to end, then having a section on the differences for TGZ. You can signpost this at the start of the installation section.

Upgrade dependency and plugin versions

As a developer
I want WaggleDance using the latest versions of various dependencies and plugins
So that I benefit from new features, bug fixes etc.

Acceptance Criteria

• Build and all tests passing with any required changes for below version upgrades.

• The following upgraded to their latest (or indicated) versions in waggle-dance-parent:

  • cobertura.version
  • cobertura.maven.plugin.version
  • maven.release.plugin.version
  • hive.version -> 2.3.3
  • dropwizard.version
  • aspectj-maven-plugin.version -> check whether this is actually still in use? if not remove it
  • aspectj.version -> same for this
  • beeju.version
  • hotels-oss-parent

• The following upgraded to their latest (or indicated) versions in waggle-dance module:

  • maven-assembly-plugin.version
  • maven-install-plugin.version
  • maven-deploy-plugin.version
  • Ensure the TGZ generated is as close to identical as before this change

• The following upgraded to their latest (or indicated) versions in waggle-dance-core module:

  • mockito.version
  • powermock.version
  • jcabi-aspects.version -> check whether this is actually still in use? if not remove it
  • hcommon-ssh.version

• The following upgraded to their latest (or indicated) versions in waggle-dance-rpm module:

  • rpm.maven.plugin.version
  • Ensure the RPM generated is as close to identical as before this change

mapped-databases set to [] on server restart if no initial value provided

Reported by an internal user, I'm guessing this is the sequence of events that causes a bug:

1. Create a federation yml file and for the "federated-meta-stores" completely omit the "mapped-databases" and "overwrite-config-on-shutdown" configuration so these get their default values.
2. Start and then stop Waggle Dance.
3. Check the configuration file, if the issue is reproduced then "mapped-databases" will now be in the config file and set to [] which now means no databases are being federated (instead of the default of all of them).

To fix this we need to ensure that if no mapped databases have been configured that the update of the config file on shutdown doesn't put an empty list into the file.

Select * from <VIEW> on federated metastore does not work.

If I have a View View1 over Table T1 and T2 under hive database db1 in federated metastore FM1. Following command fails with Table Not Found Exception

Select * from fm1_db1.view1 ;
This will fail with Table T1 & T2 Not Found.

Additional Info:

• WD is running in prefixed mode.

Improve memory usage in federated metastore request/response forwards

Spike

Overview
At the moment the database mappings create copies of the Thrift objects it processes. Some of these objects can be huge and the creation of copy will demand memory for the request. As we only need the original object to determine which megastore will process the request we could mutate the current object instead of making a copy.

Acceptance criteria
Evaluate memory usage before and after the change.

Tunneling mechanism in Waggle Dance not working

A bug in the passing of HiveConf's meant that the tunnel was being misconfigured

Configure and use the HADOOP_USER_NAME in WD for all requests

As a user of WD
I'd like to issue all requests to a federated metastore as one user
So that I can implement authorization for that one user and not for all clients

Acceptance Criteria:

• Waggle Dance configuration allows configuration for HADOOP_USER_NAME (Decide on property name, decide for all metastores or per metastore configuration?)
• If the HADOOP_USER_NAME is set in the WD configuration then it is used and overrides any client set username.

Example my hive cli client issues all requests as user 'hadoop'. If the WD HADOOP_USER_NAME is set then the user 'hadoop' is ignored and whatever value WD is configured with is used instead.

Notes

Might help with what is discussed in https://groups.google.com/d/msg/waggle-dance-user/rA3JPimh94A/0q0ZXasjAwAJ

Add generic whitelisting feature for every metastore mappings

At the moment the whitelisting logic is spread across the different services and it feels like these checks could be done just before the request hits the handler.

We would like to centralize the whitelisting logic in one place and make sure all requests go through the checks before they are processed. These should also simplify the mapping services.

Configuration property name 'remote-meta-store-uris' misleading

The config property name remote-meta-store-uris implies that the metastore is always remote. However, this is not necessarily the case. meta-store-uris would possibly be better.

Status field missing from rest api

Looks like since WD 2.0.0 we lost the status field from the /api/admin/federations api rest endpoint.

Replace usage of Cobertura with Jacoco

As a developer
I want to use Jacoco for code coverage rather than Cobertura since the latter is pretty much a dead project (and doesn't handle Java 8 code)
So that I have a future proof code coverage solution.

Acceptance Criteria

• Usage of Cobertura in Travis replaced with Jacoco
• Coveralls reports checked to still work

Increase code coverage (1)

As a developer
I'd like to be confident my tests can detect refactoring issues
So I can easily make code changes

Acceptance criteria
Code coverage should be at least 80%

Below is a list of classes which could benefit from some extra testing and should be the focus of this ticket. They are arranged (more or less) in increasing test complexity:

WaggleDanceException
AbstractMetaStore
Federations
CloseableThriftHiveMetastoreIfaceClientFactory
GrammarUtils
MonitoredDatabaseMappingService
AccessControlHandlerFactory

GraphiteConfiguration

StaticDatabaseMappingService
CommonVFSResource

TunnelingMetaStoreClientFactory
WaggleDanceRunner -> could at least test the builder part of this
DefaultMetaStoreClientFactory
ThriftMetastoreClient

Update hive dependency to 2.1.1

To match the metastore we're talking to

Cleanup/Refactor FederatedHMSHandler and MappingEventListener

As a developer
I want to ensure that code is logical and clean
so that I spend less time on maintenance

Acceptance Criteria
TBD based on extracting exactly what we want to from notes below

Notes
Below is a brain dump of discussions we've had on the issue that hopefully we can extract proper acceptance criteria out of:

Notes from handover discussion with @ddcprg:

In FederatedHMSHandler we have a variable called databaseMappingService but the type is MappingEventListener - this interface extends 2 others (service and handler) and is thus both, this is confusing that its handling two different responsibilities. Service and Listener should probably be different classes, a listener and a service both pointing to same thing or should we refactor the code out into two separate classes. Perhaps we should have a listener which knows about the service and tells it do things?

Comments from @patduin in Slack:

Probably a good idea if the MappingServices are split in a MappingService and a corresponding MappingServiceListener. I find classes that implement their own listeners usually confusing as well but it is often convenient. Not sure if it is easy to do though, the listen methods need access to the fields, dunno maybe worth a shot. For example PrefixBasedDatabaseMappingService
will get split? The onX methods have synchronisation locks and use fields of the service you somehow have to replace that.

enable READ_WRITE support for federated metastores

As a user of WD
I like to run DDL on federated metastores
So that I can run ETL againt primary metastore and update federated central datalake.

Acceptance Criteria:
ability to configure access-control-type option for federated metastores with atleast following possible values READ_ONLY, READ_WRITE.

Waggle Dance adds double backticks in certains places when transforming view queries.

example:

select a from (select b from c) as `d.c`

Is transformed by our transformation into:

select a from (select b from c) as ``d.c``

show create table on the view with that select will return the double backticked value.

Allow users to configure a session timeout for SSH tunnels

The admin endpoint hangs when the SSH tunnels uses an unreachable host.

Primary metastore should default to read/write

I didn't provide an access-control-type for my primary-meta-store. However, when I restarted the service it wrote out the following configuration:

primary-meta-store:
  access-control-type: READ_ONLY

I think this should have defaulted to some variant of READ_AND_WRITE_* for the primary, possibly:READ_AND_WRITE_AND_CREATE.

Restrict Read access on Federated Metastore in PREFIXED mode

As a SysOps
I want to restrict Read access to Databases in Federated Metastore when using PREFIXED mode
So that I can control visibility of databases

Acceptance Criteria:

• If Federated Metastore has three databases - db1, db2 and db3, I can only make db1 readable for users.