akolb1/gometastore

Go Client for Hive Metastore

gometastore

Tools for accessing Hive Metastore over thrift API.

The repository provides a set of tools for accessing Hive Metastore (HMS) directly using its Thrift interface rather then going via beeline. It can be use to explore and troubleshoot HMS and to develop other scripts and tools that need to access it.

Kerberos support

None of these tools work in kerberos-enabled cluster.

You should consider using Java-based toolkit if you need support for Kerberos.

Installation

Make sure that you have an up-to-date GO environment. Currently Go 1.11 or higher is required.

On MacOS this can be as easy as running

brew install go

Building:

go get -u github.com/akolb1/gometastore/...

This command will install tools in ~/go/bin directory.

• hmstool - CLI for HMS client

• GO CLient library

• REST front-end for Hive Metastore

• HMS Benchmarks

gometastore's Issues

Wrong info is printed on connection error

When HMS_HOST has port associated with it, the error message shows:

2018/06/01 15:59:03 failed to open connection to host:9084:9083: dial tcp 10.16.0.83:9084: connect: connection refused

Note that the default port is tacked to the end which is wrong.

Create View operation

Hello @akolb1,

Does this library support creating views in hive? I only see a function to create tables.

Thank you!

failed to run on go version 1.19

Allow custom HMS ports

Support table renames

How do i retrieve all columns with their data type of a table from metastore?

Hi
I am looking to extract all columns of a table using this library. But not able to locate any helpful method for it.
Any help?

hmsbench: add test for getTableObjectsByName

hmstool: Provide way to use server-side filtering for databases and tables.

hmstool allows for glob-style matching of database and table names, but the matching is performed on the client side. For instances with huge number of databases/tables this can become problematic, so it would be good to push matching to the server.

This will be done with the new -M flag for listing dbs and tables. When specified it will use server-side matching.

hmstool: provide way to show tables/partitions up to a specified timestamp

hmstool - support restoring objects from json dump

hmstool should provide a way to list files in a table or partition

It would be sometimes useful to list HDFS files belonging to a table or partition.

We can do this with a special option that fetches file locations from HDFS.

Provide PartitionBuilder

GetAllDatabases returns error 'EOF'

Hi I am running a remote Hive metastore on ec2 machine backed up by MySQL RDS as database.
HiveServer2 service is running on 10002 port.
I already created sample database in hive and a sample table inside it manually.
gometastore library is able to create a connection to Metastore but fails while getting result from GetAllDataases and return EOF error.

package main
import	(
    "log"
    "github.com/akolb1/gometastore/hmsclient"
)

func main() {
    client, err := hmsclient.Open("localhost", 10002)
    if err != nil {
	log.Println("Error when creating connection")
        log.Println(err)
    }
    databases, err := client.GetAllDatabases()
    log.Println(client)
    if err != nil {
	log.Println("Error inside GetAllDatabases")
        log.Println(err)
    }
    for _, d := range databases {
	log.Println(d)
    }
}

Hive CLI shows 2 dbs:

    > show databases;
OK
default
testdb
Time taken: 0.018 seconds, Fetched: 2 row(s)

Any help what could be wrong here?

Regenerate `hive_metastore` because of updated `TProtocol`

Since Apache thrift Golang TProtocol is updated and all the methods are expecting context we need to regenerate ourhive_metastore

report errors from failed client operations