Web UI for Trino, Hive and SparkSQL
License: Apache License 2.0
Yanagishima is an open-source Web application for Trino, Hive and Spark.
Visit the official web site for more information.
git clone https://github.com/yanagishima/yanagishima.git
cd yanagishima
git checkout -b [version] refs/tags/[version]
./gradlew distZip
cd build/distributions
unzip yanagishima-[version].zip
cd yanagishima-[version]
vim config/application.yml
nohup bin/yanagishima-start.sh >y.log 2>&1 &
bin/yanagishima-shutdown.sh
how to add user param,password param when i run query?
Bug report
yanagishima version: 21.0
presto version: 318
conf/yanagishima.properties:
# yanagishima web port
jetty.port=8080
# backend RDMS
database.type=mysql
mysql.host=yanagishima-mysql
mysql.port=3306
mysql.database=yanagishima
mysql.user=yanagishima
mysql.password=xxxxxxxxxxxxxxxxxxx
# 30 minutes. If presto query exceeds this time, yanagishima cancel the query.
presto.query.max-run-time-seconds=1800
# 1GB. If presto query result file size exceeds this value, yanagishima cancel the query.
presto.max-result-file-byte-size=1073741824
# you can specify freely. But you need to specify same name to presto.coordinator.server.[...] and presto.redirect.server.[...] and catalog.[...] and schema.[...]
presto.datasources=your-presto
auth.your-presto=false
# presto coordinator url
presto.coordinator.server.your-presto=http://presto-coordinator:8080
# almost same as presto coordinator url. If you use reverse proxy, specify it
presto.redirect.server.your-presto=http://presto.xxxxxxxx.test:30080/ui
# presto catalog name
catalog.your-presto=hive
# presto schema name
schema.your-presto=default
# if query result exceeds this limit, to show rest of result is skipped
select.limit=500
# http header name for audit log
audit.http.header.name=some.auth.header
use.audit.http.header.name=false
# limit to convert from tsv to values query
to.values.query.limit=500
# authorization feature
check.datasource=false
hive.jdbc.url.your-hive=jdbc:hive2://hiveserver2:10000/default
hive.jdbc.user.your-hive=yanagishima
hive.jdbc.password.your-hive=yanagishima
hive.query.max-run-time-seconds=3600
hive.query.max-run-time-seconds.your-hive=3600
resource.manager.url.your-hive=http://resourcemanager:8088
sql.query.engines=presto,hive
hive.datasources=your-hive
hive.disallowed.keywords.your-hive=insert,drop
# 1GB. If hive query result file size exceeds this value, yanagishima cancel the query.
hive.max-result-file-byte-size=1073741824
hive.setup.query.path.your-hive=/usr/local/yanagishima/conf/hive_setup_query_your-hive
JVM version (java -version): openjdk version "11.0.3" 2019-04-16 LTS
Server OS version (uname -a if on a Unix-like system): Linux yanagishima-6d84dd6667-sl6q9 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Client OS version: Darwin w022581712182m.local 17.7.0 Darwin Kernel Version 17.7.0: Sun Jun 2 20:31:42 PDT 2019; root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64 i386 MacBookPro14,3 Darwin
Browser: Chrome
Description of the problem including expected versus actual behavior:
When accessing yanagishima UI after haven't accessing the UI for one day,
Expected: Display UI without error popup.
Actual: Error popup "Please contact admin" displays
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):
Server log
2019/08/27 03:05:27.805 +0000 ERROR [RichSQLException] [Yanagishima] SQLException: No operations allowed after connection closed. SELECT engine, query_id, fetch_result_time_string, query_string FROM query WHERE engine='presto' and datasource='your-presto' and query_id IN () [] [1/36290]
2019/08/27 03:05:27.806 +0000 WARN [ServletHandler] [Yanagishima] /query
me.geso.jdbcutils.UncheckedRichSQLException: SQL Exception: No operations allowed after connection closed.:SELECT engine, query_id, fetch_result_time_string, query_string FROM query WHERE engine='presto' and datasource='your-presto' and query_id IN ()([])
at me.geso.tinyorm.TinyORM.prepareStatement(TinyORM.java:146)
at me.geso.tinyorm.TinyORM.searchBySQL(TinyORM.java:294)
at me.geso.tinyorm.TinyORM.searchBySQL(TinyORM.java:281)
at yanagishima.servlet.QueryServlet.doPost(QueryServlet.java:142)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:38)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: me.geso.jdbcutils.RichSQLException: SQL Exception: No operations allowed after connection closed.:SELECT engine, query_id, fetch_result_time_string, query_string FROM query WHERE engine='presto' and datasource='your-presto' and query_id IN ()([])
at me.geso.jdbcutils.UncheckedRichSQLException.<init>(UncheckedRichSQLException.java:13)
... 37 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1192)
at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1187)
at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4067)
at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4036)
at me.geso.tinyorm.TinyORM.prepareStatement(TinyORM.java:140)
... 36 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 62,523,444 milliseconds ago. The last packet sent successfully to the server was 62,523,444 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring an
d/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3751)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2512)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2011)
at me.geso.tinyorm.TinyORM.searchBySQL(TinyORM.java:296)
... 35 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3733)
... 41 more
Provide screenshots (if relevant):
No screen shot. Sorry.
I think it will be resolved by adding "autoReconnect=true" to mysql url.
Thanks in advance.
Hi,
Query result shows only 500 records, which is configurable. However, if the table has more records then it .better to have pagination feature so that the browser won't crash and the user can still continue to view data.
Thanks,
S.Shyam Kumar
yanagishima version: think this is 21.0
presto version: don't know
conf/yanagishima.properties:
JVM version (java -version):
Server OS version (uname -a if on a Unix-like system):
Client OS version MacOS Mojave 10.14.6
Browser: Chrome
Description of the problem including expected versus actual behavior:
Expected behavior: clicking 'Download' on a result downloads a csv or tsv of the result
Actual behavior: clicking 'Download' on a result throws 502 Bad Gateway error
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):
Provide screenshots (if relevant):
Hi
I get that error while starting the tool:
I have it in both V10 and V12 and both HIVE 1.2.1 and HIVE 2.1.0
Any help appreciated !
regards,
ERROR [HiveServlet] [Yanagishima] com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.sameThreadExecutor()Lcom/google/common/util/concurrent/ListeningExecutorService;
at org.apache.curator.framework.listen.ListenerContainer.addListener(ListenerContainer.java:40)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:246)
at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.configureConnParams(ZooKeeperHiveClientHelper.java:62)
at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:509)
at org.apache.hive.jdbc.Utils.parseURL(Utils.java:429)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:134)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
yanagishima version:17.0
presto version:0.213
JVM version (java -version):1.8.0_111
Server OS version (uname -a if on a Unix-like system):
2.6.32-431.29.2.el6.x86_64 #1 SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Client OS version:win10
Browser:chrome 67.0.3396.62
Description of the problem including expected versus actual behavior:
the publish button is invalid in my browser
Steps to reproduce:
1.run a query
2.click the Publish button
3.nothing happened
Provide logs (if relevant):
debug and find this exception:
TypeError: Cannot read property 'writeText' of undefined at http://10.250.100.49:8888/static/js/app.84b92e77a1367ea5c61c.js:1:72953
code: web\src\components\tabs\TabResult.vue: line 396
if (ua.indexOf('chrome') !== -1) {
navigator.clipboard.writeText(`${location.protocol}//${location.host}${path}`)
}navigator.clipboard reutnrs undefined
i tried to fix the code like this,just for suggestion:
if (ua.indexOf('chrome') !== -1 && navigator.clipboard) {
navigator.clipboard.writeText(`${location.protocol}//${location.host}${path}`)
}iv find some info here ,hope it will be helpful :
https://stackoverflow.com/questions/51805395/navigator-clipboard-is-undefined
Demand：
The result size of my query is more than 1GB。
the default property value is 1073741824
Result：
The status of my query is failed
Error: java.NumberFormatException: For input string：“10737418240”
How should I do? Is the property value cannot modified?
When click Info button, it shows
HTTP ERROR 404
Problem accessing /query.html. Reason:
Not Found
The url is /query.html?20190831_000317_00007_9bkkp
that should be /ui/query.html?20190831_000317_00007_9bkkp
Hi,
Looks awesome! but for some reason I can get it to run.
For my reasons I didn't enclose the host and port, but it is working in a different framework.
I described the issue below.
Thanks!
Roee
yanagishima version: 3.0
presto version: couldn't find how to check
conf/yanagishima.properties:
jetty.port=8080
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=presto1
presto.coordinator.server.presto1=WORKING_HOST:PORT
presto.redirect.server.presto1=WORKING_HOST:PORT
catalog.presto1=hive
schema.presto1=default
schema.presto1=default
select.limit=500
audit.http.header.name=some.auth.header
JVM version (java -version):
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
Server OS version (uname -a if on a Unix-like system):
Linux 3.19.0-74-generic #82~14.04.1-Ubuntu SMP Fri Oct 21 15:43:47 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Client OS version
localhost
Browser:
chromium
Description of the problem including expected versus actual behavior:
when opening main page,
which for some reason is "http://localhost:8080/#datasource=&tab="
with missing parameters
there is an error call to /datasource
with the following message:
HTTP ERROR: 500
Problem accessing /datasource. Reason:
java.lang.RuntimeException: Missing required header 'X-yanagishima-datasources'.
Steps to reproduce:
Provide logs (if relevant):
Provide screenshots (if relevant):
yanagishima version:
presto version: 0.170
conf/yanagishima.properties:
jetty.port=9000
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=datalake
presto.coordinator.server.datalake=http://ip-172-31-16-250.ec2.internal:8889
presto.redirect.server.datalake=http://ip-172-31-16-250.ec2.internal:8889
catalog.datalake=hive
schema.datalake=default
select.limit=500
audit.http.header.name=some.auth.header
JVM version (java -version):
1.8.0
Server OS version (uname -a if on a Unix-like system):
Linux ip-172-31-16-250 4.4.35-33.55.amzn1.x86_64
Client OS version
4.0
Browser:
Chrome
Description of the problem including expected versus actual behavior:
The UI is not reading any catalogs/schema and hence unable to make any queries!
Steps to reproduce:
I have an AWS EMR setup with one Master & 1 Core with the following setup:
Presto: Presto 0.170 with Hadoop 2.7.3 HDFS and Hive 2.1.1 Metastore
Installed the UI on Master
Created table on Hive with links to JSON data on S3
Logged on to presto-cli and was able to do a simple select
Load UI and see if it is able to detect the default schema and tables
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):
2017/05/31 12:58:49.451 +0000 INFO [YanagishimaServer] [Yanagishima] Loading yanagishima properties file
2017/05/31 12:58:50.134 +0000 INFO [log] [Yanagishima] Logging initialized @952ms
2017/05/31 12:58:50.290 +0000 INFO [YanagishimaServer] [Yanagishima] Yanagishima Server started...
2017/05/31 12:58:50.292 +0000 INFO [Server] [Yanagishima] jetty-9.3.13.M0
2017/05/31 12:58:50.724 +0000 INFO [ContextHandler] [Yanagishima] Started o.e.j.s.ServletContextHandler@5e7cd6cc{/,file:///home/hadoop/yanagishima-4.0/web/,AVAILABLE}
2017/05/31 12:58:50.733 +0000 INFO [ServerConnector] [Yanagishima] Started ServerConnector@7b4c50bc{HTTP/1.1,[http/1.1]}{0.0.0.0:9000}
2017/05/31 12:58:50.733 +0000 INFO [Server] [Yanagishima] Started @1552ms
2017/05/31 12:58:50.733 +0000 INFO [YanagishimaServer] [Yanagishima] Yanagishima Server running port 9000.
2017/05/31 12:59:03.886 +0000 ERROR [PrestoServlet] [Yanagishima] No value present
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at yanagishima.config.YanagishimaConfig.getPrestoCoordinatorServer(YanagishimaConfig.java:17)
at yanagishima.service.PrestoServiceImpl.getStatementClient(PrestoServiceImpl.java:268)
at yanagishima.service.PrestoServiceImpl.doQuery(PrestoServiceImpl.java:97)
at yanagishima.servlet.PrestoServlet.lambda$doPost$4(PrestoServlet.java:96)
at java.util.Optional.ifPresent(Optional.java:159)
at yanagishima.servlet.PrestoServlet.doPost(PrestoServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
Provide screenshots (if relevant):
Describe the feature:
yanagishima version:12.0
presto version:0.202
conf/yanagishima.properties:
jetty.port=38080
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=dev
auth.your-presto=false
presto.coordinator.server.dev=http://node101:28080
presto.redirect.server.dev=http://node101:28080
catalog.dev=hive
schema.dev=default
select.limit=500
audit.http.header.name=some.auth.header
use.audit.http.header.name=false
to.values.query.limit=500
check.datasource=false
hive.jdbc.url.your-hive=jdbc:hive2://node101:10000/default;auth=noSasl
hive.jdbc.user.your-hive=root
hive.jdbc.password.your-hive=root
hive.query.max-run-time-seconds=3600
hive.query.max-run-time-seconds.your-hive=3600
resource.manager.url.your-hive=http://node101:8088
sql.query.engines=presto
hive.datasources=your-hive
hive.disallowed.keywords.your-hive=insert,drop
hive.max-result-file-byte-size=1073741824
hive.setup.query.path.your-hive=/opt/app/hive/conf/
cors.enabled=false
JVM version (java -version):1.8.0_151
Server OS version (uname -a if on a Unix-like system): CentOS 7.2
Client OS versionCENTOS 7.2
Browser:Google Chrome v63
Description of the problem including expected versus actual behavior: I'm not sure if this is a bug or feature. So, I post as a bug first. In my environment, the presto was deploy in internal network. The server counldn't access Internet. When I visit the yanagishima service by browser, it shows blank page. The reason is the browser can't get the ccs or js files. For example, the URL in the "link href=" line. Is it possbile include these js/css files in next releases? Such as, ace.js, bootstrap.min.js, and etc.
Steps to reproduce: put the service in internal network
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):NA
Provide screenshots (if relevant):NA
yanagishima version: 4.0
presto version: 0.175
Description of the problem including expected versus actual behavior:
if a schema been selected , the schema list will not refresh while catalog changed.
Steps to reproduce:
can you add presto.disallowed.keywords parameter?
yanagishima version:
11
presto version:
0.196
conf/yanagishima.properties:
jetty.port=7080
presto.datasources=ua212
presto.coordinator.server.ua212=http://xxxx:7070
catalog.ua212=hive
schema.ua212=default
sql.query.engines=prestoJVM version (java -version):
jdk1.8.0_121
Browser:
chrome 65.0.3325.162
Description of the problem including expected versus actual behavior:
I start yanagishima and i can see web ui, but when i click TreeView button get error:
There is no db info and table schemas.
2018/03/21 13:04:19.399 +0800 ERROR [PrestoServlet] [Yanagishima] java.sql.SQLException: Query failed (#20180321_050419_00012_zgwnw) in ua212: Query did not match any selection rule
yanagishima.exception.QueryErrorException: java.sql.SQLException: Query failed (#20180321_050419_00012_zgwnw) in ua212: Query did not match any selection rule
at yanagishima.service.PrestoServiceImpl.resultsException(PrestoServiceImpl.java:350)
at yanagishima.service.PrestoServiceImpl.getPrestoQueryResult(PrestoServiceImpl.java:237)
at yanagishima.service.PrestoServiceImpl.doQuery(PrestoServiceImpl.java:106)
at yanagishima.servlet.PrestoServlet.lambda$doPost$5(PrestoServlet.java:106)
at java.util.Optional.ifPresent(Optional.java:159)
at yanagishima.servlet.PrestoServlet.doPost(PrestoServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:38)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: Query failed (#20180321_050419_00012_zgwnw) in ua212: Query did not match any selection rule
... 40 more
Caused by: com.facebook.presto.spi.PrestoException: Query did not match any selection rule
at com.facebook.presto.execution.resourceGroups.InternalResourceGroupManager.selectGroup(InternalResourceGroupManager.java:285)
at com.facebook.presto.execution.resourceGroups.InternalResourceGroupManager.submit(InternalResourceGroupManager.java:122)
at com.facebook.presto.execution.SqlQueryManager.createQuery(SqlQueryManager.java:489) Hi all , a question for help please!
Can I set the path of query result to Disk instead of the default path ( yanagishima/result/presto/) ? if yes , how can I do that ?
i fix it by this way,
Iterable<List<Object>> tempData = null;
while (client.isValid() ) {
if(client.current().getNextUri() != null) {
client.advance();
}else
break;
if(client.current().getData()!=null)
tempData = client.current().getData();
}
...
yanagishima version: 17.0
presto version: 0.215
conf/yanagishima.properties:
jetty.port=8090
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=your-presto
presto.coordinator.server.your-presto=http://server:8080
catalog.your-presto=hive
schema.your-presto=default
select.limit=500
audit.http.header.name=some.auth.header
to.values.query.limit=500
check.datasource=false
hive.jdbc.url.your-hive=jdbc:hive2://server:10000/default;
hive.jdbc.user.your-hive=hive
hive.jdbc.password.your-hive=password
hive.query.max-run-time-seconds=3600
hive.query.max-run-time-seconds.your-hive=3600
resource.manager.url.your-hive=http://server:8088
sql.query.engines=presto,hive
hive.datasources=your-hive
hive.max-result-file-byte-size=1073741824
hive.setup.query.path.your-hive=/usr/local/yanagishima/conf/hive_setup_query_your-hive
cors.enabled=false
JVM version (java -version): openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
Server OS version (uname -a if on a Unix-like system): CentOS 3.10.0-957.1.3.el7.x86_64 #1 SMP
Client OS version
Browser: Chrome - Version 71.0.3578.98 (Official Build) (64-bit)
Description of the problem including expected versus actual behavior: Cannot execute multiple statements separated by ";"
Steps to reproduce: Example:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, history etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):
Provide screenshots (if relevant):
Describe the feature:
yanagishima can not select hive view.
yanagishima version:
version: 11.0
presto version:
vesion: 0.205
conf/yanagishima.properties:
jetty.port=8889
presto.datasources=prod_offline
presto.coordinator.server.prod_offline=http://xxxx:8888
catalog.prod_offline=hive
schema.prod_offline=dw
sql.query.engines=prestoJVM version (java -version):
java version "1.8.0_60"
conf:
jetty.port=8082
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=presto1
presto.coordinator.server.presto1=http://node1:9090
presto.redirect.server.presto1=http://node1:9090
catalog.presto1=hive
schema.presto1=default
select.limit=500
audit.http.header.name=some.auth.header
to.values.query.limit=500
check.datasource=false
2017/06/09 17:07:59.193 +0800 WARN [ServletHandler] [Yanagishima] /query
java.lang.RuntimeException: Missing required parameter 'datasource'.
at yanagishima.util.HttpRequestUtil.getParam(HttpRequestUtil.java:11)
at yanagishima.servlet.QueryServlet.doGet(QueryServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:22)
I am using yanagishima 10 / presto 195.
I have setup the ssl connection with user and password. Then the user need to login, and this works great.
The concern is about the query list panel: all users are able to read any result from other users, and worst, get their results.
Then the question is, have you in mind an update that would secure the access to other users queries at the backend level ?
Thanks
Our server is filling up with result files.
I think the following features would be awesome.
Thanks!
Hello, first of all thanks you for this awesome project, it just works! (using it with presto 0.161),
A javascript error:
Uncaught TypeError: Cannot read property 'totalDrivers' of undefined
at Array.<anonymous> (yanagishima.js:993)
at Array._a.data (d3.min.js:3)
at renderDoneQueries (yanagishima.js:992)
at yanagishima.js:846
at d3.min.js:1
at Object.<anonymous> (d3.min.js:1)
at Object.t (d3.min.js:1)
at XMLHttpRequest.u (d3.min.js:1)
Describe the feature:
When I start server, web ui like this but no error logs:
HTTP ERROR: 404
Problem accessing /. Reason:
Not Found
Powered by Jetty:// 9.3.13.M0yanagishima version:
3.0
presto version:
0.149
conf/yanagishima.properties:
jetty.port=8081
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=presto1
presto.coordinator.server.presto1=http://:8080
presto.redirect.server.presto1=http://:8080
catalog.presto1=xxxx
schema.presto1=xxxx
select.limit=500
audit.http.header.name=httpJVM version (java -version):
java version "1.8.0_60"
Server OS version (uname -a if on a Unix-like system):
Linux xxxx-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Describe the feature: Auto-complete / SQL dialect support
yanagishima version: 10
presto version: 0.187
JVM version (java -version): openjdk version "1.8.0_151"
Browser: chome
Hi
I am querying a presto coordinator. I am able to scroll over tables and schemas in the treeview. The queries I am running are succeeding (from presto admin serveur) and I am able to read the tsv in the result filesystem.
However, the webclient does not show the results (in firefox or chrome), and I get that error in the console of chrome:
vue.min.js:6 TypeError: Date is not valid
at Nc (sugar.min.js:31)
at Wc (sugar.min.js:37)
at format (sugar.min.js:97)
at Date.<anonymous> (sugar.min.js:13)
at extractDate (index.js:2659)
at pt.eval (eval at yi (vue.min.js:7), <anonymous>:2:19170)
at pt.e._render (vue.min.js:7)
at pt.r (vue.min.js:6)
at So.get (vue.min.js:7)
at So.run (vue.min.js:7)
C @ vue.min.js:6
Apparently a problem with sugar.js ?
Thanks
How do I upgrade from version 18 to 20?
In the readme, it says:
wget http://.../yanagishima-yyy.zip
what is the url for zip file?
Hi,
If any user(like Shyam) logs and run query, then his query history is visible is stored in the browser. When another user(like Amol) login, this history is available to another user(like Amol, can also view Shyam table ) too. This should not be the case. Each user should have access to his own bookmark and history only.
I'm using 15.0verson of yanagishima
thanks,
S.Shyam Kumar
I use nginx for reverse proxy ,nginx config is
server {
listen 80;
server_name localhost;
include /etc/nginx/default.d/*.conf;
}
location /presto-ui {
proxy_pass http://10.214.2.82:8889;
}
yanagishima.properties config is
jetty.port=8889
presto.coordinator.server=http://master:8888
presto.redirect.server=http://10.214.0.96/presto-ui
catalog=hive
schema=default
select.limit=500
but,I access get error,these is not css, image,etc,only have text in index.html,like can't found web base directory
how config use reverse proxy?
When using Hive, YarnJobList throws a NullPointerException due to Hive not returning any apps - this is due to Hive not using MapReduce for its jobs. Checking for null would suffice.
yanagishima version: 6.0
presto version:0.170
conf/yanagishima.properties:
jetty.port=8280
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=ha-presto
presto.coordinator.server.ha-presto=http://localhost:8889
presto.redirect.server.ha-presto=http://localhost:8889
catalog.ha-presto=hive
schema.ha-presto=default
select.limit=500
audit.http.header.name=some.auth.header
to.values.query.limit=500
check.datasource=false
JVM version (java -version): openjdk version "1.8.0_141"
Server OS version :Linux 4.4.35-33.55.amzn1.x86_64
Client OS version:windows 7
Browser: chrome
function, table completion(ESC) does not work
Currently yanagishima do not support multiple statements in a single query, for example:
create temporary table tmp1 as select .... ;
create temporary table tmp2 as select .... ;
select count(*) from tmp1 join tmp2 on (...) where ... ;
Highly recommend to support multiple statements in a single query.
I propose the result of multiple statements query simply like below:
table 1: no result set
table 2: no result set
table 3: a table shows result set
Hi
When exporting textual content through tsv way, the encoding is fine (utf-8)
However, when exporting a csv, the encoding is broken. Looks like the csv transformation step does not handle utf8 well
Thanks
WEB UI HTTP ERROR: 404*:
yanagishima 16.0:
presto version presto-server-0.214:
conf/yanagishima.properties:
jetty.port=8088
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=presto1
auth.your-presto=false
presto.coordinator.server.presto1=http://localhost:8081
presto.redirect.server.presto1=http://localhost:8081
catalog.presto1=mongodb
schema.presto1=test
select.limit=500
to.values.query.limit=500
check.datasource=false
sql.query.engines=presto
cors.enabled=false
JVM version (java -version):
[root@localhost target]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Server OS version (uname -a if on a Unix-like system):
[root@localhost target]# uname -a
Linux xxxxxxx.xxx 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Browser:
Description of the problem including expected versus actual behavior:
Steps to reproduce:
start yanagishima but i see 404 not found
Describe the feature:
a variable reference multiple times will cause error.
ie: SELECT ${a} FROM users GROUP by ${a} LIMIT 100
it seems the second variable will not assign value.
yanagishima version:
3.0
presto version:
0.175
conf/yanagishima.properties:
JVM version (java -version):
1.8.0_131
Server OS version (uname -a if on a Unix-like system):
macOS 10.12.2
Provide logs (if relevant):
Query failed (#20170523_101239_01035_464fz): line 1:57: mismatched input '$' expecting {'(', '?', 'ADD', 'ALL', 'SOME', 'ANY', 'DISTINCT', 'GROUPING', 'CUBE', 'ROLLUP', 'AT', 'NOT', 'NO', 'EXISTS', 'NULL', 'TRUE', 'FALSE', 'SUBSTRING', 'POSITION', 'TINYINT', 'SMALLINT', 'INTEGER', 'DATE', 'TIME', 'TIMESTAMP', 'INTERVAL', 'YEAR', 'MONTH', 'DAY', 'HOUR', 'MINUTE', 'SECOND', 'ZONE', 'CURRENT_DATE', 'CURRENT_TIME', 'CURRENT_TIMESTAMP', 'LOCALTIME', 'LOCALTIMESTAMP', 'EXTRACT', 'CASE', 'FILTER', 'OVER', 'PARTITION', 'RANGE', 'ROWS', 'PRECEDING', 'FOLLOWING', 'CURRENT', 'ROW', 'SCHEMA', 'COMMENT', 'VIEW', 'REPLACE', 'GRANT', 'REVOKE', 'PRIVILEGES', 'PUBLIC', 'OPTION', 'GRANTS', 'EXPLAIN', 'ANALYZE', 'FORMAT', 'TYPE', 'TEXT', 'GRAPHVIZ', 'LOGICAL', 'DISTRIBUTED', 'VALIDATE', 'CAST', 'TRY_CAST', 'SHOW', 'TABLES', 'SCHEMAS', 'CATALOGS', 'COLUMNS', 'COLUMN', 'USE', 'PARTITIONS', 'FUNCTIONS', 'TO', 'SYSTEM', 'BERNOULLI', 'POISSONIZED', 'TABLESAMPLE', 'ARRAY', 'MAP', 'SET', 'RESET', 'SESSION', 'DATA', 'START', 'TRANSACTION', 'COMMIT', 'ROLLBACK', 'WORK', 'ISOLATION', 'LEVEL', 'SERIALIZABLE', 'REPEATABLE', 'COMMITTED', 'UNCOMMITTED', 'READ', 'WRITE', 'ONLY', 'CALL', 'INPUT', 'OUTPUT', 'CASCADE', 'RESTRICT', 'INCLUDING', 'EXCLUDING', 'PROPERTIES', 'NORMALIZE', 'NFD', 'NFC', 'NFKD', 'NFKC', 'IF', 'NULLIF', 'COALESCE', '+', '-', STRING, UNICODE_STRING, BINARY_LITERAL, INTEGER_VALUE, DECIMAL_VALUE, IDENTIFIER, DIGIT_IDENTIFIER, QUOTED_IDENTIFIER, BACKQUOTED_IDENTIFIER, DOUBLE_PRECISION}
Provide screenshots (if relevant):
Describe the feature:
yanagishima version:14.0
presto version:0.107
conf/yanagishima.properties:
JVM version (java -version):1.8.0_101
Server OS version (uname -a if on a Unix-like system):
2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Client OS version
Browser:chrome 68.0.3440.106
Description of the problem including expected versus actual behavior:
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
1.select one 'mysql' data type catalog in "treeview" pannel.
2.select one "schema" and "table" ,
3.then wen can get all the fields of the selected presto table from the "column" html table.
the "column" result have 4 headers.(Column,Type,Extra,Comment).
BUG: the html table header and value do not match after third html column,(ref snapshot)
note: we can get the ajax result "data.headers" from chrome ,and find the headers have five info.["Column", "Type", "Null", "Partition Key", "Comment"].
and also we get the data.result have comment info,but display nothing.
Provide logs (if relevant):
Provide screenshots (if relevant):
Not a big deal, but there are 3 vertical scroll bars:
yanagishima version: 14.0
presto version: 0.193
Hive version: 2.3.3
conf/yanagishima.properties:
jetty.port=8080
# 30 minutes. If presto query exceeds this time, yanagishima cancel the query.
presto.query.max-run-time-seconds=1800
# 1GB. If presto query result file size exceeds this value, yanagishima cancel the query.
presto.max-result-file-byte-size=1073741824
# you can specify freely. But you need to specify same name to presto.coordinator.server.[...] and presto.redirect.server.[...] and catalog.[...] and schema.[...]
presto.datasources=butane
auth.butane=false
# presto coordinator url
presto.coordinator.server.butane=http://abby-butane-presto-coordinator:8080
# presto catalog name
catalog.butane=hive
# presto schema name
schema.butane=default
# if query result exceeds this limit, to show rest of result is skipped
select.limit=500
# http header name for audit log
audit.http.header.name=some.auth.header
use.audit.http.header.name=false
# limit to convert from tsv to values query
to.values.query.limit=500
sql.query.engines=presto,hive
hive.jdbc.url.butane=jdbc:hive2://abby-butane-hive-metastore:10000/default?auth=noSasl
hive.jdbc.user.butane=scott
hive.jdbc.password.butane=tiger
hive.query.max-run-time-seconds=300
hive.query.max-run-time-seconds.butane=300
hive.datasources=butane
hive.disallowed.keywords.butane=insert
JVM version (java -version):
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (IcedTea 3.8.0) (Alpine 8.171.11-r0)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)
Server OS version (uname -a if on a Unix-like system): Linux <xxx> 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 Linux
Client OS version: macOS 10.13
Browser: Chrome version 68
Description of the problem including expected versus actual behavior:
When performing any Hive query, I expect query is executed with completion like any presto query (works very well).
Actual result: it keeps poping up error message in red alert box like 500 Please inform admin. Although this very Hive query is sent to Hive server2 and completed (the Hive server2 log shows the query is executed successfully)
I found this request failed:
POST /hiveQueryStatus
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /hiveQueryStatus. Reason:
<pre> java.util.NoSuchElementException: No value present</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.7.v20160115</a><hr/>
</body>
</html>
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
http://localhost:8080/#datasource=butane&engine=hive&tab=result in browserSHOW catalogs then press button RunLoading...Provide logs (if relevant):
2018/07/31 13:25:44.592 +0000 WARN [ServletHandler] [Yanagishima] /hiveQueryStatus
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at yanagishima.servlet.HiveQueryStatusServlet.doPost(HiveQueryStatusServlet.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:38)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:748)
Provide screenshots (if relevant):
when database name(schema) contains char '-', it can not describe table info because it missed " wrapper.
Example:
yanagishima runs somthing like describe xxx.test-dev.tableName
it should be describe xxx."test-dev"."tableName"
Describe the feature:
yanagishima version:14.0
presto version:0.206
conf/yanagishima.properties:
jetty.port=8982
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=yixin-presto
auth.yixin-presto=false
presto.coordinator.server.yixin-presto=http://10.134.0.46:8980
presto.redirect.server.yixin-presto=http://10.134.0.46:8980
catalog.yixin-presto=hive
schema.yixin-presto=kg
select.limit=5000
audit.http.header.name=some.auth.header
use.audit.http.header.name=false
to.values.query.limit=500
check.datasource=false
hive.jdbc.url.yixin-hive=jdbc:hive2://c1-hd-nn1:2181,c1-hd-nn2:2181,c1-hd-nn3:2181/;auth=noSasl
hive.jdbc.user.yixin-hive=yisou
hive.jdbc.password.yixin-hive=9Kraq+h1Q
hive.query.max-run-time-seconds=3600
hive.query.max-run-time-seconds.your-hive=3600
resource.manager.url.yixin-hive=thrift://c1-hd-dn31.bdp.idc:9088
sql.query.engines=presto
hive.datasources=yixin-hive
hive.disallowed.keywords.yixin-hive=insert,drop
hive.max-result-file-byte-size=1073741824
hive.setup.query.path.yixin-hive=/usr/local/yanagishima/conf/hive_setup_query_your-hive
cors.enabled=false
JVM version (java -version):1.8.0_181
Server OS version (uname -a if on a Unix-like system):
Linux c2-hd-dn31 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Client OS version：
Darwin lionkingkdeMacBook-Pro.local 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64
Browser:Chrome
Description of the problem including expected versus actual behavior:
when I create a table like this:
CREATE TABLE temp (
a array(row(name varchar, mobile varchar))
)
and I 'select * from temp' then it will preparing all the time
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including
(e.g.) query execution, bookmark, settings, hisotry etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.
Provide logs (if relevant):
Provide screenshots (if relevant):
yanagishima v3.0
Use the F12, Browser error and I can't see the data list.
TypeError: item.queryStats is undefined
anonymous/<@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js line 7 > Function:1:17901
Qe@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:6:15665
anonymous@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js line 7 > Function:1:17124
e.prototype._render@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:7:27848
r@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:6:8937
So.prototype.get@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:7:23157
So.prototype.run@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:7:23800
$e@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:6:10334
ro</</<@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:7:19550
e@https://cdnjs.cloudflare.com/ajax/libs/vue/2.3.0/vue.min.js:7:19003some times ,time-consuming queries could not response
and
unable to support access control.I'm going to use shiro to achieve it
英语不好，哥们将就着看
yanagishima version: 17.0
presto version: 0.215
conf/yanagishima.properties:
jetty.port=8088
presto.query.max-run-time-seconds=14400
presto.max-result-file-byte-size=1073741824
presto.datasources=reporting
presto.coordinator.server.reporting=http://...:8080
presto.redirect.server.reporting=http:/...:8080
catalog.reporting=hive
schema.reporting=default
select.limit=500
audit.http.header.name=some.auth.header
to.values.query.limit=500
check.datasource=false
sql.query.engines=presto
JVM version (java -version): java version "1.7.0_45"
Server OS version (uname -a if on a Unix-like system):
Linux 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
Client OS version
Darwin MacBook-Pro.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64
Browser: Chrome, Vivaldi, Safari
Description of the problem including expected versus actual behavior: When running a bad query, the Result tab gets stuck with the header Loading and a Preparing loader. This also blocks the input box and Run + Explain buttons. Was this introduced when switching frontend to Vue.js (we recently upgraded from a version prior to Vue.js use)?
Steps to reproduce:
Run a query that instantly fails, such as a "Table x does not exist".
Provide logs (if relevant):
2019/01/14 20:01:37.986 +0000 WARN [PrestoServiceImpl] [Yanagishima] Query 20190114_200137_02486_si2cr has no columns
Provide screenshots (if relevant):
One day our yanagishima process crashed because of insufficient memory.
Here is the last part of y.log. (standard output of yanagishima process)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# An error report file with more information is saved as:
# /home/user1/yanagishima-0.0.9/hs_err_pid98805.log
I checked sar log under /var/log/sa and "kbcommit" went down about 32GB at the time yanagishima process died. This means yanagishima used this size of virtual memory.
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
...
12:21:01 PM 22745608 10116924 30.79 316052 2414628 35039220 94.56
12:22:01 PM 22746684 10115848 30.78 316052 2414672 35103912 94.73
12:23:01 PM 26228032 6634500 20.19 316056 2422864 652408 1.76
12:24:01 PM 26229060 6633472 20.19 316056 2422892 654596 1.77
Actually, 10 hours after I restarted yanagishima process, it already takes about 17GB VSZ. (-Xmx is just 3GB by the way)
$ ps -eo pid,vsz,rss,cmd | grep yanagishima
139080 17523888 908436 java -Xmx3G -cp :bin/../lib/annotations-2.0.3.jar:bin/../lib/antlr4-runtime-4.5.1.jar:bin/../ ....
If I check hs_err_pid98805.log, there are lots of "http-client-anonymous" threads.
Java Threads: ( => current thread )
0x00007f5f92203000 JavaThread "http-client-anonymous3611-scheduler-41356" daemon [_thread_blocked, id=138452, stack(0x00007f5f40304000,0x00007f5f40405000)]
0x00007f5f921aa000 JavaThread "http-client-anonymous3611-41355" daemon [_thread_blocked, id=138451, stack(0x00007f5f40405000,0x00007f5f40506000)]
0x00007f5f921a8000 JavaThread "http-client-anonymous3611-41354" daemon [_thread_blocked, id=138450, stack(0x00007f5f40506000,0x00007f5f40607000)]
0x00007f5f921a6000 JavaThread "http-client-anonymous3611-41353" daemon [_thread_blocked, id=138449, stack(0x00007f5f40607000,0x00007f5f40708000)]
There were 32530 threads and 32499 threads were "http-client-anonymous".
$ grep JavaThread ~/yanagishima-0.0.9/hs_err_pid98805.log | wc -l
32530
$ grep JavaThread ~/yanagishima-0.0.9/hs_err_pid98805.log | grep http-client-anonymous | wc -l
32499
It seems that yanagishima creates huge number of threads, so the total amount of thread stack becomes GBs of memory.
I have not found the root cause yet, but these "http-client-anonymous" seems to be created in JettyHttpClient in airlift http-client. So are there any problem with the usage of that library?
When I read /src/main/java/yanagishima/util/DownloadUtil.java，I found csvDownload method in the character encoding for Shift_JIS, Chinese will be garbled, can be changed to UTF-8?
However, when I try to modify the encoding format for UTF-8, and recompiling, run the error as follows:
2017/05/04 08:34:34.965 -0700 ERROR [PrestoServlet] [Yanagishima] Missing required header 'X-yanagishima-datasources'.
java.lang.RuntimeException: Missing required header 'X-yanagishima-datasources'.
at yanagishima.util.HttpRequestUtil.getHeader(HttpRequestUtil.java:20)
at yanagishima.util.AccessControlUtil.checkDatasource(AccessControlUtil.java:14)
at yanagishima.servlet.PrestoServlet.lambda$doPost$4(PrestoServlet.java:80)
at java.util.Optional.ifPresent(Optional.java:159)
at yanagishima.servlet.PrestoServlet.doPost(PrestoServlet.java:76)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:22)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)I input http://192.168.11.29:8080,then it redirection to:http://192.168.11.29:8080/#datasource=your-presto&engine=presto&tab=treeview .
Below is my configuation:
jetty.port=8080
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=your-presto
auth.your-presto=false
presto.coordinator.server.your-presto=http://192.168.11.29:6080
presto.redirect.server.your-presto=http://192.168.11.29:6080
catalog.your-presto=hive
schema.your-presto=default
select.limit=500
audit.http.header.name=some.auth.header
use.audit.http.header.name=false
to.values.query.limit=500
check.datasource=false
hive.jdbc.url.your-hive=jdbc:hive2://192.168.11.29:10000/default;auth=noSasl
hive.jdbc.user.your-hive=yanagishima
hive.jdbc.password.your-hive=yanagishima
hive.query.max-run-time-seconds=3600
hive.query.max-run-time-seconds.your-hive=3600
resource.manager.url.your-hive=http://localhost:8088
sql.query.engines=presto,hive
hive.datasources=your-hive
hive.disallowed.keywords.your-hive=insert,drop
hive.max-result-file-byte-size=1073741824
hive.setup.query.path.your-hive=/usr/local/yanagishima/conf/hive_setup_query_your-hive
cors.enabled=false
I read the code and find es sql use unofficial-elasticsearch-jdbc-driver, but it based on elasticsearch 6.3+'s rest api.
Whether to consider using a more general es jdbc? Like this: https://github.com/NLPchina/elasticsearch-sql
Would be great if you could generate a docker image on Docker Hub
Thanks for this very usefull tool by the way
Hi wyukawa
I use the conf as following, but when I click demo button, the exception as follows.
jetty.port=8001
presto.query.max-run-time-seconds=1800
presto.max-result-file-byte-size=1073741824
presto.datasources=demo
presto.coordinator.server.your-presto=http://localhost:8080
presto.redirect.server.your-presto=http://localhost:8080
catalog.your-presto=hive
schema.your-presto=default
select.limit=500
audit.http.header.name=some.auth.header
2017/04/21 10:51:38.648 +0800 ERROR [PrestoServlet] [Yanagishima] No value present
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at yanagishima.config.YanagishimaConfig.getPrestoCoordinatorServer(YanagishimaConfig.java:17)
at yanagishima.service.PrestoServiceImpl.getStatementClient(PrestoServiceImpl.java:212)
at yanagishima.service.PrestoServiceImpl.doQuery(PrestoServiceImpl.java:57)
at yanagishima.servlet.PrestoServlet.lambda$doPost$4(PrestoServlet.java:79)
at java.util.Optional.ifPresent(Optional.java:159)
at yanagishima.servlet.PrestoServlet.doPost(PrestoServlet.java:72)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at yanagishima.filter.YanagishimaFilter.doFilter(YanagishimaFilter.java:17)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
at org.eclipse.jetty.server.Server.handle(Server.java:517)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
Could you please show me your conf?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.