dacort / metabase-athena-driver Goto Github PK

View Code? Open in Web Editor NEW

225.0 22.0 32.0 146 KB

An Amazon Athena driver for Metabase 0.32 and later

License: Apache License 2.0

Clojure 92.54% Dockerfile 5.47% Makefile 1.99%

metabase athena athena-driver aws aws-athena amazon-athena metabase-driver

metabase-athena-driver's People

Contributors

Stargazers

Watchers

metabase-athena-driver's Issues

Bug in chart URL sharing for Native queries with a boolean field filter pre-selected

Hello! :)
I recently noticed a bug related to chart URL sharing for Native Athena queries with a boolean field filter preselected.
I have a simple query

SELECT id, boolean_field
FROM dataset.test_table
WHERE true
    [[AND {{boolean_field_filter}}]]
ORDER BY id ASC
LIMIT 5

With boolean_field_filter being connected to dataset.test_table.boolean_field (params: Field filter/Category).

Issue:

When sharing the chart URL with boolean filter selected to true/false on gets the following error:
[Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. SYNTAX_ERROR: line 6:44: IN value and list items must be the same type: boolean [Execution ID: 09f16e70-e65e-441b-96f8-b6a6e6fc8eb1]

Example:

Query with no filters - works fine:

Query with manually selected filters - works fine:

After hitting refresh in the browser/sharing a link - fails:

Cheers!

No suitable driver found for jdbc:awsathena://athena.us-west-2.amazonaws.com

I followed the guide IAM policy, but I'm getting the following error when trying to set up the database. A POST to localhost:3000/api/database returns:

{"valid":false,"dbname":"No suitable driver found for jdbc:awsathena://athena.us-west-2.amazonaws.com:443","message":"No suitable driver found for jdbc:awsathena://athena.us-west-2.amazonaws.com:443"}

Any suggestions?

Getting Type expect error

"[Simba]AthenaJDBC An error has been thrown from the AWS Athena client. HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Error: type expected at the position 0 of 'bool' but 'bool' is found. (Service: null; Status Code: 0; Error Code: null"

Driver cannot display date nor timestamp

When I try to display date related fields (date or timestamp) the driver cannot display those fields.

Anyone experiencing this same problem?

Building from source - Multiple Systems Error java_time__init.class

I've tried to build from source following the instructions on separate systems (Amazon Linux, Ubuntu 14,18,20 and MAC OSX 10.14.6) with multiple java versions from OpenJDK/Oracle/Adopt, yet all give me the same error.:

Caused by: java.io.FileNotFoundException: Could not locate java_time__init.class, java_time.clj or java_time.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.

Here are the versions I'm using on OSX
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
Leiningen 2.9.3 on Java 1.8.0_251 Java HotSpot(TM) 64-Bit Server VM
yarn 1.22.4

The JDBC driver is at the correct location:
~/.m2/repository/athena/athena-jdbc/2.0.9/
athena-jdbc-2.0.9.jar

I'm trying to get a build with the preview https://athena-downloads.s3.amazonaws.com/drivers/JDBC/athena-preview/SimbaAthenaJDBC_2.0.11_preview/AthenaJDBC42_preview.jar JDBC driver once I can get the current release to compile.

Any help would be much appreciated. Thanks

Error running sync step

Trying this for the first time and seems to fail at syncing. Clean install of Metabase v0.36.6

Also tried different log level settings for the DB but logging doesn't seem to change...

[6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:08-07:00 INFO metabase.sync.util STARTING: Sync metadata for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:08-07:00 INFO metabase.sync.util STARTING: step 'sync-timezone' for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:08-07:00 INFO metabase.sync.util FINISHED: step 'sync-timezone' for athena Database 2 'Sandbox' (854.2 µs) [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:08-07:00 INFO metabase.sync.util STARTING: step 'sync-tables' for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 ERROR metabase.sync.util Error running sync step [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util FINISHED: Sync metadata for athena Database 2 'Sandbox' (1.0 s) [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util STARTING: Analyze data for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util STARTING: step 'fingerprint-fields' for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util FINISHED: step 'classify-fields' for athena Database 2 'Sandbox' (196.0 µs) [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util FINISHED: step 'fingerprint-fields' for athena Database 2 'Sandbox' (305.6 µs) [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util STARTING: step 'classify-fields' for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util FINISHED: step 'classify-tables' for athena Database 2 'Sandbox' (272.5 µs) [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util STARTING: step 'classify-tables' for athena Database 2 'Sandbox' [6477ad8b-15ca-4eda-8113-5044c62ed21c] 2020-09-17T17:21:09-07:00 INFO metabase.sync.util FINISHED: Analyze data for athena Database 2 'Sandbox' (10.0 ms)

Create Dockerfiles for source install (beyond just binary install)

Hi! Thanks for this wonderful driver. I was pointed here as the best way to learn driver development, and this looks really clear.

Just wanted to make the small suggestion, so people with Docker already set up can build HEAD (and see the high-level build steps laid out from scratch)

Dockerfile.binary and Dockerfile.source would be clear. Parking this for now, and maybe I'll come back to work on it myself when I'm getting my driver-dev legs underneath me :)

Connection to multiple catalogs not supported

I have made a connection using the driver in metabase but it showing me all the database and tables in default AwsDataCatalog.

I have created other catalogs also in athena using hive metastore but that is not visible in metabase. How can i specify catalog name that i want to connect.

Connection timeout though data is being created

Hi there,

I installed the driver from the latest release and my metabase interface gives me the proper form to fill in my credentials for athena. I created an empty bucket and on "Save" I end up getting metadata as .txt and .csv inside my bucket, but metabase tells me the connection timed out.
I did set the ENV to have 10 second timeout. This did not help. I assume my credentials are correct because metabase is writing into my bucket. Here are the logs from my system:

[ea90e48c-6a84-4ac9-865c-0a0938c5c854] 2019-12-10T15:33:23+01:00 ERROR metabase.driver.util Database connection error [ea90e48c-6a84-4ac9-865c-0a0938c5c854] 2019-12-10T15:33:23+01:00 DEBUG metabase.middleware.log POST /api/database 400 10.0 s (0 DB calls) {:valid false, :dbname "Timed out after 10,000 milliseconds.", :message "Timed out after 10,000 milliseconds."}

driver is error for cn regions

Dear Developer:
I used your latest Athena driver ,v1.10, but driver also throw exception for China region；

An error has been thrown from the AWS SDK client. Unable to execute HTTP request: athena.cn-north-1.amazonaws.com: Name or service not known [Execution ID not available]

Please forgive my bad english

Upgrade Athena JDBC Driver

Current version is: 2.0.13

Interesting new features since 2.0.9:

Retrieve virtual views as views (#11)
Updated catalog support to return metadata from any catalog (#49)

Please add support for China Region

Dear Developer:

I find your Athena driver doesn't support China Region due to the endpoint difference.
For Athena endpoint list : https://docs.aws.amazon.com/general/latest/gr/athena.html
The China regions all ends with amazonaws.com.cn instead amazonaws.com.

And the config document , https://s3.amazonaws.com/athena-downloads/drivers/JDBC/SimbaAthenaJDBC_2.0.9/docs/Simba+Athena+JDBC+Driver+Install+and+Configuration+Guide.pdf , the EndpointOverride config seems not change the suffix of the endpoint.

Thanks.

Improve query cancellation handling

Queries cancel fine, but there's still a WARN message in the Metabase logs.

05-09 08:42:44 DEBUG async.util :: Request canceled, canceling future.
05-09 08:42:44 WARN middleware.async :: Unhandled exception, exepected `catch-exceptions` middleware to handle it.
java.lang.Exception: Unexpected nil response from query processor.
	at metabase.query_processor.middleware.process_userland_query$format_userland_query_result.invokeStatic(process_userland_query.clj:94)
	at metabase.query_processor.middleware.process_userland_query$format_userland_query_result.invoke(process_userland_query.clj:87)
	at clojure.core$partial$fn__5828.invoke(core.clj:2638)
	at metabase.query_processor.middleware.async_wait$wait_for_permit$fn__33563$fn__33602$state_machine__8574__auto____33623$fn__33625.invoke(async_wait.clj:49)
	at metabase.query_processor.middleware.async_wait$wait_for_permit$fn__33563$fn__33602$state_machine__8574__auto____33623.invoke(async_wait.clj:49)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__8592.invoke(ioc_macros.clj:986)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__3543.invoke(channels.clj:265)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
05-09 08:42:44 WARN sql-jdbc.execute :: Client closed connection, canceling query

Failing to parse nested array expression

Query:

SELECT ARRAY[ARRAY[1]]

Expected:

{[1]}

Actual:

Output of parse-tokens* does not match schema: �[0;33m [(named [nil nil (not (matches-some-precondition? nil))] "parsed tokens") (not (present? "remaining tokens"))] �[0m

Add assume role option if running metabase in an AWS environment

I am currently running Metabase from a docker image on ecs. It would be awesome if the athena driver would allow an option to assume the role of the ecs task instead of passing in aws access keys. Or make them optional and, by default, assume the ecs task role.

Feature Request: Cache feature of metabase for athena driver

Currently the athena plugin does not support the caching feature of metabase.

We quite a few metric dashboards running.
Problems with no cache:

AWS service limits
Some of the queries take quite a few minutes to run, which delays the dashboard load

We would ideally want these to be cached.

Specify database during setup

The driver currently syncs all databases and all tables - this may not be desirable behavior.

Metabase 0.38 removed sql.qp/unix-timestamp->timestamp

It seems there are compatibilty issues of Athena driver (version 1.1.0) with the Metabse 0.38, our service driver is throwing error after we upgraded metabase to 0.38 -> “Attempting to call unbound fn: #'metabase.driver.sql.query-processor/cast-unix-timestamp-field-if-needed”.
Also, it is mentioned in metabase discourse that they have removed the support for unix-timestamp https://discourse.metabase.com/t/changes-in-0-38-for-driver-developers/13473

Figure out a good way to add tests

We need a way to add a test suite.

Unfortunately this requires uploading sample data to S3 and executing Athena queries so it could be a little complex.

Allow access to fields of unstructured data in filters and custom expression

I have a column that contains unstructured JSON data. Would it be possible to access these nested fields in the filters and custom expressions somehow?

feature request: option to use DefaultAWSCredentialsProviderChain

When deployed to AWS EC2, it's best to use instance profile credentials delivered through the Amazon EC2 metadata service, which are considered more secure than using long-lived tokens. The Athena JDBC driver docs say that DefaultAWSCredentialsProviderChain is supported. That would enable the standard list of credential provider options including instance profile credentials, environment variables, and JVM system properties.

I'd be glad to implement this and submit a PR if it makes sense.

Version 0.35 upgrade

Metabase 0.35 has some breaking change.

Need to update the driver to reflect that.

sync step failed after upgrading to metabase athena driver(v1.1.0)

Hi,

After upgrading to new version(v1.1.0) we are not able to sync the athena to metabase.

Error log.

08-03 15:09:16 INFO sync.util :: STARTING: Sync metadata for athena Database 2 'Athena'
08-03 15:09:16 DEBUG middleware.log :: POST /api/database/2/sync_schema 200 1.5 ms (1 DB calls) App DB connections: 0/15 Jetty threads: 3/50 (6 idle, 0 queued) (112 total active threads) Queries in flight: 0 (0 queued)
08-03 15:09:16 INFO sync.util :: STARTING: step 'sync-timezone' for athena Database 2 'Athena'
08-03 15:09:16 INFO sync.util :: FINISHED: step 'sync-timezone' for athena Database 2 'Athena' (94.0 µs)
08-03 15:09:16 INFO sync.util :: STARTING: step 'sync-tables' for athena Database 2 'Athena'
08-03 15:09:24 ERROR sync.util :: Error running sync step
java.sql.SQLException: [Simba]AthenaJDBC An error has been thrown from the AWS Athena client. Failed to invoke lambda function due to Unsupported Arrow Type [Timestamp(MILLISECOND, null)] in Lambda Data Source [Execution ID not available]
at com.simba.athena.athena.api.AJClient.checkAndThrowException(Unknown Source)
at com.simba.athena.athena.api.AJClient.fetchTablesWithProxyApi(Unknown Source)
at com.simba.athena.athena.api.AJClient.getTablesMetadata(Unknown Source)
at com.simba.athena.athena.dataengine.metadata.AJTablesMetadataSource.initMetadata(Unknown Source)
at com.simba.athena.athena.dataengine.metadata.AJTablesMetadataSource.(Unknown Source)
at com.simba.athena.athena.dataengine.AJDataEngine.makeNewMetadataSource(Unknown Source)
at com.simba.athena.dsi.dataengine.impl.DSIDataEngine.makeNewMetadataResult(Unknown Source)
at com.simba.athena.athena.dataengine.AJDataEngine.makeNewMetadataResult(Unknown Source)
at com.simba.athena.jdbc.jdbc42.S42DatabaseMetaData.createMetaDataResult(Unknown Source)
at com.simba.athena.jdbc.common.SDatabaseMetaData.getTables(Unknown Source)
at com.mchange.v2.c3p0.impl.NewProxyDatabaseMetaData.getTables(NewProxyDatabaseMetaData.java:3010)
at metabase.driver.athena$get_tables.invokeStatic(athena.clj:242)
at metabase.driver.athena$get_tables.invoke(athena.clj:240)
at metabase.driver.athena$fast_active_tables$iter__553__559$fn__560.invoke(athena.clj:253)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:58)
at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
at clojure.core$chunk_next.invokeStatic(core.clj:708)
at clojure.core$reduce1.invokeStatic(core.clj:942)
at clojure.core$set.invokeStatic(core.clj:4113)
at clojure.core$set.invoke(core.clj:4105)
at metabase.driver.athena$fast_active_tables.invokeStatic(athena.clj:252)
at metabase.driver.athena$fast_active_tables.doInvoke(athena.clj:247)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at metabase.driver.athena$fn__578$fn__579.invoke(athena.clj:267)
at metabase.driver.athena$fn__578.invokeStatic(athena.clj:266)
at metabase.driver.athena$fn__578.invoke(athena.clj:265)
at clojure.lang.MultiFn.invoke(MultiFn.java:234)
at metabase.sync.fetch_metadata$fn__52935$db_metadata__52940$fn__52941.invoke(fetch_metadata.clj:13)
at metabase.sync.fetch_metadata$fn__52935$db_metadata__52940.invoke(fetch_metadata.clj:10)
at metabase.sync.sync_metadata.tables$fn__54361$db_metadata__54366$fn__54367.invoke(tables.clj:146)
at metabase.sync.sync_metadata.tables$fn__54361$db_metadata__54366.invoke(tables.clj:143)
at metabase.sync.sync_metadata.tables$fn__54422$sync_tables_BANG___54427$fn__54428.invoke(tables.clj:163)
at metabase.sync.sync_metadata.tables$fn__54422$sync_tables_BANG___54427.invoke(tables.clj:158)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223$fn__42227$fn__42229.invoke(util.clj:345)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_debug_logging.invokeStatic(util.clj:119)
at metabase.sync.util$with_start_and_finish_debug_logging.invoke(util.clj:116)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223$fn__42227.invoke(util.clj:342)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223.invoke(util.clj:337)
at metabase.sync.util$fn__42410$run_sync_operation__42415$fn__42416$fn__42417.invoke(util.clj:430)
at clojure.core$mapv$fn__8445.invoke(core.clj:6912)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
at clojure.core$reduce.invokeStatic(core.clj:6827)
at clojure.core$mapv.invokeStatic(core.clj:6903)
at clojure.core$mapv.invoke(core.clj:6903)
at metabase.sync.util$fn__42410$run_sync_operation__42415$fn__42416.invoke(util.clj:430)
at metabase.sync.util$fn__42410$run_sync_operation__42415.invoke(util.clj:424)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480$fn__54481$fn__54482.invoke(sync_metadata.clj:51)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:149)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:142)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:145)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:142)
at clojure.core$partial$fn__5839.invoke(core.clj:2623)
at metabase.driver$fn__22445.invokeStatic(driver.clj:540)
at metabase.driver$fn__22445.invoke(driver.clj:540)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at metabase.sync.util$sync_in_context$fn__42130.invoke(util.clj:138)
at metabase.sync.util$with_db_logging_disabled$fn__42127.invoke(util.clj:129)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_logging$fn__42116.invoke(util.clj:114)
at metabase.sync.util$with_sync_events$fn__42111.invoke(util.clj:88)
at metabase.sync.util$with_duplicate_ops_prevented$fn__42102.invoke(util.clj:67)
at metabase.sync.util$do_sync_operation.invokeStatic(util.clj:164)
at metabase.sync.util$do_sync_operation.invoke(util.clj:161)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480$fn__54481.invoke(sync_metadata.clj:50)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480.invoke(sync_metadata.clj:47)
at metabase.api.database$fn__55211$fn__55213.invoke(database.clj:614)
at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused by: com.simba.athena.support.exceptions.GeneralException: [Simba]AthenaJDBC An error has been thrown from the AWS Athena client. Failed to invoke lambda function due to Unsupported Arrow Type [Timestamp(MILLISECOND, null)] in Lambda Data Source [Execution ID not available]
... 75 more
Caused by: com.simba.athena.amazonaws.services.athena.model.MetadataException: Failed to invoke lambda function due to Unsupported Arrow Type [Timestamp(MILLISECOND, null)] in Lambda Data Source (Service: AmazonAthena; Status Code: 400; Error Code: MetadataException; Request ID: 60dc434f-36d9-41a9-af7a-8e20aaee05f9; Proxy: null)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.simba.athena.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.simba.athena.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.simba.athena.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.simba.athena.amazonaws.services.athena.AmazonAthenaClient.doInvoke(AmazonAthenaClient.java:1948)
at com.simba.athena.amazonaws.services.athena.AmazonAthenaClient.invoke(AmazonAthenaClient.java:1915)
at com.simba.athena.amazonaws.services.athena.AmazonAthenaClient.invoke(AmazonAthenaClient.java:1904)
at com.simba.athena.amazonaws.services.athena.AmazonAthenaClient.executeListTableMetadata(AmazonAthenaClient.java:1380)
at com.simba.athena.amazonaws.services.athena.AmazonAthenaClient.listTableMetadata(AmazonAthenaClient.java:1351)
at com.simba.athena.athena.api.AJClient.fetchTablesWithProxyApi(Unknown Source)
at com.simba.athena.athena.api.AJClient.getTablesMetadata(Unknown Source)
at com.simba.athena.athena.dataengine.metadata.AJTablesMetadataSource.initMetadata(Unknown Source)
at com.simba.athena.athena.dataengine.metadata.AJTablesMetadataSource.(Unknown Source)
at com.simba.athena.athena.dataengine.AJDataEngine.makeNewMetadataSource(Unknown Source)
at com.simba.athena.dsi.dataengine.impl.DSIDataEngine.makeNewMetadataResult(Unknown Source)
at com.simba.athena.athena.dataengine.AJDataEngine.makeNewMetadataResult(Unknown Source)
at com.simba.athena.jdbc.jdbc42.S42DatabaseMetaData.createMetaDataResult(Unknown Source)
at com.simba.athena.jdbc.common.SDatabaseMetaData.getTables(Unknown Source)
at com.mchange.v2.c3p0.impl.NewProxyDatabaseMetaData.getTables(NewProxyDatabaseMetaData.java:3010)
at metabase.driver.athena$get_tables.invokeStatic(athena.clj:242)
at metabase.driver.athena$get_tables.invoke(athena.clj:240)
at metabase.driver.athena$fast_active_tables$iter__553__559$fn__560.invoke(athena.clj:253)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:58)
at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
at clojure.core$chunk_next.invokeStatic(core.clj:708)
at clojure.core$reduce1.invokeStatic(core.clj:942)
at clojure.core$set.invokeStatic(core.clj:4113)
at clojure.core$set.invoke(core.clj:4105)
at metabase.driver.athena$fast_active_tables.invokeStatic(athena.clj:252)
at metabase.driver.athena$fast_active_tables.doInvoke(athena.clj:247)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at metabase.driver.athena$fn__578$fn__579.invoke(athena.clj:267)
at metabase.driver.athena$fn__578.invokeStatic(athena.clj:266)
at metabase.driver.athena$fn__578.invoke(athena.clj:265)
at clojure.lang.MultiFn.invoke(MultiFn.java:234)
at metabase.sync.fetch_metadata$fn__52935$db_metadata__52940$fn__52941.invoke(fetch_metadata.clj:13)
at metabase.sync.fetch_metadata$fn__52935$db_metadata__52940.invoke(fetch_metadata.clj:10)
at metabase.sync.sync_metadata.tables$fn__54361$db_metadata__54366$fn__54367.invoke(tables.clj:146)
at metabase.sync.sync_metadata.tables$fn__54361$db_metadata__54366.invoke(tables.clj:143)
at metabase.sync.sync_metadata.tables$fn__54422$sync_tables_BANG___54427$fn__54428.invoke(tables.clj:163)
at metabase.sync.sync_metadata.tables$fn__54422$sync_tables_BANG___54427.invoke(tables.clj:158)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223$fn__42227$fn__42229.invoke(util.clj:345)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_debug_logging.invokeStatic(util.clj:119)
at metabase.sync.util$with_start_and_finish_debug_logging.invoke(util.clj:116)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223$fn__42227.invoke(util.clj:342)
at metabase.sync.util$fn__42218$run_step_with_metadata__42223.invoke(util.clj:337)
at metabase.sync.util$fn__42410$run_sync_operation__42415$fn__42416$fn__42417.invoke(util.clj:430)
at clojure.core$mapv$fn__8445.invoke(core.clj:6912)
at clojure.lang.PersistentVector.reduce(PersistentVector.java:343)
at clojure.core$reduce.invokeStatic(core.clj:6827)
at clojure.core$mapv.invokeStatic(core.clj:6903)
at clojure.core$mapv.invoke(core.clj:6903)
at metabase.sync.util$fn__42410$run_sync_operation__42415$fn__42416.invoke(util.clj:430)
at metabase.sync.util$fn__42410$run_sync_operation__42415.invoke(util.clj:424)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480$fn__54481$fn__54482.invoke(sync_metadata.clj:51)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:149)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:142)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:145)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:142)
at clojure.core$partial$fn__5839.invoke(core.clj:2623)
at metabase.driver$fn__22445.invokeStatic(driver.clj:540)
at metabase.driver$fn__22445.invoke(driver.clj:540)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at metabase.sync.util$sync_in_context$fn__42130.invoke(util.clj:138)
at metabase.sync.util$with_db_logging_disabled$fn__42127.invoke(util.clj:129)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_logging$fn__42116.invoke(util.clj:114)
at metabase.sync.util$with_sync_events$fn__42111.invoke(util.clj:88)
at metabase.sync.util$with_duplicate_ops_prevented$fn__42102.invoke(util.clj:67)
at metabase.sync.util$do_sync_operation.invokeStatic(util.clj:164)
at metabase.sync.util$do_sync_operation.invoke(util.clj:161)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480$fn__54481.invoke(sync_metadata.clj:50)
at metabase.sync.sync_metadata$fn__54475$sync_db_metadata_BANG___54480.invoke(sync_metadata.clj:47)
at metabase.api.database$fn__55211$fn__55213.invoke(database.clj:614)
at clojure.core$binding_conveyor_fn$fn__5754.invoke(core.clj:2030)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
08-03 15:09:24 INFO sync.util :: FINISHED: Sync metadata for athena Database 2 'Athena' (8.5 s)

Timestamp errors

I'm not sure if this is related to issue 38 so I thought I'd post a new one.

Full disclosure: I'm more of an Ops guy, I've started up a container of Metabase and included the Athena driver. I have a Glue Crawler that looks at an S3 RDS MySQL export (Parquet format) and then populates the Athena DB schema. This appears to be ok, however, when I try to explore tables in Metabase, a lot of them throw up an error:

[JDBC Driver]Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]

Stacktrace shows:

25/04/2020 09:29:1304-25 08:29:13 ERROR middleware.catch-exceptions :: Error processing query: null
25/04/2020 09:29:13{:database_id 2,
25/04/2020 09:29:13 :started_at (t/zoned-date-time "2020-04-25T08:29:06.589055Z[GMT]"),
25/04/2020 09:29:13 :via
25/04/2020 09:29:13 [{:status :failed,
25/04/2020 09:29:13   :class java.sql.SQLException,
25/04/2020 09:29:13   :error "[JDBC Driver]Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]",
25/04/2020 09:29:13   :stacktrace
25/04/2020 09:29:13   ["java.sql/java.sql.Timestamp.valueOf(Unknown Source)"
25/04/2020 09:29:13    "com.simba.athena.athena.dataengine.AJBaseResultSet.getData(Unknown Source)"
25/04/2020 09:29:13    "com.simba.athena.jdbc.common.SForwardResultSet.getData(Unknown Source)"
25/04/2020 09:29:13    "com.simba.athena.jdbc.common.SForwardResultSet.getString(Unknown Source)"
25/04/2020 09:29:13    "com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:4189)"
25/04/2020 09:29:13    "--> driver.sql_jdbc.execute.legacy_impl$fn__72760$fn__72761.invoke(legacy_impl.clj:77)"
25/04/2020 09:29:13    "driver.sql_jdbc.execute$row_thunk$row_thunk_STAR___70724.invoke(execute.clj:348)"
25/04/2020 09:29:13    "query_processor.reducible$reducible_rows$reify__37968.reduce(reducible.clj:156)"
25/04/2020 09:29:13    "query_processor.context.default$default_reducef$fn__37866.invoke(default.clj:59)"
25/04/2020 09:29:13    "query_processor.context.default$default_reducef.invokeStatic(default.clj:58)"
25/04/2020 09:29:13    "query_processor.context.default$default_reducef.invoke(default.clj:49)"
25/04/2020 09:29:13    "query_processor.context$reducef.invokeStatic(context.clj:69)"
25/04/2020 09:29:13    "query_processor.context$reducef.invoke(context.clj:62)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf$respond_STAR___37870.invoke(default.clj:70)"
25/04/2020 09:29:13    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:392)"
25/04/2020 09:29:13    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:377)"
25/04/2020 09:29:13    "driver.sql_jdbc$fn__72711.invokeStatic(sql_jdbc.clj:50)"
25/04/2020 09:29:13    "driver.sql_jdbc$fn__72711.invoke(sql_jdbc.clj:48)"
25/04/2020 09:29:13    "driver.athena$fn__609.invokeStatic(athena.clj:273)"
25/04/2020 09:29:13    "driver.athena$fn__609.invoke(athena.clj:271)"
25/04/2020 09:29:13    "query_processor.context$executef.invokeStatic(context.clj:59)"
25/04/2020 09:29:13    "query_processor.context$executef.invoke(context.clj:48)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf.invokeStatic(default.clj:69)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf.invoke(default.clj:67)"
25/04/2020 09:29:13    "query_processor.context$runf.invokeStatic(context.clj:45)"
25/04/2020 09:29:13    "query_processor.context$runf.invoke(context.clj:39)"
25/04/2020 09:29:13    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
25/04/2020 09:29:13    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
25/04/2020 09:29:13    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__43003.invoke(mbql_to_native.clj:26)"
25/04/2020 09:29:13    "query_processor.middleware.check_features$check_features$fn__42317.invoke(check_features.clj:42)"
25/04/2020 09:29:13    "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__43168.invoke(optimize_datetime_filters.clj:133)"
25/04/2020 09:29:13    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__47065.invoke(wrap_value_literals.clj:137)"
25/04/2020 09:29:13    "query_processor.middleware.annotate$add_column_info$fn__40946.invoke(annotate.clj:577)"
25/04/2020 09:29:13    "query_processor.middleware.permissions$check_query_permissions$fn__42192.invoke(permissions.clj:64)"
25/04/2020 09:29:13    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__43667.invoke(pre_alias_aggregations.clj:40)"
25/04/2020 09:29:13    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__42390.invoke(cumulative_aggregations.clj:61)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_joins$resolve_joins$fn__44199.invoke(resolve_joins.clj:183)"
25/04/2020 09:29:13    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__39133.invoke(add_implicit_joins.clj:245)"
25/04/2020 09:29:13    "query_processor.middleware.limit$limit$fn__42989.invoke(limit.clj:38)"
25/04/2020 09:29:13    "query_processor.middleware.format_rows$format_rows$fn__42970.invoke(format_rows.clj:81)"
25/04/2020 09:29:13    "query_processor.middleware.desugar$desugar$fn__42456.invoke(desugar.clj:22)"
25/04/2020 09:29:13    "query_processor.middleware.binning$update_binning_strategy$fn__41490.invoke(binning.clj:229)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_fields$resolve_fields$fn__41998.invoke(resolve_fields.clj:24)"
25/04/2020 09:29:13    "query_processor.middleware.add_dimension_projections$add_remapping$fn__38669.invoke(add_dimension_projections.clj:272)"
25/04/2020 09:29:13    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__38889.invoke(add_implicit_clauses.clj:147)"
25/04/2020 09:29:13    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__39282.invoke(add_source_metadata.clj:105)"
25/04/2020 09:29:13    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__43864.invoke(reconcile_breakout_and_order_by_bucketing.clj:98)"
25/04/2020 09:29:13    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__41131.invoke(auto_bucket_datetimes.clj:125)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__42045.invoke(resolve_source_table.clj:46)"
25/04/2020 09:29:13    "query_processor.middleware.parameters$substitute_parameters$fn__43649.invoke(parameters.clj:97)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__42097.invoke(resolve_referenced.clj:80)"
25/04/2020 09:29:13    "query_processor.middleware.expand_macros$expand_macros$fn__42712.invoke(expand_macros.clj:158)"
25/04/2020 09:29:13    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__39313.invoke(add_timezone_info.clj:15)"
25/04/2020 09:29:13    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__46949.invoke(splice_params_in_response.clj:32)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875$fn__43879.invoke(resolve_database_and_driver.clj:33)"
25/04/2020 09:29:13    "driver$do_with_driver.invokeStatic(driver.clj:61)"
25/04/2020 09:29:13    "driver$do_with_driver.invoke(driver.clj:57)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875.invoke(resolve_database_and_driver.clj:27)"
25/04/2020 09:29:13    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__42918.invoke(fetch_source_query.clj:243)"
25/04/2020 09:29:13    "query_processor.middleware.store$initialize_store$fn__46958$fn__46959.invoke(store.clj:11)"
25/04/2020 09:29:13    "query_processor.store$do_with_store.invokeStatic(store.clj:46)"
25/04/2020 09:29:13    "query_processor.store$do_with_store.invoke(store.clj:40)"
25/04/2020 09:29:13    "query_processor.middleware.store$initialize_store$fn__46958.invoke(store.clj:10)"
25/04/2020 09:29:13    "query_processor.middleware.cache$maybe_return_cached_results$fn__41974.invoke(cache.clj:208)"
25/04/2020 09:29:13    "query_processor.middleware.validate$validate_query$fn__46967.invoke(validate.clj:10)"
25/04/2020 09:29:13    "query_processor.middleware.normalize_query$normalize$fn__43016.invoke(normalize_query.clj:22)"
25/04/2020 09:29:13    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__39151.invoke(add_rows_truncated.clj:36)"
25/04/2020 09:29:13    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__46934.invoke(results_metadata.clj:128)"
25/04/2020 09:29:13    "query_processor.middleware.constraints$add_default_userland_constraints$fn__42333.invoke(constraints.clj:42)"
25/04/2020 09:29:13    "query_processor.middleware.process_userland_query$process_userland_query$fn__43738.invoke(process_userland_query.clj:136)"
25/04/2020 09:29:13    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__42276.invoke(catch_exceptions.clj:174)"
25/04/2020 09:29:13    "query_processor.reducible$async_qp$qp_STAR___37952$thunk__37953.invoke(reducible.clj:101)"
25/04/2020 09:29:13    "query_processor.reducible$async_qp$qp_STAR___37952.invoke(reducible.clj:107)"
25/04/2020 09:29:13    "query_processor.reducible$sync_qp$qp_STAR___37961$fn__37964.invoke(reducible.clj:133)"
25/04/2020 09:29:13    "query_processor.reducible$sync_qp$qp_STAR___37961.invoke(reducible.clj:132)"
25/04/2020 09:29:13    "query_processor$process_userland_query.invokeStatic(query_processor.clj:218)"
25/04/2020 09:29:13    "query_processor$process_userland_query.doInvoke(query_processor.clj:214)"
25/04/2020 09:29:13    "query_processor$fn__47133$process_query_and_save_execution_BANG___47142$fn__47145.invoke(query_processor.clj:233)"
25/04/2020 09:29:13    "query_processor$fn__47133$process_query_and_save_execution_BANG___47142.invoke(query_processor.clj:225)"
25/04/2020 09:29:13    "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186$fn__47189.invoke(query_processor.clj:245)"
25/04/2020 09:29:13    "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186.invoke(query_processor.clj:238)"
25/04/2020 09:29:13    "api.dataset$fn__50475$fn__50478.invoke(dataset.clj:52)"
25/04/2020 09:29:13    "query_processor.streaming$streaming_response_STAR_$fn__33990$fn__33991.invoke(streaming.clj:73)"
25/04/2020 09:29:13    "query_processor.streaming$streaming_response_STAR_$fn__33990.invoke(streaming.clj:72)"
25/04/2020 09:29:13    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:66)"
25/04/2020 09:29:13    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:64)"
25/04/2020 09:29:13    "async.streaming_response$do_f_async$fn__23079.invoke(streaming_response.clj:85)"],
25/04/2020 09:29:13   :state "HY000"}
25/04/2020 09:29:13  {:status :failed,
25/04/2020 09:29:13   :class clojure.lang.ExceptionInfo,
25/04/2020 09:29:13   :error "Error reducing result rows",
25/04/2020 09:29:13   :stacktrace
25/04/2020 09:29:13   ["--> query_processor.context.default$default_reducef$fn__37866.invoke(default.clj:61)"
25/04/2020 09:29:13    "query_processor.context.default$default_reducef.invokeStatic(default.clj:58)"
25/04/2020 09:29:13    "query_processor.context.default$default_reducef.invoke(default.clj:49)"
25/04/2020 09:29:13    "query_processor.context$reducef.invokeStatic(context.clj:69)"
25/04/2020 09:29:13    "query_processor.context$reducef.invoke(context.clj:62)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf$respond_STAR___37870.invoke(default.clj:70)"
25/04/2020 09:29:13    "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:392)"
25/04/2020 09:29:13    "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:377)"
25/04/2020 09:29:13    "driver.sql_jdbc$fn__72711.invokeStatic(sql_jdbc.clj:50)"
25/04/2020 09:29:13    "driver.sql_jdbc$fn__72711.invoke(sql_jdbc.clj:48)"
25/04/2020 09:29:13    "driver.athena$fn__609.invokeStatic(athena.clj:273)"
25/04/2020 09:29:13    "driver.athena$fn__609.invoke(athena.clj:271)"
25/04/2020 09:29:13    "query_processor.context$executef.invokeStatic(context.clj:59)"
25/04/2020 09:29:13    "query_processor.context$executef.invoke(context.clj:48)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf.invokeStatic(default.clj:69)"
25/04/2020 09:29:13    "query_processor.context.default$default_runf.invoke(default.clj:67)"
25/04/2020 09:29:13    "query_processor.context$runf.invokeStatic(context.clj:45)"
25/04/2020 09:29:13    "query_processor.context$runf.invoke(context.clj:39)"
25/04/2020 09:29:13    "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
25/04/2020 09:29:13    "query_processor.reducible$pivot.invoke(reducible.clj:31)"
25/04/2020 09:29:13    "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__43003.invoke(mbql_to_native.clj:26)"
25/04/2020 09:29:13    "query_processor.middleware.check_features$check_features$fn__42317.invoke(check_features.clj:42)"
25/04/2020 09:29:13    "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__43168.invoke(optimize_datetime_filters.clj:133)"
25/04/2020 09:29:13    "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__47065.invoke(wrap_value_literals.clj:137)"
25/04/2020 09:29:13    "query_processor.middleware.annotate$add_column_info$fn__40946.invoke(annotate.clj:577)"
25/04/2020 09:29:13    "query_processor.middleware.permissions$check_query_permissions$fn__42192.invoke(permissions.clj:64)"
25/04/2020 09:29:13    "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__43667.invoke(pre_alias_aggregations.clj:40)"
25/04/2020 09:29:13    "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__42390.invoke(cumulative_aggregations.clj:61)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_joins$resolve_joins$fn__44199.invoke(resolve_joins.clj:183)"
25/04/2020 09:29:13    "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__39133.invoke(add_implicit_joins.clj:245)"
25/04/2020 09:29:13    "query_processor.middleware.limit$limit$fn__42989.invoke(limit.clj:38)"
25/04/2020 09:29:13    "query_processor.middleware.format_rows$format_rows$fn__42970.invoke(format_rows.clj:81)"
25/04/2020 09:29:13    "query_processor.middleware.desugar$desugar$fn__42456.invoke(desugar.clj:22)"
25/04/2020 09:29:13    "query_processor.middleware.binning$update_binning_strategy$fn__41490.invoke(binning.clj:229)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_fields$resolve_fields$fn__41998.invoke(resolve_fields.clj:24)"
25/04/2020 09:29:13    "query_processor.middleware.add_dimension_projections$add_remapping$fn__38669.invoke(add_dimension_projections.clj:272)"
25/04/2020 09:29:13    "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__38889.invoke(add_implicit_clauses.clj:147)"
25/04/2020 09:29:13    "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__39282.invoke(add_source_metadata.clj:105)"
25/04/2020 09:29:13    "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__43864.invoke(reconcile_breakout_and_order_by_bucketing.clj:98)"
25/04/2020 09:29:13    "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__41131.invoke(auto_bucket_datetimes.clj:125)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__42045.invoke(resolve_source_table.clj:46)"
25/04/2020 09:29:13    "query_processor.middleware.parameters$substitute_parameters$fn__43649.invoke(parameters.clj:97)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__42097.invoke(resolve_referenced.clj:80)"
25/04/2020 09:29:13    "query_processor.middleware.expand_macros$expand_macros$fn__42712.invoke(expand_macros.clj:158)"
25/04/2020 09:29:13    "query_processor.middleware.add_timezone_info$add_timezone_info$fn__39313.invoke(add_timezone_info.clj:15)"
25/04/2020 09:29:13    "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__46949.invoke(splice_params_in_response.clj:32)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875$fn__43879.invoke(resolve_database_and_driver.clj:33)"
25/04/2020 09:29:13    "driver$do_with_driver.invokeStatic(driver.clj:61)"
25/04/2020 09:29:13    "driver$do_with_driver.invoke(driver.clj:57)"
25/04/2020 09:29:13    "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875.invoke(resolve_database_and_driver.clj:27)"
25/04/2020 09:29:13    "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__42918.invoke(fetch_source_query.clj:243)"
25/04/2020 09:29:13    "query_processor.middleware.store$initialize_store$fn__46958$fn__46959.invoke(store.clj:11)"
25/04/2020 09:29:13    "query_processor.store$do_with_store.invokeStatic(store.clj:46)"
25/04/2020 09:29:13    "query_processor.store$do_with_store.invoke(store.clj:40)"
25/04/2020 09:29:13    "query_processor.middleware.store$initialize_store$fn__46958.invoke(store.clj:10)"
25/04/2020 09:29:13    "query_processor.middleware.cache$maybe_return_cached_results$fn__41974.invoke(cache.clj:208)"
25/04/2020 09:29:13    "query_processor.middleware.validate$validate_query$fn__46967.invoke(validate.clj:10)"
25/04/2020 09:29:13    "query_processor.middleware.normalize_query$normalize$fn__43016.invoke(normalize_query.clj:22)"
25/04/2020 09:29:13    "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__39151.invoke(add_rows_truncated.clj:36)"
25/04/2020 09:29:13    "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__46934.invoke(results_metadata.clj:128)"
25/04/2020 09:29:13    "query_processor.middleware.constraints$add_default_userland_constraints$fn__42333.invoke(constraints.clj:42)"
25/04/2020 09:29:13    "query_processor.middleware.process_userland_query$process_userland_query$fn__43738.invoke(process_userland_query.clj:136)"
25/04/2020 09:29:13    "query_processor.middleware.catch_exceptions$catch_exceptions$fn__42276.invoke(catch_exceptions.clj:174)"
25/04/2020 09:29:13    "query_processor.reducible$async_qp$qp_STAR___37952$thunk__37953.invoke(reducible.clj:101)"
25/04/2020 09:29:13    "query_processor.reducible$async_qp$qp_STAR___37952.invoke(reducible.clj:107)"
25/04/2020 09:29:13    "query_processor.reducible$sync_qp$qp_STAR___37961$fn__37964.invoke(reducible.clj:133)"
25/04/2020 09:29:13    "query_processor.reducible$sync_qp$qp_STAR___37961.invoke(reducible.clj:132)"
25/04/2020 09:29:13    "query_processor$process_userland_query.invokeStatic(query_processor.clj:218)"
25/04/2020 09:29:13    "query_processor$process_userland_query.doInvoke(query_processor.clj:214)"
25/04/2020 09:29:13    "query_processor$fn__47133$process_query_and_save_execution_BANG___47142$fn__47145.invoke(query_processor.clj:233)"
25/04/2020 09:29:13    "query_processor$fn__47133$process_query_and_save_execution_BANG___47142.invoke(query_processor.clj:225)"
25/04/2020 09:29:13    "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186$fn__47189.invoke(query_processor.clj:245)"
25/04/2020 09:29:13    "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186.invoke(query_processor.clj:238)"
25/04/2020 09:29:13    "api.dataset$fn__50475$fn__50478.invoke(dataset.clj:52)"
25/04/2020 09:29:13    "query_processor.streaming$streaming_response_STAR_$fn__33990$fn__33991.invoke(streaming.clj:73)"
25/04/2020 09:29:13    "query_processor.streaming$streaming_response_STAR_$fn__33990.invoke(streaming.clj:72)"
25/04/2020 09:29:13    "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:66)"
25/04/2020 09:29:13    "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:64)"
25/04/2020 09:29:13    "async.streaming_response$do_f_async$fn__23079.invoke(streaming_response.clj:85)"],
25/04/2020 09:29:13   :error_type :qp,
25/04/2020 09:29:13   :ex-data {:type :qp}}],
25/04/2020 09:29:13 :error_type :qp,
25/04/2020 09:29:13 :json_query {:database 2, :query {:source-table 42}, :type "query", :parameters [], :middleware {:add-default-userland-constraints? true}},
25/04/2020 09:29:13 :native
25/04/2020 09:29:13 {:query
25/04/2020 09:29:13  "SELECT \"db\".\"activity_log\".\"id\" AS \"id\", \"db\".\"activity_log\".\"causer_id\" AS \"causer_id\", \"db\".\"activity_log\".\"causer_type\" AS \"causer_type\", \"db\".\"activity_log\".\"created_at\" AS \"created_at\", \"db\".\"activity_log\".\"description\" AS \"description\", \"db\".\"activity_log\".\"log_name\" AS \"log_name\", \"db\".\"activity_log\".\"properties\" AS \"properties\", \"db\".\"activity_log\".\"subject_id\" AS \"subject_id\", \"db\".\"activity_log\".\"subject_type\" AS \"subject_type\", \"db\".\"activity_log\".\"updated_at\" AS \"updated_at\" FROM \"db\".\"activity_log\" LIMIT 2000",
25/04/2020 09:29:13  :params nil},
25/04/2020 09:29:13 :status :failed,
25/04/2020 09:29:13 :class java.lang.IllegalArgumentException,
25/04/2020 09:29:13 :stacktrace
25/04/2020 09:29:13 ["java.sql/java.sql.Timestamp.valueOf(Unknown Source)"
25/04/2020 09:29:13  "com.simba.athena.athena.dataengine.AJBaseResultSet.getData(Unknown Source)"
25/04/2020 09:29:13  "com.simba.athena.jdbc.common.SForwardResultSet.getData(Unknown Source)"
25/04/2020 09:29:13  "com.simba.athena.jdbc.common.SForwardResultSet.getString(Unknown Source)"
25/04/2020 09:29:13  "com.mchange.v2.c3p0.impl.NewProxyResultSet.getString(NewProxyResultSet.java:4189)"
25/04/2020 09:29:13  "--> driver.sql_jdbc.execute.legacy_impl$fn__72760$fn__72761.invoke(legacy_impl.clj:77)"
25/04/2020 09:29:13  "driver.sql_jdbc.execute$row_thunk$row_thunk_STAR___70724.invoke(execute.clj:348)"
25/04/2020 09:29:13  "query_processor.reducible$reducible_rows$reify__37968.reduce(reducible.clj:156)"
25/04/2020 09:29:13  "query_processor.context.default$default_reducef$fn__37866.invoke(default.clj:59)"
25/04/2020 09:29:13  "query_processor.context.default$default_reducef.invokeStatic(default.clj:58)"
25/04/2020 09:29:13  "query_processor.context.default$default_reducef.invoke(default.clj:49)"
25/04/2020 09:29:13  "query_processor.context$reducef.invokeStatic(context.clj:69)"
25/04/2020 09:29:13  "query_processor.context$reducef.invoke(context.clj:62)"
25/04/2020 09:29:13  "query_processor.context.default$default_runf$respond_STAR___37870.invoke(default.clj:70)"
25/04/2020 09:29:13  "driver.sql_jdbc.execute$execute_reducible_query.invokeStatic(execute.clj:392)"
25/04/2020 09:29:13  "driver.sql_jdbc.execute$execute_reducible_query.invoke(execute.clj:377)"
25/04/2020 09:29:13  "driver.sql_jdbc$fn__72711.invokeStatic(sql_jdbc.clj:50)"
25/04/2020 09:29:13  "driver.sql_jdbc$fn__72711.invoke(sql_jdbc.clj:48)"
25/04/2020 09:29:13  "driver.athena$fn__609.invokeStatic(athena.clj:273)"
25/04/2020 09:29:13  "driver.athena$fn__609.invoke(athena.clj:271)"
25/04/2020 09:29:13  "query_processor.context$executef.invokeStatic(context.clj:59)"
25/04/2020 09:29:13  "query_processor.context$executef.invoke(context.clj:48)"
25/04/2020 09:29:13  "query_processor.context.default$default_runf.invokeStatic(default.clj:69)"
25/04/2020 09:29:13  "query_processor.context.default$default_runf.invoke(default.clj:67)"
25/04/2020 09:29:13  "query_processor.context$runf.invokeStatic(context.clj:45)"
25/04/2020 09:29:13  "query_processor.context$runf.invoke(context.clj:39)"
25/04/2020 09:29:13  "query_processor.reducible$pivot.invokeStatic(reducible.clj:34)"
25/04/2020 09:29:13  "query_processor.reducible$pivot.invoke(reducible.clj:31)"
25/04/2020 09:29:13  "query_processor.middleware.mbql_to_native$mbql__GT_native$fn__43003.invoke(mbql_to_native.clj:26)"
25/04/2020 09:29:13  "query_processor.middleware.check_features$check_features$fn__42317.invoke(check_features.clj:42)"
25/04/2020 09:29:13  "query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__43168.invoke(optimize_datetime_filters.clj:133)"
25/04/2020 09:29:13  "query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__47065.invoke(wrap_value_literals.clj:137)"
25/04/2020 09:29:13  "query_processor.middleware.annotate$add_column_info$fn__40946.invoke(annotate.clj:577)"
25/04/2020 09:29:13  "query_processor.middleware.permissions$check_query_permissions$fn__42192.invoke(permissions.clj:64)"
25/04/2020 09:29:13  "query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__43667.invoke(pre_alias_aggregations.clj:40)"
25/04/2020 09:29:13  "query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__42390.invoke(cumulative_aggregations.clj:61)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_joins$resolve_joins$fn__44199.invoke(resolve_joins.clj:183)"
25/04/2020 09:29:13  "query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__39133.invoke(add_implicit_joins.clj:245)"
25/04/2020 09:29:13  "query_processor.middleware.limit$limit$fn__42989.invoke(limit.clj:38)"
25/04/2020 09:29:13  "query_processor.middleware.format_rows$format_rows$fn__42970.invoke(format_rows.clj:81)"
25/04/2020 09:29:13  "query_processor.middleware.desugar$desugar$fn__42456.invoke(desugar.clj:22)"
25/04/2020 09:29:13  "query_processor.middleware.binning$update_binning_strategy$fn__41490.invoke(binning.clj:229)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_fields$resolve_fields$fn__41998.invoke(resolve_fields.clj:24)"
25/04/2020 09:29:13  "query_processor.middleware.add_dimension_projections$add_remapping$fn__38669.invoke(add_dimension_projections.clj:272)"
25/04/2020 09:29:13  "query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__38889.invoke(add_implicit_clauses.clj:147)"
25/04/2020 09:29:13  "query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__39282.invoke(add_source_metadata.clj:105)"
25/04/2020 09:29:13  "query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__43864.invoke(reconcile_breakout_and_order_by_bucketing.clj:98)"
25/04/2020 09:29:13  "query_processor.middleware.auto_bucket_datetimes$auto_bucket_datetimes$fn__41131.invoke(auto_bucket_datetimes.clj:125)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_source_table$resolve_source_tables$fn__42045.invoke(resolve_source_table.clj:46)"
25/04/2020 09:29:13  "query_processor.middleware.parameters$substitute_parameters$fn__43649.invoke(parameters.clj:97)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_referenced$resolve_referenced_card_resources$fn__42097.invoke(resolve_referenced.clj:80)"
25/04/2020 09:29:13  "query_processor.middleware.expand_macros$expand_macros$fn__42712.invoke(expand_macros.clj:158)"
25/04/2020 09:29:13  "query_processor.middleware.add_timezone_info$add_timezone_info$fn__39313.invoke(add_timezone_info.clj:15)"
25/04/2020 09:29:13  "query_processor.middleware.splice_params_in_response$splice_params_in_response$fn__46949.invoke(splice_params_in_response.clj:32)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875$fn__43879.invoke(resolve_database_and_driver.clj:33)"
25/04/2020 09:29:13  "driver$do_with_driver.invokeStatic(driver.clj:61)"
25/04/2020 09:29:13  "driver$do_with_driver.invoke(driver.clj:57)"
25/04/2020 09:29:13  "query_processor.middleware.resolve_database_and_driver$resolve_database_and_driver$fn__43875.invoke(resolve_database_and_driver.clj:27)"
25/04/2020 09:29:13  "query_processor.middleware.fetch_source_query$resolve_card_id_source_tables$fn__42918.invoke(fetch_source_query.clj:243)"
25/04/2020 09:29:13  "query_processor.middleware.store$initialize_store$fn__46958$fn__46959.invoke(store.clj:11)"
25/04/2020 09:29:13  "query_processor.store$do_with_store.invokeStatic(store.clj:46)"
25/04/2020 09:29:13  "query_processor.store$do_with_store.invoke(store.clj:40)"
25/04/2020 09:29:13  "query_processor.middleware.store$initialize_store$fn__46958.invoke(store.clj:10)"
25/04/2020 09:29:13  "query_processor.middleware.cache$maybe_return_cached_results$fn__41974.invoke(cache.clj:208)"
25/04/2020 09:29:13  "query_processor.middleware.validate$validate_query$fn__46967.invoke(validate.clj:10)"
25/04/2020 09:29:13  "query_processor.middleware.normalize_query$normalize$fn__43016.invoke(normalize_query.clj:22)"
25/04/2020 09:29:13  "query_processor.middleware.add_rows_truncated$add_rows_truncated$fn__39151.invoke(add_rows_truncated.clj:36)"
25/04/2020 09:29:13  "query_processor.middleware.results_metadata$record_and_return_metadata_BANG_$fn__46934.invoke(results_metadata.clj:128)"
25/04/2020 09:29:13  "query_processor.middleware.constraints$add_default_userland_constraints$fn__42333.invoke(constraints.clj:42)"
25/04/2020 09:29:13  "query_processor.middleware.process_userland_query$process_userland_query$fn__43738.invoke(process_userland_query.clj:136)"
25/04/2020 09:29:13  "query_processor.middleware.catch_exceptions$catch_exceptions$fn__42276.invoke(catch_exceptions.clj:174)"
25/04/2020 09:29:13  "query_processor.reducible$async_qp$qp_STAR___37952$thunk__37953.invoke(reducible.clj:101)"
25/04/2020 09:29:13  "query_processor.reducible$async_qp$qp_STAR___37952.invoke(reducible.clj:107)"
25/04/2020 09:29:13  "query_processor.reducible$sync_qp$qp_STAR___37961$fn__37964.invoke(reducible.clj:133)"
25/04/2020 09:29:13  "query_processor.reducible$sync_qp$qp_STAR___37961.invoke(reducible.clj:132)"
25/04/2020 09:29:13  "query_processor$process_userland_query.invokeStatic(query_processor.clj:218)"
25/04/2020 09:29:13  "query_processor$process_userland_query.doInvoke(query_processor.clj:214)"
25/04/2020 09:29:13  "query_processor$fn__47133$process_query_and_save_execution_BANG___47142$fn__47145.invoke(query_processor.clj:233)"
25/04/2020 09:29:13  "query_processor$fn__47133$process_query_and_save_execution_BANG___47142.invoke(query_processor.clj:225)"
25/04/2020 09:29:13  "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186$fn__47189.invoke(query_processor.clj:245)"
25/04/2020 09:29:13  "query_processor$fn__47177$process_query_and_save_with_max_results_constraints_BANG___47186.invoke(query_processor.clj:238)"
25/04/2020 09:29:13  "api.dataset$fn__50475$fn__50478.invoke(dataset.clj:52)"
25/04/2020 09:29:13  "query_processor.streaming$streaming_response_STAR_$fn__33990$fn__33991.invoke(streaming.clj:73)"
25/04/2020 09:29:13  "query_processor.streaming$streaming_response_STAR_$fn__33990.invoke(streaming.clj:72)"
25/04/2020 09:29:13  "async.streaming_response$do_f_STAR_.invokeStatic(streaming_response.clj:66)"
25/04/2020 09:29:13  "async.streaming_response$do_f_STAR_.invoke(streaming_response.clj:64)"
25/04/2020 09:29:13  "async.streaming_response$do_f_async$fn__23079.invoke(streaming_response.clj:85)"],
25/04/2020 09:29:13 :context :ad-hoc,
25/04/2020 09:29:13 :error "[JDBC Driver]Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]",
25/04/2020 09:29:13 :row_count 0,
25/04/2020 09:29:13 :running_time 0,
25/04/2020 09:29:13 :preprocessed
25/04/2020 09:29:13 {:database 2,
25/04/2020 09:29:13  :query
25/04/2020 09:29:13  {:source-table 42,
25/04/2020 09:29:13   :fields
25/04/2020 09:29:13   [[:field-id 354]
25/04/2020 09:29:13    [:field-id 351]
25/04/2020 09:29:13    [:field-id 350]
25/04/2020 09:29:13    [:datetime-field [:field-id 353] :default]
25/04/2020 09:29:13    [:field-id 356]
25/04/2020 09:29:13    [:field-id 352]
25/04/2020 09:29:13    [:field-id 358]
25/04/2020 09:29:13    [:field-id 349]
25/04/2020 09:29:13    [:field-id 357]
25/04/2020 09:29:13    [:datetime-field [:field-id 355] :default]],
25/04/2020 09:29:13   :limit 2000},
25/04/2020 09:29:13  :type :query,
25/04/2020 09:29:13  :middleware {:add-default-userland-constraints? true},
25/04/2020 09:29:13  :info
25/04/2020 09:29:13  {:executed-by 1,
25/04/2020 09:29:13   :context :ad-hoc,
25/04/2020 09:29:13   :nested? false,
25/04/2020 09:29:13   :query-hash [27, -74, 84, 30, 15, -57, -13, 8, 52, -93, 59, 102, -33, -115, 90, -20, 92, -57, 68, -60, 126, 104, 91, -84, 113, -107, -110, -49, -39, 66, -89, 95]},
25/04/2020 09:29:13  :constraints {:max-results 10000, :max-results-bare-rows 2000}},
25/04/2020 09:29:13 :data {:rows [], :cols []}}
25/04/2020 09:29:13
25/04/2020 09:29:1304-25 08:29:13 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 7.3 s (11 DB calls) App DB connections: 0/7 Jetty threads: 2/50 (5 idle, 0 queued) (91 total active threads) Queries in flight: 1 (0 queued)

I've confirmed with the devs that the date/timestamp fields are datetime and Glue correctly picks this up (as timestamp), and in Metabase they're showing as "DateTime".

Apologies if this is actually more a Metabase issue than this driver but I'm going from the first few lines of the stacktrace that mention Athena and hopefully not adding 2+2 together and getting 5.

Appreciate any help and thanks for writing this driver!

Query error even if table exists

I am trying to run a SQL command on a table in Athena but it raise an error table not found even if the table exist in the database. Could you please resolve it?

Could not locate java_time__init.class, java_time.clj or java_time.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name

Leiningen's classpath: /root/.lein/self-installs/leiningen-2.9.1-standalone.jar
Applying task uberjar to []
Applying task javac to nil
Running javac with [-target 1.8 -source 1.8 @/tmp/.leiningen-cmdline5556600515931511799.tmp]
Applying task compile to nil
Compiling metabase.driver.athena
Syntax error compiling at (athena.clj:1:1).
Exception in thread "main" Syntax error compiling at (athena.clj:1:1).
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3707)
at clojure.lang.Compiler.compile1(Compiler.java:7730)
at clojure.lang.Compiler.compile1(Compiler.java:7720)
at clojure.lang.Compiler.compile(Compiler.java:7797)
at clojure.lang.RT.compile(RT.java:415)
at clojure.lang.RT.load(RT.java:461)
at clojure.lang.RT.load(RT.java:428)
at clojure.core$load$fn__6824.invoke(core.clj:6126)
at clojure.core$load.invokeStatic(core.clj:6125)
at clojure.core$load.doInvoke(core.clj:6109)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5908)
at clojure.core$compile$fn__6829.invoke(core.clj:6136)
at clojure.core$compile.invokeStatic(core.clj:6136)
at clojure.core$compile.invoke(core.clj:6128)
at user$eval155$fn__164.invoke(form-init8047446358687895041.clj:1)
at user$eval155.invokeStatic(form-init8047446358687895041.clj:1)
at user$eval155.invoke(form-init8047446358687895041.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:7176)
at clojure.lang.Compiler.eval(Compiler.java:7166)
at clojure.lang.Compiler.eval(Compiler.java:7166)
at clojure.lang.Compiler.load(Compiler.java:7635)
at clojure.lang.Compiler.loadFile(Compiler.java:7573)
at clojure.main$load_script.invokeStatic(main.clj:452)
at clojure.main$init_opt.invokeStatic(main.clj:454)
at clojure.main$init_opt.invoke(main.clj:454)
at clojure.main$initialize.invokeStatic(main.clj:485)
at clojure.main$null_opt.invokeStatic(main.clj:519)
at clojure.main$null_opt.invoke(main.clj:516)
at clojure.main$main.invokeStatic(main.clj:598)
at clojure.main$main.doInvoke(main.clj:561)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.main.main(main.java:37)
Caused by: java.io.FileNotFoundException: Could not locate java_time__init.class, java_time.clj or java_time.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.
at clojure.lang.RT.load(RT.java:466)
at clojure.lang.RT.load(RT.java:428)
at clojure.core$load$fn__6824.invoke(core.clj:6126)
at clojure.core$load.invokeStatic(core.clj:6125)
at clojure.core$load.doInvoke(core.clj:6109)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invokeStatic(core.clj:5908)
at clojure.core$load_one.invoke(core.clj:5903)
at clojure.core$load_lib$fn__6765.invoke(core.clj:5948)
at clojure.core$load_lib.invokeStatic(core.clj:5947)
at clojure.core$load_lib.doInvoke(core.clj:5928)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$load_libs.invokeStatic(core.clj:5985)
at clojure.core$load_libs.doInvoke(core.clj:5969)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$require.invokeStatic(core.clj:6007)
at clojure.core$require.doInvoke(core.clj:6007)
at clojure.lang.RestFn.invoke(RestFn.java:3204)
at metabase.driver.athena$loading__6706__auto____171.invoke(athena.clj:1)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3702)
... 33 more
Compilation failed: Subprocess failed

Query timeout issue

Metabase version: 0.35.4
Athena Driver version: v1.0.0

We have an Athena database which isn't optimized, so queries to it often time out after 5 minutes with the following message on the front end:

I checked in the Athena console and these queries are timing out at ~298s (~5 minutes).

In the metabase logs I see the following:

Caused by: com.simba.athena.dsi.exceptions.OperationCanceledException: [Simba][AthenaJDBC](100081) Query execution got cancelled for this query

I've tried setting the MB_DB_CONNECTION_TIMEOUT_MS env var to 1200000 (20 minutes), but it had no effect. A cursory google search showed that this was something that came up with the Druid driver, and it was changed.

If I'm not missing an existing configuration parameter, would it be possible to get something similar for this Athena driver? Thanks so much in advance!

Add support for parameterized queries

Currently, ? is not getting replaced in the queries so any queries with a parameter will fail.

I'm guessing some sort of unprepare method needs to be overridden, but still figuring that out. :)

Required Table SerDe information is not populated

Thanks for creating an maintaining this plugin!

We are seeing this error on certain tables. Do you know what the root cause might be?

An error has been thrown from the AWS Athena client. HIVE_METASTORE_ERROR: com.facebook.presto.spi.PrestoException: Required Table SerDe information is not populated

Metabase v0.34 is not displaying date/datetime fields properly

As noted in metabase/metabase#11607.

Will try to take a look at both this and #26 within the next few days.

Group names not fully visible on screen

I am running metabase on my local machine but under the permissions option, I cannot see the group names properly. Can anyone suggest me what to do?

It is visible like this

Better support for federated data sources

Federated data sources can, sometimes silently, fail the sync step as seen in #55 and #63.

We should have better error handling for those.

Building from source error

Hi,

I tried to build from source on Ubuntu 18.10 following the readme instructions, but got the following error. Unfortunately I've near 0 experience with building jar packages, so maybe it's something very obvious that I'm not doing correctly:

Applying task uberjar to []
Applying task javac to nil
Running javac with [-target 1.8 -source 1.8 @/tmp/.leiningen-cmdline15224665806995874236.tmp]
Applying task compile to nil
Compiling metabase.driver.athena
Syntax error compiling at (metabase/driver/athena.clj:142:1).
Exception in thread "main" Syntax error compiling at (metabase/driver/athena.clj:142:1).
        at clojure.lang.Compiler.analyze(Compiler.java:6808)
        at clojure.lang.Compiler.analyze(Compiler.java:6745)
        at clojure.lang.Compiler$HostExpr$Parser.parse(Compiler.java:986)
        at clojure.lang.Compiler.analyzeSeq(Compiler.java:7106)
        at clojure.lang.Compiler.analyze(Compiler.java:6789)
        at clojure.lang.Compiler.analyze(Compiler.java:6745)
        at clojure.lang.Compiler.compile1(Compiler.java:7725)
        at clojure.lang.Compiler.compile(Compiler.java:7797)
        at clojure.lang.RT.compile(RT.java:415)
        at clojure.lang.RT.load(RT.java:461)
        at clojure.lang.RT.load(RT.java:428)
        at clojure.core$load$fn__6824.invoke(core.clj:6126)
        at clojure.core$load.invokeStatic(core.clj:6125)
        at clojure.core$load.doInvoke(core.clj:6109)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at clojure.core$load_one.invokeStatic(core.clj:5908)
        at clojure.core$compile$fn__6829.invoke(core.clj:6136)
        at clojure.core$compile.invokeStatic(core.clj:6136)
        at clojure.core$compile.invoke(core.clj:6128)
        at user$eval155$fn__164.invoke(form-init282522768802351786.clj:1)
        at user$eval155.invokeStatic(form-init282522768802351786.clj:1)
        at user$eval155.invoke(form-init282522768802351786.clj:1)
        at clojure.lang.Compiler.eval(Compiler.java:7176)
        at clojure.lang.Compiler.eval(Compiler.java:7166)
        at clojure.lang.Compiler.eval(Compiler.java:7166)
        at clojure.lang.Compiler.load(Compiler.java:7635)
        at clojure.lang.Compiler.loadFile(Compiler.java:7573)
        at clojure.main$load_script.invokeStatic(main.clj:452)
        at clojure.main$init_opt.invokeStatic(main.clj:454)
        at clojure.main$init_opt.invoke(main.clj:454)
        at clojure.main$initialize.invokeStatic(main.clj:485)
        at clojure.main$null_opt.invokeStatic(main.clj:519)
        at clojure.main$null_opt.invoke(main.clj:516)
        at clojure.main$main.invokeStatic(main.clj:598)
        at clojure.main$main.doInvoke(main.clj:561)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.lang.Var.applyTo(Var.java:705)
        at clojure.main.main(main.java:37)
Caused by: java.lang.RuntimeException: No such var: driver/date-add
        at clojure.lang.Util.runtimeException(Util.java:221)
        at clojure.lang.Compiler.resolveIn(Compiler.java:7387)
        at clojure.lang.Compiler.resolve(Compiler.java:7357)
        at clojure.lang.Compiler.analyzeSymbol(Compiler.java:7318)
        at clojure.lang.Compiler.analyze(Compiler.java:6768)
        ... 37 more
Compilation failed: Subprocess failed

Docs: AWS IAM Policy

Thanks for the work on this @dacort.

I've been trying to set up the plugin with Metabase with a minimal IAM policy but I can't figure out exactly what's needed. I know it needs to access Athena,S3 and Glue but would it be possible to have an IAM Policy to share and add to https://github.com/dacort/metabase-athena-driver#configuring ?

No columns

Hi, I'm trying to setup Athena as a Metabase database. I already added the driver and configured the database. It detects my tables, but it doesn't detect their columns. Can you please help me? It's an IAM permission problem?

Unable to view tables in database

I added a redshift database yesterday and i was able to see the data in tables through querying or filtering. But today when i opened the tables to see the data it is just loading and i cannot view data in tables. What is the reason behind this? Even after syncing my database i am unable to see the data.

Upgradation of metabase

I want to upgrade my metabase version but when I tried it gives me docker image of metabase/metabase. How to upgrade metabase-athena which I am using from this repository?

Support 0.36 release

0.36 requires a database_position field to be returned per https://discourse.metabase.com/t/database-sync-issue-with-new-version-0-36-0/11028/

AWS Credentials Profiles

Shouldn't passing Profile=Profile_Name in the Additional Athena connection string options field work to select credentials stored in the ~/.aws/credentials file? We have a tool that generates our temporary credentials and automatically refreshes them.

[ProfileName]
aws_access_key_id=
aws_secret_access_key=
aws_session_token=
region=

It doesn't appear that the driver is picking up these creds; however, if I export them as environment variables instead, it works.
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN

Driver doesn't work anymore on latest metabase version 0.38.0

Updated to the new metabase version (v0.38.0) this morning and it seems like it broke the connection to Athena, as I'm not able to query it anymore from Metabase. This is the error I'm getting when trying to query it:
No suitable driver found for jdbc:awsathena://athena.eu-west-1.amazonaws.com:443

consistent "Error syncing Fields for Table"

Hey @dacort, I started hitting this at some point in the recent past.

I did some looking around and it seems like this is driver related. There's no trace I can provide you with right now, but here's a snippet of logs:

[9f6bbde4-1ab5-4d3e-abf5-c92e4b14d26c] 2020-08-31T09:00:05-04:00 INFO metabase.sync.util STARTING: step 'sync-timezone' for athena Database 3 'Datalake'
[9f6bbde4-1ab5-4d3e-abf5-c92e4b14d26c] 2020-08-31T09:00:06-04:00 INFO metabase.sync.util FINISHED: step 'sync-tables' for athena Database 3 'Datalake' (630.1 ms)
[9f6bbde4-1ab5-4d3e-abf5-c92e4b14d26c] 2020-08-31T09:00:06-04:00 INFO metabase.sync.util STARTING: step 'sync-fields' for athena Database 3 'Datalake'
[9f6bbde4-1ab5-4d3e-abf5-c92e4b14d26c] 2020-08-31T09:00:06-04:00 ERROR metabase.sync.util Error syncing Fields for Table foo```

```{
  "browser-info": {
    "language": "en-CA",
    "platform": "MacIntel",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:79.0) Gecko/20100101 Firefox/79.0",
    "vendor": ""
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.8+10",
    "java.vendor": "AdoptOpenJDK",
    "java.vendor.url": "https://adoptopenjdk.net/",
    "java.version": "11.0.8",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.8+10",
    "os.name": "Linux",
    "os.version": "4.14.181-108.257.amzn1.x86_64",
    "user.language": "en",
    "user.timezone": "GMT"
  },
  "metabase-info": {
    "databases": [
      "postgres",
      "athena"
    ],
    "hosting-env": "unknown",
    "application-database": "postgres",
    "application-database-details": {
      "database": {
        "name": "PostgreSQL",
        "version": "11.8"
      },
      "jdbc-driver": {
        "name": "PostgreSQL JDBC Driver",
        "version": "42.2.8"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2020-08-17",
      "tag": "v0.36.4",
      "branch": "release-0.36.x",
      "hash": "196c1f6"
    },
    "settings": {
      "report-timezone": null
    }
  }
}```

Migrating Metabase application data from H2 to MySQL

Hello there,

I have been using the docker file to start and run Metabase with a pre-loaded Athena driver which by default stores all the application data inside an H2 database. However, now want to configure Metabase to use MySQL instead of H2 as its application database. I know there are some notes on the Metabase website to do this task. However, I'm not sure how to incorporate those configurations inside the docker container. Can you please suggest a way to do it for Metabase with Athena-driver?

Thank you
Shalabh

Ensure timezone correctness

Currently, when queries are built, we'll use the user.timezone setting. This could lead to weirdness because timestamps in Athena are UTC.

For example, a simple query that tries to filter for a timestamp in the past 30 days results in a query that looks like this:

SELECT 
  "default"."mb_relative"."id" AS "id", "default"."mb_relative"."name" AS "name",
  "default"."mb_relative"."timestamp" AS "timestamp"
FROM "default"."mb_relative"
WHERE (
  "default"."mb_relative"."timestamp" >= timestamp '2020-01-01 00:00 America/Los_Angeles'
  AND "default"."mb_relative"."timestamp" < timestamp '2020-01-04 01:01 America/Los_Angeles'
) LIMIT 2000

Note that the time comparison uses the America/Los_Angeles timezone, which will lead to incorrect (or at best misleading) results.

I'm unsure of the right approach here - perhaps we hardcode UTC in the unprepare/unprepare-value functions? Or maybe something to do with driver/db-default-timezone?

Error when using relative date query in metabase 0.34.0

Hi,
first, thanks for the driver :)

While testing out the new Metabase 0.34.0 Version i stumbled upon an error when doing relative date queries, for example when filtering 'today' on a date field.
I receive this error when using your provided jar from the release section:

I tried rebuilding the athena driver from source. The resulting jar works fine when using metabase 0.33.7.3 but again i get an error in 0.34.0 but a different one this time:

I quickly looked at the metabase source code and saw that in the release a lot a work was done concerning timezones and dates. Also the structure of the files changed, for example metabase/util/date.clj is now called metabase/util/date_2.clj.
Maybe this has something to do with this?

New JDBC Driver/Catalog access requires additional IAM permissions

I have a limited IAM user I use for connecting and when upgrading to the latest JDBC driver that supports multiple catalogs I got a few IAM permissions. I should update the readme with these additional permissions and also test if they're required for all use cases, or only when multiple catalogs exist.

athena:ListDataCatalogs
athena:ListDatabases
athena:ListTableMetadata
athena:GetDataCatalog

In addition, if you're using Athena Federation, the IAM user needs lambda:invokeFunction access as well.

Continuous events in Amazon CloudTrail

Hello. I recently started using metabase-athena-driver and observed that there are huge number of events in Amazon CloudTrail. I only fired a single query but there were a lot of events in CloudTrail. Also these events are coming continuously till the time metabase UI is open resulting in large number of events. Seems like some kind of polling. There are no queries in athena history corresponding to these events. Any reason why these events coming? And is there a way can stop these? Attaching a screenshot for the events. We are flooded with these.

Edit: So I just figured out that it is trying to get the queryExecution. So my question now is that even if there is no query that is pending why is it still hitting getQueryExecution? Is there a way to avoid it?

Athena Driver version: v1.0.0

Provide :status in driver response

Queries execute successfully, but there's a WARN log message regarding the driver response.

05-09 08:39:13 WARN middleware.async :: Unhandled exception, exepected `catch-exceptions` middleware to handle it.
java.lang.Exception: Invalid response from database driver. No :status provided.true
	at metabase.query_processor.middleware.process_userland_query$format_userland_query_result.invokeStatic(process_userland_query.clj:97)
	at metabase.query_processor.middleware.process_userland_query$format_userland_query_result.invoke(process_userland_query.clj:87)
	at clojure.core$partial$fn__5828.invoke(core.clj:2638)
	at metabase.query_processor.middleware.async_wait$wait_for_permit$fn__33563$fn__33602$state_machine__8574__auto____33623$fn__33625.invoke(async_wait.clj:49)
	at metabase.query_processor.middleware.async_wait$wait_for_permit$fn__33563$fn__33602$state_machine__8574__auto____33623.invoke(async_wait.clj:49)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__8592.invoke(ioc_macros.clj:986)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__3478$fn__3479.invoke(channels.clj:95)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

And I got the following error:

[Simba]AthenaJDBC An error has been thrown from the AWS Athena client. SYNTAX_ERROR: line 2:9: Unknown type: float

Looking at the logs, the query executes a cast to float to the field1

Do you know if is it a problem with the driver? may be I can change the datatype mappings?

dacort / metabase-athena-driver Goto Github PK

metabase-athena-driver's People

Contributors

Stargazers

Watchers

Forkers

metabase-athena-driver's Issues

Recommend Projects

Recommend Topics

Recommend Org