fredrikhgrelland / data-mesh Goto Github PK
View Code? Open in Web Editor NEWA cloud native data mesh implementation
License: Apache License 2.0
A cloud native data mesh implementation
License: Apache License 2.0
Documentation https://www.postgresql.org/docs/12/app-pg-isready.html
Consul-checks mark example csv as unhealthy, but data is actually available via both presto and hue.
This has been started and is close to finished. In branch feature/vagrantfile in m88614's fork
Remove all code not used in the first release. Create a separate branch with the original content as a reference point.
Step 8/17 : COPY ca_certificates/* /usr/local/share/ca-certificates/
COPY failed: no source files were specified
Add dummy source so builds auto-run
proper documentation of how to fo everything from vagrant up to the point where you have a hue-dashboard in the browser.
I think fredrikhgrelland/hive:3.1.0
image on docker hub was pushed with wrong entrypoint.sh
or outdated.
It fails on line 90. Latest entrypoint.sh
has 46 lines total.
...
hive-metastore | - Setting hive.metastore.warehouse.dir=s3a://hive/warehouse
hive-metastore | Configuring hivemetastore
hive-metastore | Configuring metastore
hive-metastore | /usr/local/bin/entrypoint.sh: line 90: exec: hivemetastore: not found
hive-metastore exited with code 127
fredrikhgrelland/hive:3.1.0
from dockerhubmake hive
Add explicit service names
Add presto (standalone) to current stack
Refactor so that there is a resource
folder with all resources (hcl-files, ansible playbooks, etc.)
#13 proposed health check for testdata batch job.
Should we do this?
Relates to timing problem:
TODO:
update
in hive-connect.hcl
to job levelhealthy_deadline
and progress_deadline
Even though hive-server starts successfully and is marked as healthy in consul, it occasionally goes back to unhealthy for a second, before going back to healthy. Looks to be stable after one "blip" of unhealthy.
Steps to reproduce:
vagrant up
in data-mesh
rootmake minio
make hive
make presto
presto.hcl
templatestemplate {
destination = "/local/presto/catalog/hive.properties"
data = <<EOH
...
hive.s3.aws-access-key=$MINIO_ACCESS_KEY
hive.s3.aws-secret-key=$MINIO_SECRET_KEY
...
EOH
}
template {
destination = "/local/presto/catalog/hive.properties"
data = <<EOH
...
hive.s3.aws-access-key={{ env "MINIO_ACCESS_KEY" }}
hive.s3.aws-secret-key={{ env "MINIO_SECRET_KEY" }}
...
EOH
}
template {
destination = "/local/presto/catalog/hive.properties"
data = <<EOH
...
hive.s3.aws-access-key=adminminio
hive.s3.aws-secret-key=adminminio
...
EOH
}
From what I can gather these are no longer in use. Please correct me if I'm wrong.
presto-connect.hcl
file. Has been replaced by presto.hcl
.build certificate-handler
rule in Makefilepresto
rule in Makefile. Has been replaced by rule presto1
.presto1
rule in Makefile to presto
, and change make presto1
to make presto
in README.md
Could potentially leave presto-connect.hcl
as an example of a setup with more than one presto. If so, refactor so that its function as an example is obvious.
Spin up the system that lies in the branch feature/vagrant in m88614's fork. Check that hue works correctly and manages to query the example data.
Reference from docker-hive: https://github.com/fredrikhgrelland/docker-hive/tree/master/test
User token does not work, must be changed to use master token when creating proxy connection.
Makefile. Names collision and presto1 does not exist
https://github.com/fredrikhgrelland/data-mesh/blob/master/Makefile#L75-L83
presto: presto presto-service-update
presto:
NOMAD_ADDR=http://${HOST_DOCKER}:4646 nomad stop -purge presto | true
sleep 2
NOMAD_ADDR=http://${HOST_DOCKER}:4646 nomad run nomad-jobs/presto-connect.hcl
sleep 10
curl -s -H X-Consul-Token:${CONSUL_MASTER_TOKEN} -X POST -d '{"SourceName": "presto", "DestinationName": "hive-metastore", "SourceType": "consul", "Action": "allow"}' http://127.0.0.1:8500/v1/connect/intentions
curl -s -H X-Consul-Token:${CONSUL_MASTER_TOKEN} -X POST -d '{"SourceName": "presto", "DestinationName": "minio", "SourceType": "consul", "Action": "allow"}' http://127.0.0.1:8500/v1/connect/intentions
and here https://github.com/fredrikhgrelland/data-mesh/blob/master/Makefile#L203
up: minio hive presto1 example-csv
I see the same. Wait-for needs to be changed for something more apropriate. Maybe we should extend a base-container to do a consul health check by looping over a call to the health-check api in consul?
Originally posted by @fredrikhgrelland in #33 (comment)
~/src/github.com/zhenik/data-mesh │ master *2 !1 ?2 make example-csv 2 х │ 20s │ 14:06:12
NOMAD_ADDR=http://172.17.0.1:4646 nomad stop -purge example-csv | true
No job(s) with prefix or id "example-csv" found
sleep 2
NOMAD_ADDR=http://172.17.0.1:4646 nomad run nomad-jobs/example-csv.hcl
Error submitting job: Unexpected response code: 500 (1 error occurred:
* Task group test validation failed: 1 error occurred:
* Task group service validation failed: 1 error occurred:
* Check testdata-available-via-beeline invalid: refers to non-existent task beeline
)
Makefile:187: recipe for target 'example-csv' failed
make: *** [example-csv] Error 1
The pg_isready check has a timeout of default 3s. The full stack runs slow, and when it was 2s it timed out every now and then. Change this to 5s to make it more robust.
reference: https://www.postgresql.org/docs/9.3/app-pg-isready.html
Relevant to issue #20
This group "beeline" does literally nothing
There are currently no healthchecks for hue.
Hive-database is unhealthy at random intervals in consul. Might because pg_isready has a timeout of 2s, which might be too short because the system is fairly slow atm with everything provisioned.
data-mesh/resources/nomad-jobs/hue-connect.hcl
Lines 41 to 49 in 5ca8341
Sorry, yes I meant socket-timeout. It is required to be at least as long as any insert statement we will run. ( it governs how long we eill keep the multipart upload running. It will not fail until all files are uploaded, when it will fail because the first connection wich holds the multipart upload to minio has been ended and thus is nonexisting. Maybe set it to 15 minutes?
Originally posted by @fredrikhgrelland in #10
I suspect that alioygur/wait-for
as a prestart hook does not work as we expect in nomad context.
Steps to reproduce.
0. Use vagrant box
While hive-connect groups in process of downloading the image, prestart hook (with image run minio-connect.hcl
) are exited with 0.
I think nomad allocate ports at allocation time, and ports become available. But it does not mean that applications starts and available on those ports.
Add users and intentions using terraform.
The task waitfor-minio-has-required-bucket
in hive-connect.hcl
exits with code 0 immediately after starting up, even though the bucket has not been created in minio. This causes metastore to fail further down the road becuase of missing buckets, but will fix itself by restarting the whole job after 2 retries, at which point waitfor-minio-has-required-bucket
correctly creates the hive
bucket, and metastore spins up successfully. Creates a huge delay in the provisioning time, needs to be fixed before demo on friday.
Relevant to commands in Makefile which create proxy for user to any service using CONSUL_USER_TOKEN
403 with CONSUL_USER_TOKEN
make connect-allow-user-to-minio
, make proxy-user-to-minio
)PS> works fine with CONSUL_MASTER_TOKEN
~/src/github.com/zhenik/data-mesh │ feature/add-presto make connect-allow-user-to-minio ✔ │ 12:03:11
curl -s -H X-Consul-Token:b6e29626-e23d-98b4-e19f-c71a96fbdef7 -X POST -d '{"SourceName": "user", "DestinationName": "minio", "SourceType": "consul", "Action": "allow"}' http://127.0.0.1:8500/v1/connect/intentions
{
"ID": "ad438dcc-6f46-dce8-0e5a-32bebd8f7e68"
}
~/src/github.com/zhenik/data-mesh │ feature/add-presto make proxy-user-to-minio ✔ │ 12:04:14
consul connect proxy -token 47ec8d90-bf0a-4c18-8506-8d492b131b6d -service user -upstream minio:8090 -log-level debug
==> Consul Connect proxy starting...
Configuration mode: Flags
Service: user
Upstream: minio => :8090
Public listener: Disabled
==> Log data will now stream in as it occurs:
2020-05-25T12:04:22.455+0200 [DEBUG] proxy: got new config
2020-05-25T12:04:22.456+0200 [INFO] proxy: Starting listener: listener=127.0.0.1:8090->service:default/minio bind_addr=127.0.0.1:8090
2020-05-25T12:04:22.458+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=5s
2020-05-25T12:04:27.459+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=20s
2020-05-25T12:04:47.460+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=45s
2020-05-25T12:05:32.462+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=1m20s
2020-05-25T12:06:52.464+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=2m5s
2020-05-25T12:08:57.467+0200 [ERROR] proxy.connect.watch: Watch errored: service=user type=connect_leaf error="Unexpected response code: 403 (ACL not found)" retry=3m0s
Needs better trigger
tail: can't open '/tmp/root/hive.log': No such file or directory
Connecting to jdbc:hive2://localhost:10000/default;auth=noSasl
20/05/28 06:33:50 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:176) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:163) ~[hive-exec-3.1.0.jar:3.1.0]
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:715) [hive-jdbc-3.1.0.jar:3.1.0]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) [hive-jdbc-3.1.0.jar:3.1.0]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) [hive-jdbc-3.1.0.jar:3.1.0]
at java.sql.DriverManager.getConnection(DriverManager.java:664) [?:1.8.0_212]
at java.sql.DriverManager.getConnection(DriverManager.java:208) [?:1.8.0_212]
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.Commands.connect(Commands.java:1641) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.Commands.connect(Commands.java:1536) [hive-beeline-3.1.0.jar:3.1.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1384) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1423) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:900) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:795) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1048) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538) [hive-beeline-3.1.0.jar:3.1.0]
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520) [hive-beeline-3.1.0.jar:3.1.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
at org.apache.hadoop.util.RunJar.run(RunJar.java:308) [hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:222) [hadoop-common-3.1.0.jar:?]
20/05/28 06:33:50 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default;auth=noSasl: Could not establish connection to jdbc:hive2://localhost:10000/default;auth=noSasl: null (state=08S01,code=0)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.