Presto is a distributed query engine used in big data application. It can query data from multiple sources like mysql, postgres, redis, kafka, hive elasticsearch and many more using sql like syntax. It can scale to multiple workers
Apache hive is a data warehouse software. It can manage, read and write large datasets in distributed storage using SQL.
MySQL is a relational database management system. It stores data in row based table structure.
In large systems there is a possibility to have different flavours of databases. It is messy to use different query languages that every database offers. Presto offers a unified query syntex for all data sources which makes it easy to use different data sources for analytical purposes.
This project uses docker compose to build a mysql, hive and presto container. Data can be queried from mysql and hive using presto cli.
To run the setup
- Install necessary software by
sudo apt update
sudo apt install -y docker.io docker-compose git nano
-
Clone the project
-
cd presto-mysql-hive
-
Run the service by
sudo docker-compose up -d
to run in detached mode. It will take few time to initialize the services. -
Access the presto terminal by typing
sudo docker exec -it presto-server presto
-
In presto terminal run
select * from mysql.world_x.city;
we should get the result -
Exit from presto container
-
To enter data to hive run
docker exec -it hive-server /bin/bash
cd /hive-data-employee
hive -f employee_table.hql
hadoop fs -put employee.csv hdfs://namenode:8020/user/hive/warehouse/testdb.db/employee
-
Exit from this container and access the presto terminal
sudo docker exec -it presto-server presto
-
Run the following query to get data from hive
select * from hive.testdb.employee